Let’s start.<\/p>\n
What Is HTML Agility Pack and How to Use It<\/h2>\n
HTML Agility Pack is a tool to read, write and update HTML documents<\/strong>. It is commonly used for web scraping, which is the process of programmatically extracting information from public websites<\/strong>.<\/p>\n To start using HTML Agility Pack, we can install it using NuGet Package Manager:<\/p>\n Once done, we can easily parse an HTML string:<\/p>\n Here, we parse a string containing some basic HTML to get an The While parsing HTML documents from strings is simple, sometimes we will need to obtain our HTML from other sources.<\/p>\n We can easily load HTML from files located on a local hard drive<\/strong>. To demonstrate that, let’s first create an HTML file and save it with the name Then, we can instantiate a new Once loaded, we can query the document contents by using Let’s say our goal is to get HTML from a public website. To parse content straight from a URL, we need to use an instance of the Once we parse the content by calling the Often, websites use client code like javascript to render HTML elements dynamically. This may be a problem when we try to parse HTML from a remote website, causing the content to be unavailable to our program since the client code hasn’t been executed.<\/p>\n If we need to parse dynamically rendered HTML content we can use a browser automation tool like Selenium WebDriver<\/strong>. This works because we will be using an actual browser to retrieve the HTML page. A real browser like Chrome is capable of executing any client code present on the page thus generating all the dynamic content.<\/p>\n We can easily find resources to learn how to work with Selenium WebDriver<\/a> to load a remote website. Once done, we can use the content loaded in the driver’s Inside a<\/strong> Each node has a Each We will often use Finally, we can access neighboring nodes with the Putting all this into practice, we can select all nodes of a specific name regardless of their position in the document tree using Here, we select all the Alternatively, we can use an expression to select a node by explicitly defining its position in the hierarchy using To select nodes relative to the current node we can use the dot ( We can also select nodes based on their attributes<\/strong> like XPath expressions can select specific items in a collection by its zero-based index<\/strong> or using functions like Once we have an Here we select a node in our Similarly, we can remove HTML nodes from a document<\/strong>:<\/p>\n In this case, starting from the same unordered list, we remove the first list item by calling the Likewise, we can alter existing nodes<\/strong> using properties exposes by the Starting with the same unordered list, we replace the inner text in each one of the items in the list and append a Often, we need to write HTML to a file after working with it. We can use the Equally important is writing\u00a0out only part of a document<\/strong>, usually the nodes under a specific known node. The There are several properties and methods that allow us to conveniently navigate the tree of nodes that make the document.<\/p>\n Here, we traverse all nodes in document order and save all the headers we find along the way in the On the other hand, methods like Here, we get all the descendants of the root node and group them by tag name. Finally, we count the occurrences of each tag used in the document. If we apply this to the example HTML that we’ve used before, the output should look like this:<\/p>\n There are some packages that, despite being external to HTML Agility PPack work on top of it to provide additional features.<\/strong><\/p>\n Hazz<\/a> adds W3C-style CSS selectors as an alternative to the XPath syntax that comes bundled with HTML Agility Pack. These are JQuery-style selectors that we may like or know better than XPath.<\/p>\n ScrapySharp<\/a> and DotnetSpider<\/a> are higher-level web scraping frameworks that use HTML Agility Pack as their core HTML parsing engine.<\/p>\n In this article, we’ve learned what HTML Agility Pack is and how to work with it. We’ve also learned how to parse HTML from various sources and how to correctly parse websites that use client code to render dynamic content.<\/p>\n Then, we talked about the structure of an HTML document, how to use selectors to query it, and how to read and manipulate the elements in an HTML document.<\/p>\n Finally, we’ve seen some examples of how to traverse the entire document tree and learned about third-party libraries that work with HTML Agility Pack.<\/p>\n","protected":false},"excerpt":{"rendered":" In this article, we’re going to learn how to use HTML Agility Pack in C# and review some examples of its most important features. Let’s start. What Is HTML Agility Pack and How to Use It HTML Agility Pack is a tool to read, write and update HTML documents. It is commonly used for web […]<\/p>\n","protected":false},"author":6,"featured_media":62189,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[12],"tags":[10,944,1480,1481],"class_list":["post-74727","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-csharp","tag-net","tag-net-core-client-libraries","tag-html","tag-html-agility-pack","et-has-post-format-content","et_post_format-et-post-format-standard"],"yoast_head":"\nInstall-Package HtmlAgilityPack<\/code><\/p>\nvar html = @\"<!DOCTYPE html>\r\n <html>\r\n <body>\r\n <h1>Learn To Code in C#<\/h1>\r\n <p>Programming is really <i>easy<\/i>.<\/p>\r\n <\/body>\r\n <\/html>\";\r\n\r\nvar dom = new HtmlDocument();\r\ndom.LoadHtml(html);\r\n\r\nvar documentHeader = dom.DocumentNode.SelectSingleNode(\"\/\/h1\");\r\n\r\nAssert.Equal(\"Learn To Code in C#\", documentHeader.InnerHtml);<\/pre>\n
HtmlDocument<\/code> object.<\/p>\nHtmlDocument<\/code> object exposes a DocumentNode<\/code> property that represents the root tag of the snippet. We use SelectSingleNode()<\/code> on it to query the document model searching for the h1<\/code> tag inside the document. And, finally, we access the text content of the h1<\/code> tag through the InnerHtml<\/code> property.<\/p>\nParsing HTML With HTML Agility Pack<\/h2>\n
Parsing HTML From a Local File<\/h3>\n
test.html<\/code>:<\/p>\n<!DOCTYPE html>\r\n<html>\r\n<body>\r\n <h1>Learn To Code in C#<\/h1>\r\n <p>Programming is really <i>easy<\/i>.<\/p>\r\n <h2>HTML Agility Pack<\/h2>\r\n <p id='second'>HTML Agility Pack is a popular web scraping tool.<\/p>\r\n <p>Features:<\/p>\r\n <ul>\r\n <li>Parser<\/li>\r\n <li>Selectors<\/li>\r\n <li>DOM management<\/li>\r\n <\/ul>\r\n<\/body>\r\n<\/html><\/pre>\n
HtmlDocument<\/code> object and use its Load()<\/code> method to parse the content of our HTML file:<\/p>\nvar path = @\"test.html\";\r\n\r\nvar doc = new HtmlDocument();\r\ndoc.Load(path);\r\n\r\nvar htmlHeader = doc.DocumentNode.SelectSingleNode(\"\/\/h2\");\r\n\r\nAssert.Equal(\"HTML Agility Pack\", htmlHeader.InnerHtml);<\/pre>\n
DocumentNode.SelectSingleNode()<\/code> method. In this case, we are retrieving the second-level header text via the InnerHtml<\/code> of the h2<\/code> tag in the document.<\/p>\nParsing HTML From the Internet<\/h3>\n
HtmlWeb<\/code> class instead of HtmlDocument<\/code>:<\/p>\nvar url = @\"https:\/\/code-maze.com\/\";\r\n\r\nHtmlWeb web = new HtmlWeb();\r\nvar htmlDoc = web.Load(url);\r\n\r\nvar node = htmlDoc.DocumentNode.SelectSingleNode(\"\/\/head\/title\");\r\n\r\nAssert.Equal(\"Code Maze - C#, .NET and Web Development Tutorials\", node.InnerHtml);<\/pre>\n
Load()<\/code> method of the\u00a0 HtmlWeb<\/code> instance with the site’s URL, we can use the methods we already know to access the content. In this case, we are selecting the title<\/code> tag inside the head<\/code> section of the document.<\/p>\nParsing HTML From a Browser Using Selenium<\/h3>\n
PageSource<\/code> property:<\/p>\nvar options = new ChromeOptions();\r\noptions.AddArguments(\"headless\");\r\n\r\nusing (var driver = new ChromeDriver(options))\r\n{\r\n driver.Navigate().GoToUrl(\"https:\/\/code-maze.com\/\");\r\n\r\n var doc = new HtmlDocument();\r\n doc.LoadHtml(driver.PageSource);\r\n\r\n var node = doc.DocumentNode.SelectSingleNode(\"\/\/head\/title\");\r\n\r\n Assert.Equal(\"Code Maze - C#, .NET and Web Development Tutorials\", node.InnerHtml);\r\n}<\/pre>\nStructure of HtmlDocument<\/h2>\n
HtmlDocument<\/code> instance, there’s a tree of<\/strong> HtmlNode<\/code> elements with a single root node<\/strong>. The root node can be accessed through the DocumentNode<\/code> property.<\/p>\nName<\/code> property that will match the HTML tag that represents, like body<\/code>, or h2<\/code>. On the other hand, elements that are not HTML tags also have nodes whose names will start with a #<\/code>. Examples of this are #document<\/code>, #comment<\/code> or #text<\/code>:<\/p>\n![\"html](\"https:\/\/code-maze.com\/wp-content\/uploads\/2022\/11\/HTMLnodes.drawio.png\")
<\/a><\/p>\nHtmlNode<\/code> exposes the SelectSingleNode()<\/code> and SelectNodes()<\/code> methods to query the entire tree using XPath expressions.<\/p>\nSelectSingleNode()<\/code> will return the first HtmlNode<\/code> that matches the XPath expression along with all its descendants, while if there are no matching nodes it will return null<\/code>.<\/p>\nSelectNodes()<\/code> will return a HtmlNodeCollection<\/code> object containing all nodes that match the XPath expression with its descendants.\u00a0<\/p>\nHtmlNode<\/code> properties InnerHtml<\/code>, InnerText,<\/code> and OuterHtml<\/code> to access the node’s content.<\/p>\nChildNodes<\/code>, FirstChild<\/code>, and ParentNode<\/code> properties among others.<\/p>\nUsing Selectors<\/h2>\n
\/\/<\/code>:<\/p>\nvar doc = new HtmlDocument();\r\ndoc.Load(\"test.html\");\r\n\r\nvar nodes = doc.DocumentNode.SelectNodes(\"\/\/li\");\r\n\r\nAssert.Equal(\"Parser\", nodes[0].InnerHtml);\r\nAssert.Equal(\"Selectors\", nodes[1].InnerHtml);\r\nAssert.Equal(\"DOM Management\", nodes[2].InnerHtml);<\/pre>\n
li<\/code> elements in the HTML file we used in a previous example without having to specify the exact path to the elements.<\/p>\n\/<\/code>:<\/p>\nvar node = doc.DocumentNode.SelectSingleNode(\"\/html\/body\/h2\");\r\n\r\nAssert.Equal(\"HTML Agility Pack\", node.InnerHtml);<\/pre>\n
.<\/code>)\u00a0 expression:<\/p>\nvar body = dom.DocumentNode.SelectSingleNode(\"\/html\/body\");\r\nvar listItems = body.SelectNodes(\".\/ul\/li\");\r\n\r\nAssert.Equal(3, listItems.Count);<\/pre>\n
Attribute Selectors<\/h3>\n
class<\/code> or even id<\/code>. This is done using square bracket syntax:<\/p>\nvar node = dom.DocumentNode.SelectSingleNode(\"\/\/p[@id='second']\");\r\n\r\nAssert.Equal(\"HTML Agility Pack is a popular web scraping tool.\", node.InnerHtml);<\/pre>\n
Collections<\/h3>\n
first()<\/code> or last()<\/code>:<\/p>\nvar secondParagraph = dom.DocumentNode.SelectSingleNode(\"\/\/p[1]\");\r\nvar lastParagraph = dom.DocumentNode.SelectSingleNode(\"\/\/p[last()]\");\r\n\r\nAssert.Equal(\"Programming is really <i>easy<\/i>.\", secondParagraph.InnerHtml);\r\nAssert.Equal(\"Features:\", lastParagraph.InnerHtml);<\/pre>\n
HTML Manipulation<\/h2>\n
HtmlDocument<\/code> object, we can change the structure of the underlying HTML <\/strong>using a collection of methods that work with document nodes. We can manipulate a document by adding and removing nodes as well as changing their content or even their attributes<\/strong>:<\/p>\nvar dom = new HtmlDocument();\r\ndom.Load(\"test.html\");\r\n\r\nvar list = dom.DocumentNode.SelectSingleNode(\"\/\/ul\");\r\n\r\nlist.ChildNodes.Add(HtmlNode.CreateNode(\"<li>Added dynamically<\/li>\"));\r\n\r\nAssert.Equal(@\"<ul>\r\n <li>Parser<\/li>\r\n <li>Selectors<\/li>\r\n <li>DOM management<\/li>\r\n <li>Added dynamically<\/li><\/ul>\", list.OuterHtml);<\/pre>\n
HtmlDocument<\/code> corresponding to the unordered list ul<\/code> that originally contains three list items. Then, we add a newly created HtmlNode<\/code> to the ChildNodes<\/code> collection property of the selected node. Once done, we can inspect the OuterHtml<\/code> property of the ul<\/code> node and see how the new list item node has been added to the document.<\/p>\nvar list = dom.DocumentNode.SelectSingleNode(\"\/\/ul\");\r\n\r\nlist.RemoveChild(list.SelectNodes(\"li\").First());\r\n\r\nAssert.Equal(@\"<ul>\r\n \r\n <li>Selectors<\/li>\r\n <li>DOM management<\/li>\r\n <\/ul>\", list.OuterHtml);<\/pre>\n
RemoveChild()<\/code> method in the previously selected HtmlNode<\/code>.<\/p>\nHtmlNode<\/code> object:<\/p>\nvar list = dom.DocumentNode.SelectSingleNode(\"\/\/ul\");\r\n\r\nforeach (var node in list.ChildNodes.Where(x => x.Name == \"li\"))\r\n{\r\n node.FirstChild.InnerHtml = \"List Item Text\";\r\n node.Attributes.Append(\"class\", \"list-item\");\r\n}\r\n\r\nAssert.Equal(@\"<ul>\r\n <li class=\"\"list-item\"\">List Item Text<\/li>\r\n <li class=\"\"list-item\"\">List Item Text<\/li>\r\n <li class=\"\"list-item\"\">List Item Text<\/li>\r\n<\/ul>\", list.OuterHtml);<\/pre>\nclass<\/code> attribute using Attributes.Append()<\/code>.<\/p>\nWriting Out HTML<\/h2>\n
Save()<\/code> method of the HtmlDocument<\/code> class to do it. This method will save all the nodes in the document to a file including all the changes we may have done using the manipulation API<\/strong>:<\/p>\nvar dom = new HtmlDocument();\r\ndom.Load(\"test.html\");\r\n\r\nusing var textWriter = File.CreateText(\"test_out.html\");\r\ndom.Save(textWriter);<\/pre>\n
HtmlNode<\/code> class exposes the WriteTo()<\/code> method that writes the current node along with all its descendants and the WriteContentTo()<\/code> method that will output only its children:<\/p>\nusing (var textWriter = File.CreateText(\"list.html\"))\r\n{\r\n list.WriteTo(textWriter);\r\n}\r\n\r\nusing (var textWriter = File.CreateText(\"items_only.html\"))\r\n{\r\n list.WriteContentTo(textWriter);\r\n}\r\n\r\nAssert.Equal(\r\n@\"<ul>\r\n <li>Parser<\/li>\r\n <li>Selectors<\/li>\r\n <li>DOM management<\/li>\r\n<\/ul>\", File.ReadAllText(\"list.html\"));\r\n\r\nAssert.Equal(\r\n@\"\r\n <li>Parser<\/li>\r\n <li>Selectors<\/li>\r\n <li>DOM management<\/li>\r\n\", File.ReadAllText(\"items_only.html\"));<\/pre>\nTraversing the DOM<\/h2>\n
HtmlNode<\/code>‘s properties ParentNode<\/code>, ChildNodes<\/code>, NextSibling<\/code>, and others let us access neighboring nodes in the document’s hierarchy. We can use these properties to traverse the node tree one node at a time<\/strong>. To optimally traverse the entire document, it may be a good idea to use recursion<\/a>:<\/p>\nvar toc = new List<HtmlNode>();\r\nvar headerTags = new string[] { \"h1\", \"h2\", \"h3\", \"h4\", \"h5\", \"h6\" };\r\n\r\nvoid VisitNodesRecursively(HtmlNode node)\r\n{\r\n if (headerTags.Contains(node.Name))\r\n toc.Add(node);\r\n\r\n foreach(var child in node.ChildNodes)\r\n VisitNodesRecursively(child);\r\n}\r\n\r\nVisitNodesRecursively(dom.DocumentNode);\r\n\r\n\/\/ extracted nodes:\r\n\/\/ h1 -> Learn To Code in C#\r\n\/\/ h2 --> HTML Agility Pack<\/pre>\ntoc<\/code> collection to build a table of contents for the document. We use the ChildNodes<\/code> property to recursively process all nodes.<\/p>\nDescendants()<\/code>, DescendantsAndSelf()<\/code>, Ancestors()<\/code>, and AncestorsAndSelf()<\/code> return a flat list of nodes relative to the node we call the method on<\/strong>:<\/p>\nvar groups = dom.DocumentNode.DescendantsAndSelf()\r\n .Where(n => !n.Name.StartsWith(\"#\"))\r\n .GroupBy(n => n.Name);\r\n \r\nforeach (var group in groups)\r\n Console.WriteLine($\"Tag '{group.Key}' found {group.Count()} times.\");<\/pre>\nTag 'html' found 1 times.\r\nTag 'body' found 1 times.\r\nTag 'h1' found 1 times.\r\nTag 'p' found 3 times.\r\nTag 'i' found 1 times.\r\nTag 'h2' found 1 times.\r\nTag 'ul' found 1 times.\r\nTag 'li' found 3 times.<\/pre>\n
Third-Party Libraries<\/h2>\n
Conclusion<\/h2>\n