![\"Python](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/build-python-web-crawler-scratch.png\")
\nWhy would anyone want to collect more data when there is so much already? Even though the magnitude of information is alarmingly large, you often find yourself looking for data that is unique to your needs.<\/p>\n
For example, what would you do if you wanted to collect info on the history of your favorite basketball team or your favorite ice cream flavor?<\/p>\n
Enterprise data collection is essential in the day-to-day life of a data scientist because the ability to collect actionable data on trends of the modern-day means possible business opportunities.<\/p>\n
In this tutorial, you\u2019ll learn about web crawling via a simple online store.<\/p>\n
Let\u2019s review basic HTML anatomy. Nearly all websites on the Internet<\/a> are built using the combination of HTML and CSS code (including JavaScript, but we won\u2019t talk about it here).<\/p>\n Below is a sample HTML code with some critical parts annotated.<\/p>\n The HTML code on the web will be a bit more complicated than this, however. It will be nearly impossible to just look at the code and figure out what it\u2019s doing. For this reason, we will learn about more sophisticated tools to make sense of massive HTML pages, starting with XPath syntax.<\/p>\n The whole idea behind web scraping is to use automation to extract information from the massive sea of HTML tags and their attributes. One of the tools, among many, to use in this process is using XPath.<\/p>\n XPath stands for XML path language. XPath syntax contains intuitive rules to locate HTML tags and extract information from their attributes and texts. For this section, we will practice using XPath on the HTML code you saw in the above picture:<\/p>\n To start using XPath to query this HTML code, we will need a small library:<\/p>\n LXML allows you to read HTML code as a string and query it using XPath. First, we will convert the above string to an HTML element using the Now, let\u2019s write our first XPath code. We will select the bookstore tag first:<\/p>\n Simple! Just write a double forward slash followed by a tag name to select the tag from anywhere of the HTML tree. We can do the same for the book tag:<\/p>\n As you can see, we get a list of two book tags. Now, let\u2019s see how to choose an immediate child of a tag. For example, let\u2019s select the title tag that comes right inside the book tag:<\/p>\n We only have a single element, which is the first title tag. We didn\u2019t choose the second tag because it is not an immediate child of the second book tag. But we can replace the single forward slash with a double one to choose both title tags:<\/p>\n Now, let\u2019s see how to choose the text inside a tag:<\/p>\n Here, we are selecting the text inside the first title tag. As you can see, we can also specify which of the title tags we want using brackets notation. To choose the text inside that tag, just follow it with a forward slash and a Finally, we look at how to locate tags based on their attributes like As expected, we get a single element. Here are a few examples of choosing other tags using attributes:<\/p>\n I suggest you look at this page<\/a> to learn more about XPath.<\/p>\n For this tutorial, we will be scraping this online store\u2019s computers section<\/a>:<\/p>\n We will be extracting every item\u2019s name, manufacturer, and price. To make things easier, we will create a class with these attributes:<\/p>\n Let\u2019s initialize the first item manually:<\/p>\n Now, let\u2019s get down to the serious business. To scrape the website, we will need its HTML source. Achieving this requires using another library:<\/p>\n Requests allow you to send HTTPS requests to websites and, of course, get back the result with their HTML code. It is as easy as calling its get method and passing the webpage address:<\/p>\n If the response comes with a Above, we are converting the result to an LXML compatible object. As we probably repeat this process a few times, we will convert it into a function:<\/p>\n But, here is a problem \u200a\u2014 \u200aany website contains tens of thousands of HTML code, which makes visual exploration of the code impossible. For this reason, we will turn to our browser to figure out which tags and attributes contain the information we want.<\/p>\n After loading the page, right-click anywhere on the page and choose Inspect<\/strong> to open developer tools:<\/p>\n Using the selector arrow,<\/strong> you can hover over and click on parts of the page to find out the element below the cursor and figure out their associated attributes and info. It will also change the bottom window to move to the location of the selected element.<\/p>\n As we can see, all stored items are within Because the class names are changing, we are using part of the class name that is common in all Now, let\u2019s start extracting the item details from the The item names are located inside tags with class names that contain the As expected, we got 28 item names. This time, we are using chained XPath on We have everything we need to scrape all the items on the page. Let\u2019s do it in a loop:<\/p>\n We now have all items on this page. However, if you scroll down, you\u2019ll see the Next<\/strong> button, indicating that there are more items to scrape. We don\u2019t want to visit all pages manually one by one because there can be hundreds.<\/p>\n But if you pay attention to the URL when we click on the Next<\/strong> button every time:<\/p>\n The page number changes at the end. Now, I\u2019ve checked that there are 22 pages of items on the website. So, we will create a simple loop to iterate through the pagination and repeat the scraping process:<\/p>\n I am also using the tqdm<\/a> library, which displays a progress bar when wrapped around an iterable:<\/p>\n Let\u2019s check how many items we have:<\/p>\n 588 computers! Now, let\u2019s store the items we have into a CSV file.<\/p>\n To store the data, we will use the Pandas<\/a> library to create a There you go! Let\u2019s finally save it to a file:<\/p>\n This tutorial was a straightforward example of how to use a web crawler in Python. While mastering the tools you learned today will be more than enough for most of your scraping needs, you may need a few additional tools for particularly nasty websites.<\/p>\n Specifically, I suggest you learn about BeautifulSoup<\/a> if you don\u2019t feel like learning XPath syntax, as BeautifulSoup offers an OOP approach to querying the HTML code<\/a>.<\/p>\n For websites that require logging in or changes dynamically using JavaScript, you should learn one of the best libraries in Python\u200a, \u200aSelenium<\/a>. Finally, for enterprise web scraping, there is Scrapy<\/a>, which covers pretty much every aspect there is to web scraping. Thanks for reading!<\/p>\n<\/html>\n","protected":false},"excerpt":{"rendered":" In this tutorial, you\u2019ll learn about web crawling via a simple online store and how to build a Python web crawler from scratch.<\/p>\n","protected":false},"author":156415844,"featured_media":85183,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2147999,1],"tags":[2109833],"class_list":["post-85180","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dev","category-uncategorized","tag-python"],"yoast_head":"\n![\"Sample](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/sample-html-code.png\")
<\/p>\nXPath with lxml<\/h2>\n
sample_html = \"\"\"\n<bookstore id='main'>\n\n <book>\n <img src='https:\/\/books.toscrape.com\/index.html'>\n <title lang=\"en\" class='name'>Harry Potter<\/title>\n <price>29.99<\/price>\n <\/book>\n\n <book>\n <a href='https:\/\/www.w3schools.com\/xml\/xpath_syntax.asp'>\n <title lang=\"en\">Learning XML<\/title>\n <\/a>\n <price>39.95<\/price>\n <\/book>\n\n<\/bookstore>\n\"\"\"<\/pre>\n
pip install lxml<\/pre>\n
fromstring <\/code>function:<\/p>\nfrom lxml import html\n\nsource = html.fromstring(sample_html)\n\n>>> source\n<Element bookstore at 0x1e612a769a0>\n>>> type(source)\nlxml.html.HtmlElement<\/pre>\n
>>> source.xpath(\"\/\/bookstore\")\n[<Element bookstore at 0x1e612a769a0>]<\/pre>\n
>>> source.xpath(\"\/\/book\")\n[<Element book at 0x1e612afcb80>, <Element book at 0x1e612afcbd0>]<\/pre>\n
>>> source.xpath(\"\/\/book\/title\")\n[<Element title at 0x1e6129dfa90>]<\/pre>\n
>>> source.xpath(\"\/\/book\/\/title\")\n[<Element title at 0x1e6129dfa90>, <Element title at 0x1e612b0edb0>]<\/pre>\n
>>> source.xpath(\"\/\/book\/title[1]\/text()\")\n['Harry Potter']<\/pre>\n
text()<\/code> function.<\/p>\nid<\/code>, class<\/code>, href, <\/code>or any other attribute inside <><\/code>. Below, we will choose the title tag with the name class:<\/p>\n>>> source.xpath(\"\/\/title[@class='name']\")\n[<Element title at 0x1e6129dfa90>]<\/pre>\n
>>> source.xpath(\"\/\/*[@id='main']\") # choose any element with id 'main'\n[<Element bookstore at 0x1e612a769a0>]\n>>> source.xpath(\"\/\/title[@lang='en']\") # choose a title tag with 'lang' attribute of 'en'.\n[<Element title at 0x1e6129dfa90>, <Element title at 0x1e612b0edb0>]<\/pre>\n
Creating a class to store the data<\/h2>\n
![\"Online](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/online-store-screenshot.png\")
<\/p>\nclass StoreItem:\n \"\"\"\n A general class to store item data concisely.\n \"\"\"\n\n def __init__(self, name, price, manufacturer):\n self.name = name\n self.price = price\n self.manufacturer = manufacturer<\/pre>\n
item1 = StoreItem(\"Lenovo IdeaPad\", 749, \"Walmart\")<\/pre>\n
Getting the page source<\/h2>\n
pip install requests<\/pre>\n
import requests\n\nHOME_PAGE = \"https:\/\/slickdeals.net\/computer-deals\/?page=1\"\n>>> requests.get(HOME_PAGE)\n<Response [200]><\/pre>\n
200<\/code> status code, the request was successful. To get the HTML code, we use the content attribute:<\/p>\nr = requests.get(HOME_PAGE)\n\nsource = html.fromstring(r.content)\n\n>>> source\n<Element html at 0x1e612ba63b0><\/pre>\n
def get_source(page_url):\n \"\"\"\n A function to download the page source of the given URL.\n \"\"\"\n r = requests.get(page_url)\n source = html.fromstring(r.content)\n\n return source\nsource = get_source(HOME_PAGE)\n\n>>> source\n<Element html at 0x1e612d11770><\/pre>\n
![\"Choosing](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/choosing-inspect.gif\")
<\/p>\nli<\/code> elements, with a class attribute containing the words fpGridBox<\/code> grid. Let\u2019s choose them using XPath:<\/p>\nsource = get_source(HOME_PAGE)\n\nli_list = source.xpath(\"\/\/li[contains(@class, 'fpGridBox grid')]\")\n>>> len(li_list)\n28<\/pre>\n
li <\/code>elements. As a result, we have selected 28 li<\/code> elements, which you can double-check by counting them on the web page itself.<\/p>\nExtracting the data<\/h2>\n
li<\/code> elements. Let\u2019s first look at how to find the item\u2019s name using the selector arrow:<\/p>\n![\"Find](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/find-item-name.gif\")
<\/p>\nitemTitle<\/code> keyword. Let\u2019s select them with XPath to make sure:<\/p>\nitem_names = [\n li.xpath(\".\/\/a[@class='itemTitle bp-p-dealLink bp-c-link']\") for li in li_list\n]\n\n>>> len(item_names)\n28<\/pre>\n
li<\/code> elements, which requires starting the syntax with a dot. Below, I will write the XPath for other item details using the browser tools:<\/p>\nli_xpath = \"\/\/li[contains(@class, 'fpGridBox grid')]\" # Choose the `li` items\n\nnames_xpath = \".\/\/a[@class='itemTitle bp-p-dealLink bp-c-link']\/text()\"\nmanufacturer_xpath = \".\/\/*[contains(@class, 'itemStore bp-p-storeLink')]\/text()\"\nprice_xpath = \".\/\/*[contains(@class, 'itemPrice')]\/text()\"<\/pre>\n
li_list = source.xpath(li_xpath)\n\nitems = list()\nfor li in li_list:\n name = li.xpath(names_xpath)\n manufacturer = li.xpath(manufacturer_xpath)\n price = li.xpath(price_xpath)\n\n # Store inside a class\n item = StoreItem(name, price, manufacturer)\n items.append(item)\n>>> len(items)\n28<\/pre>\n
Handling the pagination<\/h2>\n
![\"Next](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/next-button-url-1.gif\")
<\/p>\nfrom tqdm.notebook import tqdm # pip install tqdm\n\n# Create a list to store all\nitems = list()\nfor num in tqdm(range(1, 23)):\n url = f\"https:\/\/slickdeals.net\/computer-deals\/?page={num}\"\n source = get_source(url) # Get HTML code\n\n li_list = source.xpath(li_xpath)\n\n for li in li_list:\n name = clean_text(li.xpath(names_xpath))\n manufacturer = clean_text(li.xpath(manufacturer_xpath))\n price = clean_text(li.xpath(price_xpath))\n\n # Store inside a class\n item = StoreItem(name, price, manufacturer)\n items.append(item)<\/pre>\n![\"Progress](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/progress-bar-loading.gif\")
<\/p>\n>>> len(items)\n588<\/pre>\n
Storing the data<\/h3>\n
DataFrame<\/code> and save it to a CSV:<\/p>\nimport pandas as pd\n\ndf = pd.DataFrame(\n {\n \"name\": [item.name for item in items],\n \"price\": [item.price for item in items],\n \"manufacturer\": [item.manufacturer for item in items],\n }\n)\n\ndf.head()<\/pre>\n![\"Name,](\"https:\/\/blog.logrocket.com\/wp-content\/uploads\/2022\/01\/name-price-manufacturer-table.png\")
<\/p>\ndf.to_csv(\"data\/scaped.csv\", index=False)<\/pre>\n
Conclusion<\/h2>\n