How to Build a Web Crawler from Scratch

How often have you wanted a piece of information and have turned to Google for a quick answer? Every piece of information that we need in our daily lives can be obtained from the internet. You can extract data from the web and use it to make the most effective business decisions. This makes web scraping and crawling a powerful tool. If you want to programmatically capture specific information from a website for further processing, you need to either build or use a web scraper or a web crawler. We aim to help you build a web crawler for your own customized use.

But first, let us cover the basics of a web scraper or a web crawler.

Demystifying the terms ‘Web Scraper’ and ‘Web Crawler’

A web scraper is a systematic, well-defined process of extracting specific data about a topic. For instance, if you need to extract the prices of products from an e-commerce website, you can design a custom scraper to pull this information from the correct source.

A web crawler, also known as a ‘spider’ has a more generic approach! You can define a web crawler as a bot that systematically scans the Internet for indexing and pulling content/information. It follows internal links on web pages. In general, a “crawler” navigates web pages on its own, at times even without a clearly defined end goal.

Hence, it is more like an exploratory search of the content on the Web. Search engines such as Google, Bing, and others often employ web crawlers to extract content for a URL or for other links, get URLs of these links and other purposes.

However, it is important to note that web scraping and crawling are not mutually exclusive activities. While web crawling creates a copy of the content, web scraping extracts specific data for analysis, or to create something new.

However, in order to scrape data from the web, you would first have to conduct some sort of web crawling to index and find the information you need. On the other hand, data crawling also involves a certain degree of scraping, like saving all the keywords, the images and the URLs of the web page.

More about Web Crawlers

A web crawler is nothing but a few lines of code. This program or code works as an Internet bot. The task is to index the contents of a website on the internet. Now we know that most web pages are made and described using HTML structures and keywords. Thus, if you can specify a category of the content you need, for instance, a particular HTML tag category, the crawler can look for that particular attribute and scan all pieces of information matching that attribute.

You can write this code in any computer language to scrape any information or data from the internet automatically. You can use this bot and even customize the same for multiple pages that allow web crawling. You just need to adhere to the legality of the process.

There are multiple types of web crawlers. These categories are defined by the application scenarios of the web crawlers. Let us go through each of them and cover them in some detail.

1. General Purpose Web Crawler

A general purpose Web crawler, as the name suggests, gathers as many pages as it can from a particular set of URLs to crawl large-scale data and information. You require a high internet speed and large storage space are required for running a general purpose web crawler. Primarily, it is built to scrape massive data for search engines and web service providers.

2. Focused Web Crawler

Focused Web Crawler is characterized by a focused search criterion or a topic. It selectively crawls pages related to pre-defined topics. Hence, while a general purpose web crawler would search and index all the pages and URLs on a site, the focused crawler only needs to crawl the pages related to the pre-defined topics, for instance, the product information on an e-commerce website.

Thus, you can run this crawler with smaller storage space and slower internet speed. Most search engines, such as Google, Yahoo, and Baidu use this kind of web crawler.

3. Incremental Web Crawler

Imagine you have been crawling a particular page regularly and want to search, index and update your existing information repository with the newly updated information on the site. Would you crawl the entire site every time you want to update the information? That sounds unwanted extra cost of computation, time and memory on your machine. The alternative is to use an incremental web crawler.

An incremental web crawler crawls only newly generated information in web pages. They only look for updated information and do not re-download the information that has not changed, or the previously crawled information. Thus it can effectively save crawling time and storage space.

4. Deep Web Crawler

Most of the pages on the internet can be divided into Surface Web and Deep Web (also called Invisible Web Pages or Hidden Web). You can index a surface page with the help of a traditional search engine. It is basically a static page that can be reached using a hyperlink. Web pages in the Deep Web contain content that cannot be obtained through static links. It is hidden behind the search form.

In other words, you cannot simply search for these pages on the web. Users cannot see it without submitting some certain keywords. For instance, some pages are visible to users only after they are registered. Deep web crawler helps us crawl the information from these invisible web pages.

When do you need a web crawler?

From the above sections, we can infer that a web crawler can imitate the human actions to search the web and pull your content from the same. Using a web crawler, you can search for all the possible content you need. You might need to build a web crawler in one of these two scenarios:

1. Replicating the action of a Search Engine- Search Action

Most search engines or the general search function on any portal sites use focused web crawlers for their underlying operations. It helps the search engine locate the web pages that are most relevant to the searched-topics. Here, the crawler visits web sites and reads their pages and other information to create entries for a search engine index. Post that, you can index the data as in the search engine. In the ACP Cladding services many people are using the web scraping services to get the knowledge about competitors.

To replicate the search function as in the case of a search engine, a web crawler helps:

• Provide users with relevant and valid content
• Create a copy of all the visited pages for further processing

2. Aggregating Data for further actions - Content Monitoring

You can also use a web crawler for content monitoring. You can then use it to aggregate datasets for research, business and other operational purposes. Some obvious use-cases are:

• Collect information about customers, marketing data, campaigns and use this data to make more effective marketing decisions.
• Collect relevant subject information from the web and use it for research and academic study.
• Search information on macro-economic factors and market trends to make effective operational decisions for a company.
• Use a web crawler to extract data on real-time changes and competitor trends.

How can you build a Web Crawler?

There are a lot of open-source and paid subscriptions of competitive web crawlers in the market. You can also write the code in any programming language. Python is one such widely used language. Let us look at a few examples there.

Web Crawler using Python

Python is a computationally efficient language that is often employed to build web scrapers and crawlers. The library, commonly used to perform this action is the ‘scrapy’ package in Python. Let us look at a basic code for the same.

import scrapy

class spider1(scrapy.Spider):
      name = ‘Wikipedia’
      start_urls = [‘https://en.wikipedia.org/wiki/Battery_(electricity)’]

      def parse(self, response):
            pass

The above class consists of the following components:

• a name for identifying the spider or the crawler, “Wikipedia” in the above example.
• a start_urls variable containing a list of URLs to begin crawling from. We are specifying a URL of a Wikipedia page on clustering algorithms.
• a parse() method which will be used to process the webpage to extract the relevant and necessary content.

You can run the spider class using a simple command ‘scrapy runspider spider1.py‘. The output looks something like this.

The above output contains all the links and the information (text content) on the website in a wrapped format. A more focussed web crawler to pull product information and links from an e-commerce website looks something like this:

import requests
from bs4 import BeautifulSoup

def web(page,WebUrl):
     if(page>0):
          url = WebUrl
          code = requests.get(url)
          plain = code.text
          s = BeautifulSoup(plain, “html.parser”)
          for link in s.findAll(‘a’, {‘class’:’s-access-detail-page’}):
               tet = link.get(‘title’)
               print(tet)
               tet_2 = link.get(‘href’)
               print(tet_2)

web(1,’https://www.amazon.in/mobile-phones/b?ie=UTF8&node=1389401031&ref_=nav_shopall_sbc_mobcomp_all_mobiles’)

This snippet gives the output in the following format.

The above output shows that all the product names and their respective links have been enlisted in the output. This is a piece of more specific information pulled by the crawler.

Other crawlers in the market

There are multiple open source crawlers in the market that can help you collect/mine data from the Internet. You can conduct your due research and use the best possible tool for collecting information from the web. A lot of these crawlers are written in different languages like Java, PHP, Node, etc.

While some of these crawlers can work across multiple operating software, some are tailor-made for specific platforms like Linux. Some of them are the GNU Wget written in C, the PHP-crawler in  PHP, JSpider in Java among many others.

You may consider factors like the simplicity of the program, speed of the crawler, ability to crawl over various web sites (flexibility) and memory usage of these tools before you make your final choice.