Scraping Google SERPs (search engine result pages) is as straightforward or as complicated as the tools we use. For this tutorial, we’ll be using Scrapy, a web scraping framework designed for Python. Python and Scrapy combine to create a powerful duo that we can use to scrape almost any website. Scrapy has many useful built-in features that will make scraping Google a walk in the park without compromising any data we would like to scrape. For example, with Scrapy all it takes is a single command to format our data as CSV or JSON files – a process we would have to code ourselves otherwise. Before jumping into the code itself, let’s first explore a few reasons a Google scraper can be useful. Why Scrape Google? There’s no dispute, Google is the king of search engines. That means there’s a lot of data available in its search results for a savvy scraper to take advantage of. Here are a few applications for a Google scraper: Collecting Customer Feedback Data to Inform Your Marketing In the modern shopping experience, it is common for consumers to look for product reviews before deciding on a purchase. With this in mind, a powerful application for a Google SERPs scraper is to collect reviews and customer feedback from your competitor’s products to understand what’s working and what’s not working for them. It can be to improve your product, find a way to differentiate yourself from the competition or to know which features or experiences to highlight in your marketing. Keep this in mind because we’ll be building our scraper around this issue exactly. Inform Your SEO and PPC Strategy According to Oberlo, “Google has 92.18 percent of the market share as of July 2019” and it “has been visited 62.19 billion times this year”. With that many eyes on the SERPs, getting your business to the top of these pages for relevant keywords means a lot of money. Web scraping is primarily an info-gathering tool. We can use it to know our positions in Google better and benchmark ourselves to the competition. If we look at our positions and compare ourselves to the top pages, we can generate a strategy to outrank them. The same goes for PPC campaigns. Because ads appear at the top of every SERP – and sometimes at the bottom – we tell our scraper to bring the name, description, and link to all the ads appearing at the top of the search results for our targeted keywords. This research will help us find un-targeted keywords, understand our competitor’s strategies and evaluate the copy of their ads to differentiate ours. Generate Content Ideas Google also has many additional features in their SERPs like related searches, “people also ask” boxes and more. Scraping hundreds of keywords allows you to gather all this information in a couple of hours and organize it in an easy-to-analyze database. These are just use cases. Depending on the type of data and the ultimate goal you have, you can use a Google scraper for many different reasons. How to Build a Google Web Scraper Without Getting Blocked As we stated earlier, for this example we’ll build our Google web scraper to collect competitors’ reviews. So, let’s imagine we’re a new startup building a project management software, and we want to understand the state of the industry. Let’s start from there: 1. Choose Your Target Keywords Now that we know our main goal, it’s time to pick the keywords we want to scrape to support it. To pick your target keywords, think of the terms consumers could be searching to find your offering, and identify your competitors. In this example, we’ll target four keywords: “asana reviews”“clickup reviews”“best project management software”“best project management software for small teams” We could add many more keywords to this list, but for this scraper tutorial, they’ll be more than enough. Also, notice that the first two queries are related to direct competitors, while the last two will help us identify other competitors and get an initial knowledge of the state of the industry. 2. Setup Your Development Environment The next step is to get our machine ready to develop our Google scraper. For this, we’ll need a few things: Python version 3 or laterPip – to install Scrapy and other packages, we might need ScraperAPI. Your machine may have a pre-installed Python version. Enter into your command prompt to see if that’s the case. python -v If you need to install everything from scratch, follow our . We’ll be using the same setup, so get that done and come back. Python and Scrapy scraping tutorial Note: something to keep in mind is that the team behind Scrapy recommends installing Scrapy in a virtual environment (VE) instead of globally on your PC or laptop. If you’re unfamiliar, the above Python and Scrapy tutorial shows you how to create the VE and install all dependencies. In this tutorial, we’re also going to be using ScraperAPI to avoid any IP bans or repercussions. Google doesn’t really want us to scrape their SERPs – especially for free. As such, they have implemented advanced anti-scraping techniques that’ll quickly identify any bots trying to extract data automatically. To get around this, ScraperAPI is a complex system that utilizes third-party proxies, machine learning, huge browser farms, and years of statistical data to ensure that our scraper won’t get blocked from any site by rotating our IP address for every request, setting wait times between requests and handling CAPTCHAs. In other words, by just adding a few lines of code, ScraperAPI  will supercharge our scraper, saving us headaches and hours of work. All we need for this tutorial is to get our API Key from ScraperAPI. To get it, just to redeem 5000 free API requests. create a free ScraperAPI account 3. Create Your Project’s Folder After installing Scrapy in your VE, enter this snippet into your terminal to create the necessary folders: scrapy startproject google_scraper cd google_scraper scrapy genspider google api.scraperapi.com Scrapy will first create a new project folder called “google-scraper,” which also happens to be the project’s name. Next, go into this folder and run the command to create a web scraper named “google”. genspider We now have many configuration files, a “spiders” folder containing our scraper, and a Python modules folder containing package files. 4. Import All Necessary Dependencies to Your google.py File The next step is to build a few components that will make our script as efficient as possible. To do so, we’ll need to make our dependencies available to our scraper by adding them at the top of our file: scrapy urllib.parse urlencode urllib.parse urlparse json datetime datetime
API_KEY = import from import from import import from import 'YOUR_API_KEY' With these dependencies in place, we can use them to build requests and handle JSON files. This last detail is important because we’ll be using ScraperAPI’s autoparse functionality. After sending the HTTP request, it will return the data in JSON format, simplifying the process and making it so that we don’t have to write and maintain our own parser. 5. Construct the Google Search Query Google employs a standard and query-able URL structure. You just need to know the URL parameters for the data you need and you can generate a URL to query Google with. That said, the following makes up the URL structure for all Google search queries: http://www.google.com/search There are several standard parameters that make up Google search queries: q represents the search keyword parameter. , for example, will look for results containing the keyword “tshirt.” http://www.google.com/search?q=tshirt The offset point is specified by the start parameter. is an example. is the language parameter. is a good example. The argument allows you to search for a domain (or website). http://www.google.com/search?q=tshirt&as sitesearch=amazon.com is one example. http://www.google.com/search?q=tshirt&start=100 hl http://www.google.com/search?q=tshirt&hl=en as_sitesearch The number of results per page (maximum is 100) is specified by the num parameter. http://www.google.com/search?q=tshirt&num=50 is an example. The safe parameter generates only “safe” results. http://www.google.com/search?q=tshirt&safe=active is a good example. Note: is incredibly useful in building a query-able URL. Bookmark it for more complex scraping projects in the future. Moz’s comprehensive list of google search parameters Alright, let’s define a method to construct our Google URL using this information: google_dict = { : query, : , } site:
        web = urlparse(site).netloc
        google_dict[ ] = web + urlencode(google_dict) + urlencode(google_dict) : def create_google_url (query, site= ) '' 'q' 'num' 100 if 'as_sitesearch' return 'http://www.google.com/search?' return 'http://www.google.com/search?' In our method, we’re setting ‘q’ as query because we’ll specify our actual keywords later in the script to make it easier to make changes to our scraper. 6. Define the ScraperAPI Method To use ScraperAPI, all we need to do is to send our request through ScraperAPI’s server by appending our query URL to the proxy URL provided by ScraperAPI using payload and urlencode. The code looks like this: def get_url(url):
   payload = { : API_KEY, : url, : , : }
   proxy_url = + 
   urlencode(payload) proxy_url 'api_key' 'url' 'autoparse' 'true' 'country_code' 'us' 'http://api.scraperapi.com/?' return Now that we have defined the logic our scraper will use to construct our target URLs, it’s time to build the main spider. 7. Write the Spider Class In Scrapy we can create different classes, called spiders, to scrape specific pages or groups of sites. Thanks to this function, we can build different spiders inside the same project, making it much easier to scale and maintain. name = allowed_domains = [ ]
    custom_settings = { : , : , : , : } : class GoogleSpider (scrapy.Spider) 'google' 'api.scraperapi.com' 'ROBOTSTXT_OBEY' False 'LOG_LEVEL' 'INFO' 'CONCURRENT_REQUESTS_PER_DOMAIN' 10 'RETRY_TIMES' 5 We need to give our spider a name, as this is how Scrapy will determine which script you want to run. The name you choose should be specific to what you’re trying to scrape, as projects with multiple spiders can get confusing if they aren’t clearly named. Because our URLs will start with ScraperAPI’s domain, we’ll also need to add “api.scraper.com” to allowed_domains. ScraperAPI will change the IP address and headers between every retry before returning a failed message (which doesn’t count against our total available API calls). We also want to tell our scraper to ignore the directive in the robots.txt file. This is because by default Scrapy won’t scrape any site which has a contradictory directive inside said file. Finally, we’ve set a few constraints so that we don’t exceed the limits of our free ScraperAPI account. As you can see in the custom_settings code above, we’re telling ScraperAPI to send 10 concurrent requests and to retry 5 times after any failed response. 8. Send the Initial Request It’s finally time to send our HTTP request. It is very simple to do this with the start_requests(self) method: def start_requests(self):
        queries = [ , , , ]
            url = create_google_url(query) scrapy.Request(get_url(url), callback=self.parse, meta={ : }) 'asana+reviews' 'clickup+reviews' 'best+project+management+software' 'best+project+management+software+for+small+teams' yield 'pos' 0 It will loop through a list of queries that will be passed to the create_google_url function as query URL keywords. The query URL we created will then be sent to Google Search via the proxy connection we set up in the get_url function, utilizing Scrapy’s yield. The result will then be given to the parse function to be processed (it should be in JSON format). The {‘pos’: 0} key-value pair is also added to the meta parameter, which is used to count the number of pages scraped. Note: when typing keywords, remember that every word in a keyword is separated by a + sign, rather than a space. 9. Write the Parse Function Thanks to ScraperAPI’s auto parsing functionality, our scraper should be returning a JSON file as a response to our request. Make sure it is by enabling the parameter ‘autoparse’: ‘true’ in the get_url function. Next, we’ll load the complete JSON response and cycle through each result, taking the data and combining it into a new item that we can utilize later. This procedure checks to see whether another page of results is available. The request is invoked again if an additional page is present, repeating until there are no additional pages. def parse(self, response):
        di = json.loads(response.text)
        pos = response.meta[ ]
        dt = datetime.now().strftime( ) result di[ ]:
            title = result[ ]
            snippet = result[ ]
            link = result[ ]
            item = { : title, : snippet, : link, : pos, : dt}
            pos += item
        next_page = di[ ][ ] next_page: scrapy.Request(get_url(next_page), callback=self.parse, meta={ : pos}) 'pos' '%Y-%m-%d %H:%M:%S' for in 'organic_results' 'title' 'snippet' 'link' 'title' 'snippet' 'link' 'position' 'date' 1 yield 'pagination' 'nextPageUrl' if yield 'pos' 10. Run the Spider Congratulations, we built our first Google scraper! Remember, our code can always be changed to add functionality we discover is missing, but for now we have a functional scraper. If you’ve been following along, your google.py file should look like this by now: scrapy urllib.parse urlencode urllib.parse urlparse json datetime datetime
API_KEY = payload = { : API_KEY, : url, : , : }
    proxy_url = + urlencode(payload) proxy_url google_dict = { : query, : , } site:
        web = urlparse(site).netloc
        google_dict[ ] = web + urlencode(google_dict) + urlencode(google_dict) name = allowed_domains = [ ]
    custom_settings = { : , : , : , : } queries = [ , , , ] query queries:
            url = create_google_url(query) scrapy.Request(get_url(url), callback=self.parse, meta={ : }) di = json.loads(response.text)
        pos = response.meta[ ]
        dt = datetime.now().strftime( ) result di[ ]:
            title = result[ ]
            snippet = result[ ]
            link = result[ ]
            item = { : title, : snippet, : link, : pos, : dt}
            pos += item
        next_page = di[ ][ ] next_page: scrapy.Request(get_url(next_page), callback=self.parse, meta={ : pos}) import from import from import import from import 'YOUR_API_KEY' : def get_url (url) 'api_key' 'url' 'autoparse' 'true' 'country_code' 'us' 'http://api.scraperapi.com/?' return : def create_google_url (query, site= ) '' 'q' 'num' 100 if 'as_sitesearch' return 'http://www.google.com/search?' return 'http://www.google.com/search?' : class GoogleSpider (scrapy.Spider) 'google' 'api.scraperapi.com' 'ROBOTSTXT_OBEY' False 'LOG_LEVEL' 'INFO' 'CONCURRENT_REQUESTS_PER_DOMAIN' 10 'RETRY_TIMES' 5 : def start_requests (self) 'asana+reviews' 'clickup+reviews' 'best+project+management+software' 'best+project+management+software+for+small+teams' for in yield 'pos' 0 : def parse (self, response) 'pos' '%Y-%m-%d %H:%M:%S' for in 'organic_results' 'title' 'snippet' 'link' 'title' 'snippet' 'link' 'position' 'date' 1 yield 'pagination' 'nextPageUrl' if yield 'pos' Note: If you want to scrape Google SERPs from different countries (let’s say Italy), all you need to do is change the code inside the country_code parameter in the get_url function. . Check out our documentation to learn every parameter you can customize in ScraperAPI To run our scraper, navigate to the project’s folder inside the terminal and use the following command: scrapy crawl google -o serps.csv Now our spider will run and store all scraped data in a new CSV file named “serps.” This feature is a big-time saver and one more reason to use Scrapy for web scraping Google. The stored data can then be analyzed and used to provide insight for tools, marketing and more. If you’d like to dive deeper into web scraping with Python, check our . Beautiful Soup is a simpler web scraping framework for Python that’s just as powerful for scraping static pages. Python and Beautiful Soup tutorial To make the most out of ScraperAPI, take a look at our . You’ll learn about the most common challenges when scraping large sites and how to overcome them. web scraping and ScraperAPI best practices cheat sheet Happy scraping! . This post was originally published on ScraperAPI

Amazon

Asana

Google

Target

How to Deal With Pagination in Python; A Step-by-Step Guide

Build a Powerful Google Trends Scraper Using PyTrends: A Step-By-Step Guide

Proxy API for Web Scraping

Nominated for 2022 - Seo Sleuth

Too Long; Didn't Read

Data Scraping Google Search Results Using Python and Scrapy

Data Scraping Google Search Results Using Python and Scrapy

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Build a Powerful Google Trends Scraper Using PyTrends: A Step-By-Step Guide

Automation for Girl Scout Events

How to Master Web Scraping in Python: From Zero to Hero

How to Scrape Amazon Reviews with and without Code

How to Write a Python Script to Download Reddit videos

Scraping Google Search Console Backlinks

Build a Powerful Google Trends Scraper Using PyTrends: A Step-By-Step Guide

Automation for Girl Scout Events

How to Master Web Scraping in Python: From Zero to Hero

How to Scrape Amazon Reviews with and without Code

How to Write a Python Script to Download Reddit videos

Scraping Google Search Console Backlinks

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps