In this article, we will build a program that allows you to scrape or grab data from a website with a Python script. This method of gathering data is called web scraping. Web scraping is all about programmatically using Python or any other programming language to download, clean, and use the data from a web page. Most websites don’t want you scraping their data, and to find out what is legal and permissible for scraping, websites have a dedicated page that shows details of the endpoints allowed. Attach robots.txt to the end of any link to find out about the allowed endpoints. For example, let’s use . https://news.ycombinator.com/robots.txt The result should look like this with this text file below: The screenshot states what endpoints we are allowed and not allowed to scrape from the YCombinator website. A crawl delay means a pause when scraping data from the website with programs, thereby not overloading their servers and slowing down the website because of constant scraping. In this exercise, we scrape the news content's home page, which we can do according to the user agent. Getting Started The Python web scraper requires two necessary modules for scraping the data: Beautiful Soup Requests Beautiful Soup Beautiful Soup is a Python library for extracting data from HTML files. It modifies the file using a parser, turns the data into a valuable document, and saves programmers hours of manual and repetitive work. Requests The requests HTTP library is for downloading HTML files using the link to the website with the function. .get() Creating a Web Scraper Now to the nitty-gritty of this project. Create a new directory, and in there, a file that will contain all the scripts for the web scraper program. Copy and paste the following code: yc_web_page = response.text

print(yc_web_page) # app.py import requests response = requests.get( 'https://news.ycombinator.com/news' ) The code above does the following: Importing the module requests Using the response variable, the requests attached to the function download the HTML files from the link of the website provided .get() Reading the content of the web page with .text If you run this code with the command python and it does not give you any output, it means the two imported modules need to be installed. app.py Run the following commands to install the modules. pip install beautifulsoup4 pip3 install requests The result of the source code should look like this: Next, let’s update the file with the rest of the code using beautiful soup: app.py yc_web_page = response.text article_title = article_tag.get_text() result = { }

print(result) # main.py import requests from bs4 import BeautifulSoup # add this response = requests.get( 'https://news.ycombinator.com/news' ) # add this soup = BeautifulSoup(yc_web_page, 'html.parser' ) article_tag = soup.find(name= "a" , class_= 'titlelink' ) article_link = article_tag.get( 'href' ) article_upvote = soup.find(name= "span" , class_= "score" ).get_text() "title" : article_title, "link" : article_link, "point" : article_upvote Follow the code snippet above by doing the following:: Import the BeautifulSoup function from module bs4 Next, use the variable soup to parse the document from the using the BeautifulSoup function and to get the HTML files yc_web_page html.parser Before going over the rest of the code, let’s open our web browser with the link provided in .get() Next, right-click on the page, and click inspect to view the elements tab of the news page. YCombinator Our web page should look like this: With Beautiful Soup, we can target specific elements on the page with their class names: By assigning the article_tag variable, every page element has a tag name using the function with the element's name, the tag, and the with an underscore. This is done to prevent an overwrite of the class in the element on the web page find() a class_ Now, we want to extract one of the link titles of the using the function article_tag .get_text() Next, extract the link of the using the attribute with the function article_tag href .get() The same applies to the variable, where the tag name, , and the class name are used to extract the points for each article link article_upvote <span> Create a variable result that will display the extracted data as a dictionary with the key and value pair Print out the final result With the whole script written, our page should scrape the data from the news home page of YCombinator and look like this: Conclusion This article taught you how to use Python web scraper to extract data from a web page. Also, the functionalities of using a web scraper are that it saves time and effort in producing large data sets faster rather than manually. Learn More Beautiful Soup documentation Requests documentation Scrape the web using Bright Data Scrape at scale with this Scraping Browser

2022 - HackerNoon Contributor of the Year - Data Visualization

2022 - HackerNoon Contributor of the Year - Frontend

2022 - HackerNoon Contributor of the Year - Heroku

2022 - HackerNoon Contributor of the Year - Javascript

2022 - No No No Nodejs

2022 - Remote Work Warrior

Portfolio

Nominated for 2022 - HackerNoon Contributor of the Year - Data Visualization

Nominated for 2022 - HackerNoon Contributor of the Year - Heroku

Nominated for 2022 - HackerNoon Contributor of the Year - Javascript

Nominated for 2022 - HackerNoon Contributor of the Year - Frontend

Nominated for 2022 - Remote Work Warrior

Nominated for 2022 - No No No Nodejs

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How to Build a Python Web Scraper: Scrape Data from any Website

How to Build a Python Web Scraper: Scrape Data from any Website

Too Long; Didn't Read

Teri

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

How to Build a Python Web Scraper: Scrape Data from any Website

Too Long; Didn't Read

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES