The internet is a treasure trove of information, and sometimes, you might want to access the underlying HTML source code of a website for various purposes like data extraction, analysis, or automation. , with its rich ecosystem of libraries, makes web scraping a straightforward task. Python In this article, we'll explore how to use Python to fetch the source code of a website. 1. Installing Required Libraries To follow this article, you will need the following: Python 3.6 or higher The requests library Python 3.6 Before we dive into web scraping, ensure you have Python installed on your system. You can download the latest version from . python.org The requests library The "Requests" library in Python is a popular and widely used library for making HTTP requests to web services, websites, and APIs. It simplifies the process of sending HTTP requests and handling HTTP responses, making it easier for developers to interact with web resources. To install the "Requests" library in Python, you can use the Python package manager . Here are the steps to install Requests: pip Open your command prompt or terminal. Run the following command to install Requests: pip install requests 2. Writing Python Code Now that we have Requests installed, let's write a simple Python script to retrieve the source code of a website. import requests

# Define the URL of the website you want to scrape
url = 'https://example.com'

# Send an HTTP GET request to the URL
response = requests.get(url)

# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Print the HTML source code
    print(response.text)
else:
    print('Failed to retrieve the webpage. Status code:', response.status_code) In this script: We import the Requests library with . import requests We define the variable with the URL of the website we want to scrape. You can replace with the URL of your target website. url 'https://example.com' We use to send an HTTP GET request to the specified URL and store the response in the variable. requests.get(url) response We check if the request was successful by examining the HTTP status code. A status code of 200 indicates success. If the request was successful, we print the HTML source code of the website using . response.text The following example shows how to use the above code to get the source code of the Google homepage: import requests

# Define the URL of the website you want to scrape
url = 'https://google.com'

# Send an HTTP GET request to the URL
response = requests.get(url)

# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Print the HTML source code
    print(response.text)
else:
    print('Failed to retrieve the webpage. Status code:', response.status_code) Output: <!DOCTYPE html>
<html itemscope="" itemtype="http://schema.org/WebPage" lang="en">
<head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  <meta content="width=device-width, initial-scale=1.0" name="viewport">
  <title>Google</title>
  ...
</head>
<body>
  ...
</body>
</html> Conclusion Web scraping with Python's Requests library is a straightforward and effective way to access the source code of a website. However, it's essential to be aware of website terms of service and legal considerations when scraping websites. Always ensure that your web scraping activities are ethical and comply with the website's policies. With Requests, you have a powerful tool at your disposal to gather data, automate tasks, and explore the vast world of web content. Happy scraping! Reference: https://requests.readthedocs.io/en/latest/?embedable=true https://pytutorial.com/getting-website-source-using-python-requests/?embedable=true

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Python - Build a Domain Lookup Tool

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Getting the Source of a Website with Python's Requests Library

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

How to Write on an Image in Python

0–100 in Django: Starting an app the right way

The Noonification: Reduce Javascript: Master the Basics (1/11/2023)

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Top 10 CSS Performance Tips (11/14/2022)

How to Write on an Image in Python

0–100 in Django: Starting an app the right way

The Noonification: Reduce Javascript: Master the Basics (1/11/2023)

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Top 10 CSS Performance Tips (11/14/2022)

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps