Automating your workflow with scripts to get results efficiently is better than being painstakingly done manually. Scraping the web is all about extracting data in a clean and readable format that developers, data analysts, and scientists deploy to read and download an entire web page of its data ethically.
In this article, you will learn and explore the benefits of using Bright Data infrastructure that connects to large datasets with great proxy networks using the Scraping Browser.
Let’s get started.
Bright Data is a web data platform that helps organizations, small businesses, and academic institutions retrieve crucial public web data efficiently, reliably, and flexibly. Bright Data comprises ready-to-use datasets that are GDPR and CCPA-compliant.
Playwright is used to navigating target websites just like the function of Puppeteer interacting with the site’s HTML code to extract the data you need.
Before writing a single script, check if you have Python installed on your system using this command in the command line interface (CLI) or terminal:
python --version
If the version is not present in the terminal after running the command, go to the official website of Python to download it to your local machine.
Create a new account on Bright Data to gain access to the admin dashboard of the Scraping Browser for the proxy integration with your application.
On the left pane of the dashboard, click on the Proxies and Scraping Infra icon.
Scrolling down the page, select the Scraping Browser. After that, click on the Get started button. If you don’t find it, click the Add button dropdown from the previous image and select the Scraping Browser.
The next screen allows you to rename the proxy name. Click the Add proxy button to pop up a prompt display message. Accept the default change by clicking the Yes button.
Next, click the </> Check out code and integration examples button to configure the code in Python.
Environment variables are stored secret keys and credentials in the form of values configured to keep the app running during development and prevent unauthorized access.
Like in a Node.js app, create a new file called .env
in the root directory. But first, you will need to install the Python package python-dotenv
.
pip3 install python-dotenv
The package reads the key-value pairs of the environment variables set.
To confirm the installation of the package python-dotenv
, run this command that lists all installed packages present:
pip3 list
Next, copy-paste this code into the .env
file:
.env
USERNAME="<user-name>"
HOST="<host>"
Replace the values in the quotation with the values from Bright Data.
In the project directory, create a new file called app.py
to handle scraping the web.
Installing packages
You will need to install these two libraries, asyncio
, and playwright, with this command:
pip3 install asyncio
pip3 install playwright
Now, copy-paste this code:
app.py
import asyncio
import os
from playwright.async_api import async_playwright
from dotenv import load_dotenv
load_dotenv()
auth = os.getenv("USERNAME")
host = os.getenv("HOST")
browser_url = f'wss://{auth}@{host}'
async def main():
async with async_playwright() as pw:
print('connecting');
browser = await pw.chromium.connect_over_cdp(browser_url)
print('connected');
page = await browser.new_page()
print('goto')
await page.goto('http://lumtest.com/myip.json', timeout=120000)
print('done, evaluating')
print(await page.evaluate('()=>document.documentElement.outerHTML'))
await browser.close()
asyncio.run(main())
The code above does the following:
asyncio
, async_playwright
, load_dotenv
, and os
load_dotenv()
is responsible for reading the variables from the .env
fileos.getenv()
method returns the values of the environment variable keymain()
function is asynchronous, and within the function, the playwright module connects to the data zonenew_page()
method gets the page HTML and, with the goto
method, leads to the destination site with a timeout of 2 minutespage.evaluate()
method will query the page and print out the result after accessing the page elements and firing up the eventsbrowser.close()
method
To test this application, run with the command:
python app.py
The prospects of evaluating and extracting meaningful data are the heart and operation of what Bright Data offers.
This tutorial showed you how to use the Scraping Browser in Python with the Playwright package to read data from a website.
Try Bright Data today!