Page2API is the Ultimate Web Scraping API

Page2API

page2api

How to Use NextJS, TypeScript, Tailwind, and OneEntry CMS to Build an eCommerce Store 🛒👨‍💻

কিভাবে NextJS, TypeScript, Tailwind, এবং OneEntry CMS একটি ইকমার্স স্টোর তৈরি করতে ব্যবহার করবেন 🛒👨‍💻

So verwenden Sie NextJS, TypeScript, Tailwind und OneEntry CMS zum Aufbau eines E-Commerce-Shops 🛒👨‍💻

Cómo utilizar NextJS, TypeScript, Tailwind y OneEntry CMS para crear una tienda de comercio electrónico 🛒👨‍💻

Comment utiliser NextJS, TypeScript, Tailwind et OneEntry CMS pour créer une boutique de commerce électronique 🛒👨‍💻

ईकॉमर्स स्टोर बनाने के लिए नेक्स्टजेएस, टाइपस्क्रिप्ट, टेलविंड और वनएंट्री सीएमएस का उपयोग कैसे करें 🛒👨‍💻

NextJS、TypeScript、Tailwind、OneEntry CMS を使用して e コマース ストアを構築する方法 🛒👨‍💻

NextJS, TypeScript, Tailwind 및 OneEntry CMS를 사용하여 전자상거래 상점을 구축하는 방법 🛒👨‍💻

Como usar NextJS, TypeScript, Tailwind e OneEntry CMS para construir uma loja de comércio eletrônico 🛒👨‍💻

Как использовать CMS NextJS, TypeScript, Tailwind и OneEntry для создания магазина электронной коммерции 🛒👨‍💻

Bir e-Ticaret Mağazası Oluşturmak için NextJS, TypeScript, Tailwind ve OneEntry CMS Nasıl Kullanılır 🛒👨‍💻

Cách sử dụng NextJS, TypeScript, Tailwind và OneEntry CMS để xây dựng cửa hàng thương mại điện tử 🛒👨‍💻

如何使用 NextJS、TypeScript、Tailwind 和 OneEntry CMS 构建电子商务商店 🛒👨‍💻

Are you a QA tester eager to dive into performance testing without the need for extensive programming expertise? In this article, we'll explore an approachable way for non-programmers to conduct sort of load testing on cloud apps APIs using . Load testing without the need for complex coding - discover how even regular QA testers can use Python to find serious bugs and uncover potential performance bottlenecks. Python Performance testing is a critical aspect of ensuring your applications can handle real-world demands. I will try to explain my approach and Python scripts designed for load testing a cloud service managing browsers. Imagine a cloud service responsible for managing browser profiles (browsers for web scrapping). Users interact via API with the service to create, start, stop, delete, etc profiles. My Python script simulates this scenario, applying load to the cloud service by repeatedly performing these actions. Load Testing Scenario #  Dependencies
import asyncio
import httpx

#  Configuration
API_HOST = 'https://cloud.io'
API_KEY = 'qatest'
API_HEADERS = {
    "x-cloud-api-token": API_KEY,
    "Content-Type": "application/json"
}
CYCLES_COUNT = 3

#  Browser profile configuration
data_start = {
    "proxy": "http://127.0.0.1:8080",
    "browser_settings": {"inactive_kill_timeout": 120}
} Asynchronous functions: The heart of my load tests The function retrieves existing browser profiles from the service, simulating a scenario where users request information. Fetching Browser Profiles: get_profiles async def get_profiles(cl: httpx.AsyncClient):
    resp = await cl.get(f'{API_HOST}/profiles', params={'page_len': 10, 'page': 0}, headers=API_HEADERS)
    return resp.json() The script initiates cycles, each involving starting browser profiles asynchronously. This emulates a scenario where a user creates multiple browsers concurrently and uses browser profiles. Starting Browsers: async def start_profile(cl: httpx.AsyncClient, uuid):
    resp = await cl.post(f'{API_HOST}/profiles/{id}/start', json=data_start, headers=API_HEADERS)
    if error := resp.json().get('error'):
        print(f'Profile {id} not started with error {error}') After starting profiles, the script fetches active profiles and stops them, followed by deleting all profiles. This load scenario assesses the cloud service's responsiveness to dynamic changes and its efficiency in resource cleanup. Stopping browsers and deleting profiles: async def stop_profile(cl: httpx.AsyncClient, uuid):
    resp = await cl.post(f'{API_HOST}/profiles/{id}/stop', headers=API_HEADERS)
    if error := resp.json().get('error'):
        print(f'Profile {id} not stopped with error {error}')

async def delete_profile(cl: httpx.AsyncClient, uuid):
    resp = await cl.delete(f'{API_HOST}/profiles/{id}', headers=API_HEADERS)
    if error := resp.json().get('error'):
        print(f'Profile {id} not stopped with error {error}') The script concludes by checking and reporting on active connections. This step is crucial for understanding the load impact and identifying potential issues related to monitoring. Monitoring connections: understanding load impact for conn in cl._transport._pool.connections:
    if conn._connection._state.value != 1:
        continue

    print(f'Connection in progress: {conn}') Load testing in action The main function orchestrates the load testing cycles, iterating through profiles and executing asynchronous tasks. Each cycle represents a simulated user interaction, creating, using, and deleting browser profiles. async def main():
    async with httpx.AsyncClient(timeout=httpx.Timeout(timeout=300)) as cl:
        for _ in range(CYCLES_COUNT):
            profiles = await get_profiles(cl)
            start_tasks = [asyncio.create_task(start_profile(cl, profile['id'])) for profile in profiles]
            await asyncio.gather(*start_tasks)

            active_browsers = await get_active_profiles(cl)
            stop_tasks = [asyncio.create_task(stop_profile(cl, active_browser['id'])) for active_browser in active_browsers['data']]
            await asyncio.gather(*stop_tasks)

            profiles = await get_profiles(cl)
            del_tasks = [asyncio.create_task(delete_profile(cl, profile['id'])) for profile in profiles]
            await asyncio.gather(*del_tasks)

# Monitor active connections for insights into load impact Tailoring load tests to your needs This script shows a foundation for QAs to tailor load-testing scenarios to their applications. By customizing the number of cycles, adjusting user interactions, and modifying the script to fit specific API endpoints, testers can gain valuable insights into their application's performance under different loads. Here, you'll need essential monitoring tools to gain info on server states, assess the server load, and track resource utilization and logs. Utilize tools like Grafana, Kibana, Prometheus, etc, for comprehensive monitoring. Additionally, keep a close eye on the responses your script receives, ensuring a thorough evaluation of your application's performance. This approach is invaluable in your effective load testing and performance analysis. Additionally, for a more realistic load simulation, consider opening specific pages in your browser. While I personally used a start page in my browsers, you can also explore options like Pyppeteer or Playwright to open multiple tabs and navigate through various pages. This approach enhances the authenticity of your load-testing scenario, closely resembling user interactions with your application. # Attempt to connect to the browser using the provided profile URL
try:
    browser = await connect({'browserWSEndpoint': browser_url, 'defaultViewport': None})
except Exception as e:
    # Handle connection errors and print a message
    print(f'Error occurred when connecting to the browser: {str(e)}')
    return

# Create a new page in the connected browser
page = await browser.newPage()

# Introduce a brief delay to ensure the page is ready
await asyncio.sleep(2)

# Set the viewport dimensions for the page
width, height = 1920, 1080
await page.setViewport({'width': width, 'height': height})

# Try to navigate to a specific URL 
try:
    await page.goto('https://{your_website}')
    # Wait for 10 seconds to simulate user interaction
    await page.waitFor(10000)
    # Introduce another delay for additional stability
    await asyncio.sleep(5)
except pyppeteer.errors.PageError as e:
    # Handle page navigation errors and print a message
    print(f'Error occurred during page navigation: {str(e)}')

# Attempt to take a screenshot of the page
try:
    await page.screenshot(path='screen.png', fullPage=True)
    # Print a success message if the screenshot is captured successfully
    print('Screenshot taken successfully.')
except Exception as e:
    # Handle screenshot capture errors and print a message
    print(f'Error occurred during taking a screenshot: {str(e)}') Python's asynchronous capabilities, coupled with HTTP libraries, make it a versatile tool for load-testing cloud-based systems. This example serves as a starting point for QA engineers looking to learn Python's power in their load-testing attempts. NOTE In my scenario, the described script proved to be robust and impactful. It served as a useful tool in identifying and addressing numerous issues. The script's aggressive nature was okay in pinpointing critical issues, facilitating effective debugging, and leading the way for a seamless and improved user experience, which is pretty good for a QA. In continuation, I will briefly discuss another script utilizing Python's multiprocessing module. This approach aims to enhance load generation by concurrently executing multiple instances of the testing script. The primary goal of multiprocessing is to parallelize the execution of a script, enabling simultaneous interactions with the service. This approach contrasts with the asynchronous approach discussed earlier, where tasks are executed sequentially but concurrently managed. but it also might be very useful. This is more like spam/ddos with the same requests, Key Components Similar to the asynchronous script, we still need to fetch existing browser profiles from the cloud service. Function to retrieve profiles: def get_profiles():
    response = requests.get(url=f"{api}", params=PARAMS, headers=headers)
    return response The core of the script revolves around two functions: start_profiles and stop_profiles. These functions initiate and terminate browser profiles, respectively. Functions: def start_profiles(list_of_profiles_uuids):
    for uuid in list_of_profiles_uuids:
        # ... (API request to start profile)


def stop_profiles(internal_uuids):
    for uuid in internal_uuids:
        # ... (API request to stop profile)


def run_script():
    start_profiles(get_profile_ids())
    stop_profiles(list_of_ids) The script utilizes the multiprocessing module to run the load testing cycle multiple times in parallel. For each cycle, a new process is spawned, and the run_script function is executed concurrently. Multiprocessing execution: if __name__ == "__main__":
    for runs in range(0, 5):
        processes = []
        for i in range(20):
            p = multiprocessing.Process(target=run_script)
            processes.append(p)
            p.start()

        for p in processes:
            p.join() How to generate load with multiprocessing Each process spawned by the multiprocessing module operates independently, executing the run_script function concurrently. This parallelization significantly increases the number of requests sent to the service simultaneously. Parallel execution: By running multiple instances of the script in parallel, we simulate a higher volume of users interacting with the cloud service concurrently. This approach is particularly useful for scenarios where real-world usage involves a large number of simultaneous users. Simulating user interactions: Choosing the right approach The multiprocessing provides a strategy for load-testing applications. It allows QA engineers to experiment with different methodologies based on the unique characteristics of their applications. While asynchronous testing offers efficiency in managing concurrent tasks, multiprocessing excels in parallelizing the entire testing process. You can choose the approach that aligns best with their specific load-testing goals and application requirements. A quick reminder: This demo aims to introduce basic concepts in a beginner-friendly format, highlighting Python's simplicity for QA testers venturing into performance testing. If you have programming challenges, don't hesitate to google stuff and ask colleagues, use ChatGPT or similar tools, and use GitHub Copilot for extra assistance in writing your load-testing scripts. Also published . here

Python-Powered Performance Testing for QA Testers: A Beginner's Guide to Cloud API Load Testing

QA পরীক্ষকদের জন্য পাইথন-চালিত পারফরম্যান্স টেস্টিং: ক্লাউড API লোড পরীক্ষার জন্য একটি শিক্ষানবিস গাইড

Python-basierte Leistungstests für QA-Tester: Ein Leitfaden für Einsteiger in Cloud-API-Lasttests

Pruebas de rendimiento basadas en Python para probadores de control de calidad: una guía para principiantes sobre pruebas de carga de API en la nube

Tests de performances basés sur Python pour les testeurs d'assurance qualité : guide du débutant sur les tests de charge des API Cloud

क्यूए परीक्षकों के लिए पायथन-संचालित प्रदर्शन परीक्षण: क्लाउड एपीआई लोड परीक्षण के लिए एक शुरुआती मार्गदर्शिका

QA テスターのための Python を活用したパフォーマンス テスト: クラウド API 負荷テストの初心者ガイド

QA 테스터를 위한 Python 기반 성능 테스트: Cloud API 부하 테스트 초보자 가이드

Teste de desempenho baseado em Python para testadores de controle de qualidade: um guia para iniciantes em testes de carga de API em nuvem

Тестирование производительности на базе Python для тестировщиков QA: руководство для начинающих по нагрузочному тестированию облачных API

QA Test Uzmanları için Python Destekli Performans Testi: Yeni Başlayanlar İçin Bulut API Yük Testi Kılavuzu

Kiểm tra hiệu suất dựa trên Python dành cho người kiểm tra QA: Hướng dẫn cho người mới bắt đầu về kiểm tra tải API trên đám mây

面向 QA 测试人员的 Python 支持的性能测试：云 API 负载测试初学者指南

Playwright on Steroids: Overcoming Limits With Object-Oriented Programming

The Amazon product reviews page shows important information to customers because it provides insights into the quality and performance of products. However, it is difficult to scrape when you need login access. But tools like are useful to scrap data behind a login. Puppeteer This tutorial will show you how to scrape an Amazon product review behind login, parse the raw data, and export the reviews as CSV with a screenshot using and . Node.js Puppeteer Prerequisites Before you begin, make sure to install the following: Ensure you have Nodejs installed on your computer NodeJs: This Node library allows us to navigate the Amazon website, log in, and extract review information from the product page. Puppeteer: This library exports the data to a CSV file. Papaparse: You need an Amazon account to access the product review page. Amazon account: 1: Get Access to the Public Page This step involves retrieving the full HTML of the Aamazon public page. This HTML contains all the data displayed on the page, and you can access it by using Puppeteer to automate web interaction, which means you can automate the actions of a Chrome browser without having to launch it. Amazon public product page URL: https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/ Now that you have the URL, the next thing to do is write a Nodejs script that will retrieve the HTML content of the page above. Create a new folder for your project and initialize a Node project. Open your terminal and run the command below. Mkdri amazon-review
cd amazon-review
npm init -y This will create a package.json file for your project. Import the Puppeteer library. const puppeteer = require('puppeteer'); Set up a headless browser instance. (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); This starts an asynchronous and await javascript method, launches a headless Chromium browser using Puppeteer, and assigns it to a variable. browser The browser variable will call the page function will create a new browser page (This means a single tab in the browser) Navigate to the product page using the page.goto() function await page.goto('https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/',{ timeout: 60000 }
); This navigates to the URL added as an argument in the goto method. Sometimes, if the page is big and has a lot of information, it takes longer to load. For this tutorial, you will add an option to the method to increase the timeout limit and give the page more time to load. page.goto() { timeout: 60000 } You increased the timeout to 60 seconds (60000 milliseconds). Extract the full HTML content of the page const pageContent = await page.content(); This method returns the full html source code of the specified page and assigns it to the variable. page.content pageContent . Output the HTML content in your console console.log(pageContent); This logs the HTML content to the console, which allows you to view the raw HTML. Close the browser This closes the browser instance to free up space and resources. await browser.close(); Complete code (async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();

// Navigate to the product page
await page.goto(
"https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/",{ timeout: 60000 }

);

// Retrieve the full HTML content of the page
const pageContent = await page.content();

// Output the HTML to the console
console.log(pageContent);

// Close the browser
await browser.close();
})(); Example HTML output 2: Scrape Behind the login This web scraping method allows you to extract data from a website that requires login access. In this section, you will use to automate filling in your Amazon login details ( and ) and click the login button. Follow the steps below: Puppeteer email password Login into your Amazon account This instructs the Puppeteer to navigate to the Amazon login URL await page.goto(
"https://www.amazon.com/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fcart%2Fadd-to-cart%2Fref%3Dnav_custrec_signin&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0"
); Fill in the login details and click the login button After navigating to the login page, this await page.waitForSelector("#ap_email") This ensures that it proceeds only when the element is visible and present in the DOM. #ap_email Once the email input field is present, the script uses the method to simulate typing the Amazon email address into the input field and then clicking the button using the method await ("#continue-announce") page.type continue page.click() page.click Now, the script uses this await page.waitForSelector("#ap_password") to wait for the element to appear; once the password input field is present, the script uses to simulate typing the Amazon password ap_password page.type Replace with your email and password. After entering the password, the script finds the element with the id and simulates a click. auth-signin-button After clicking on the sign-in button, the script uses this await page.waitForNavigation(); wait for the login to complete. await page.waitForSelector("#ap_password");
await page.type("#ap_password", ""); // Replace with your Amazon password
await page.click("#auth-signin-button");
// Wait for the login to complete
await page.waitForNavigation(); . Navigate to the product review page After logging in, the scripts navigate to the product page using this method "https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/",
{ timeout: 60000 }
); 3: Parse Review Data This allows you to extract data from the Amazon product review page by doing the following. Wait for reviews to load This uses ; to wait for the element with the name “ ” to be present. Until review elements are found. page.waitForSelector(".review") review 10 These elements represent individual product reviews on the Amazon product page. Extract information from review elements. After waiting for the information to load, this code block extracts information from the review elements. It retrieves the author's name, review text, and review date. const reviews = await page.$$eval(".review", (reviewElements) => {
return reviewElements.slice(0, 10).map((review) => {
const author = review.querySelector(".a-profile-name").textContent;
const text = review.querySelector(".a-row.review-data").textContent;
const date = review.querySelector(".review-date").textContent;
return { author, text, date };
});
}); 4: Export Reviews to CSV This allows you to save them in a format that is easy to analyze; in this tutorial, you will export them to CSV. Export the data to a CSV file. This is where Papaparse comes in because it exports data to CSV; the data it extracted from the review is passed to method, which converts the review data to CSV format. Papa.unparse() After creating the CSV file, this code writes the CSV data to using Node.js inbuilt file system (fs). amazon_reviews.csv const csvData = Papa.unparse(reviews);
fs.writeFileSync("amazon_reviews.csv", csvData); Output of CSV author,text,date
Whitney Brown,Verified Purchase,"Reviewed in the United States on July 7, 2021"
Eiskatze,Verified Purchase,"Reviewed in the United States on March 4, 2019"
Billy Johnson,Verified Purchase,"Reviewed in the United States on May 24, 2019"
Katelyn goering,Verified Purchase,"Reviewed in the United States on March 24, 2021"
Timothy A. Taylor,Verified Purchase,"Reviewed in the United States on March 19, 2022"
Sandra G.,Verified Purchase,"Reviewed in the United States on March 4, 2019"
Amazon Customer,Verified Purchase,"Reviewed in the United States on March 29, 2021"
Amra,Verified Purchase,"Reviewed in the United States on February 17, 2019"
pietro,Verified Purchase,"Reviewed in Italy on December 28, 2022"
OLIVIER DETRY,Verified Purchase,"Reviewed in France on December 26, 2020" Then use method to take screenshots of the review page and give it a name. For example, you can use . page.screenshot screenshot.png await page.screenshot({ path: "screenshot.png" }); Output of the screenshot The complete codebase You can check out the complete codebase on my Github Repo: https://github.com/wise4rmgod/Amazon-webscrapping Conclusion Web scraping a website that requires a login is easy with tools like Puppeteer, and it's useful for businesses that want to scrap customer reviews, especially from a site like Amazon. This tutorial covers scraping an Amazon product page behind a login using Puppeteer.

How to Scrape Amazon Product Reviews Behind a Login

How We Enhanced Remotion Performance through GPU-Accelerated Server-Side Rendering Running with hardware acceleration in headless mode can be more challenging than it appears. We embarked on this journey with , which is an excellent framework that enables developers to "Make Videos Programmatically.” On our way, explore various Nvidia driver versions, including those from Nvidia's website and Ubuntu's official repositories. Google Chrome Remotion We collaborated with the vibrant Open-source community to find the answer for using . After encountering some setbacks on our way, we were able to make it work for Remotion eventually. We consolidated our findings in the Remotion Docs at and simplified the instructions for Remotion developers to make the most out of it. Although the whole research was done for , the same process will work for any headless application that requires GPU power, especially in headless mode. Remotion GPU with Remotion for server-side rendering Using the GPU in the Cloud Remotion Hardware Acceleration in Browser Rendering Hardware Acceleration is to your browser what " " is for a gasoline car. It uses the power of specialized hardware like GPUs to make your web pages load faster, and videos play smoothly. Hardware acceleration improves the part where the web page is displayed and shown to you, known as rendering. This results in a smoother and snappier browsing experience. Nitrogen Fuel DLAMI from Marketplace? One may wonder why we didn't opt for the AWS Deep Learning AMI or DLAMI. But we needed fine-grained control over the GPU drivers and utilities and the outdated DLAMIs despite all the bloatware was not the answer. Such tasks are often challenging to achieve with pre-packaged AMIs, as discussed in " .” Tailoring the environment to our exact requirements also helped us troubleshoot issues effectively. Why Deep Learning AMI is Holding You Back We experimented with multiple packages to identify the key ones needed for Google Chrome to recognize the GPU. Despite these efforts, GPU recognition remained elusive, and in some instances, GPU drivers were crashing even at initialization. The long and arduous journey has brought us to the end of the tunnel, yet the darkness persists, and the solution remains as elusive as ever. As we decided to call it a day, We stumbled upon on 's Github issues, which did not give us any hope but did give us a new perspective and drive to achieve our goal. this thread Puppeteer The Fresh Start It was time to return to the drawing board and start afresh. We started with a clean base Ubuntu AMI on AWS EC2 Instance. After that, we downloaded and installed the Nvidia Tesla T4G drivers from Nvidia's official website. Once it was confirmed that the GPU was working properly using , we installed Google Chrome and tested it for hardware acceleration. To our relief, Google Chrome finally gave a nod to work with Hardware Acceleration. nvidia-smi Launching an EC2 Instance Launching an EC2 instance can be achieved in various ways, with the AWS Web Console and AWS CLI being two notable options. Regardless of the method, the important parameters are instance type and AMI selection. At the time of writing, the AMI ID for our chosen instance is . For launching the EC2 instance via , the following command can be used after replacing the placeholders with actual values. g4dn.xlarge ami-053b0d53c279acc90 aws-cli aws ec2 run-instances \
  --instance-initiated-shutdown-behavior "terminate" \
  --image-id "ami-053b0d53c279acc90" \
  --instance-type "g4dn.xlarge" \
  --key-name "<KEY_NAME>" \
  --subnet-id "<SUBNET_ID>" \
  --security-group-ids "<SECURITY_GROUP_ID>" \
  --iam-instance-profile Arn="<ARN>" \
  --block-device-mappings '[
  {
    "DeviceName": "/dev/sda1",
    "Ebs": {
      "VolumeSize": 8,
      "VolumeType": "gp3"
    }
  }
]' Ubuntu and Kernel Upgrade Ubuntu 22.04 AMI on AWS comes with Linux Kernel (specifically ). At this point, we have two options either we stick with Kernel or Upgrade to ( ), we decided to upgrade the Kernel. This decision is crucial as the Nvidia driver compiled for one version will not work for the other. ami-053b0d53c279acc90 v5 5.19.0-1025-aws v5 v6 v6.2.0-1013-aws After launching the instance, SSH into the instance and initiate the Ubuntu packages upgrade process. To update all the installed packages, including Linux Kernel, execute the following one-liner. sudo bash -c "apt update && export DEBIAN_FRONTEND=noninteractive && export NEEDRESTART_MODE=a && apt upgrade -y && reboot" We can also split , and . But the reason behind doing it like this is to install these updates in non-interactive mode. apt update apt upgrade -y reboot After the execution, the system will reboot with Linux Kernel Version 6. We can confirm the kernel update with . Rebooting is essential when upgrading the Ubuntu system because it loads the new Kernel on the next boot, and Nvidia drivers will subsequently be built for the updated version. uname -r Nvidia GPU Driver Installation The process of installing GPU drivers is straightforward when we have the correct driver for the GPU and the corresponding compatible version. Before starting the installation, we need to install a couple of packages: and . The former is a bundle with a variety of compilation tools required to compile the Nvidia driver. The latter, while not mandatory, is required by Google Chrome. Therefore, it's a good idea to install it beforehand to enable support for Vulkan ICD loader for Nvidia, should we require it in the future. build-essential libvulkan1 sudo apt install -y build-essential libvulkan1 After build tools are installed, we can proceed to download and install the GPU drivers using the following commands: DRIVER_URL="https://us.download.nvidia.com/tesla/535.104.12/NVIDIA-Linux-x86_64-535.104.12.run"
DRIVER_NAME="NVIDIA-Linux-driver.run"
wget -O "$DRIVER_NAME" "$DRIVER_URL"
sudo sh "$DRIVER_NAME" --disable-nouveau --silent
rm "$DRIVER_NAME" Now we can run to confirm the GPU driver installation. The output will look similar to this: nvidia-smi Configuring the Startup Service We learned this the hard way: running once is needed for the proper initialization of and . Google Chrome and Chromium fail to initialize without this preliminary setup. To automate this process, we will create a service that runs at boot time using the following commands: nvidia-smi EGL ANGLE EGL nvidia-smi echo '[Unit]
Description=Run nvidia-smi at system startup

[Service]
ExecStart=/usr/bin/nvidia-smi
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target' | sudo tee /etc/systemd/system/nvidia-smi.service
sudo systemctl enable nvidia-smi.service
sudo systemctl start nvidia-smi.service Testing the Installation To test hardware acceleration with Google Chrome or Chromium, we can follow these steps. It's worth noting that Google Chrome has more dependencies, making it an easier choice if we plan to use the Node.js library, Puppeteer. On the other hand, Chromium is a lightweight option with a smaller footprint of dependencies, but we can still use Puppeteer with Chromium by installing a few extra dependencies. Installing Chromium Installation Installing Chromium is straightforward, as it is readily available with Ubuntu's official software repositories. We can install it using with the following command. apt sudo apt install -y chromium-browser Testing Chromium is fully prepared to handle Hardware Acceleration tasks. We can verify its readiness by launching Chromium with the specified parameters. chromium-browser --headless --use-gl=angle \
   --use-angle=gl-egl --use-cmd-decoder=passthrough \
   --print-to-pdf=output.pdf 'chrome://gpu' Installing Puppeteer Dependencies To use Puppeteer with Chromium instead of Google Chrome, there are a few extra dependencies that should be installed. sudo apt install ca-certificates fonts-liberation \
    libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 \
    libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 \
    libgbm1 libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 \
    libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 \
    libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 \
    libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
    libxtst6 lsb-release wget xdg-utils Installing Google Chrome Installation Google Chrome can be installed from multiple options, but we will use the Google Chrome official repository for Debian. We need to add it to Ubuntu's repo list along with the public signing key. curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/googlechrom-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/googlechrom-keyring.gpg] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt update
sudo apt install -y google-chrome-stable Testing Google Chrome is ready for our Hardware Acceleration tasks. We can test it by launching Google Chrome with the following parameters. google-chrome-stable --headless --use-gl=angle \
    --use-angle=gl-egl --use-cmd-decoder=passthrough \
    --print-to-pdf=output.pdf 'chrome://gpu' The above commands will generate , and transfer it to your local machine to check the status of Hardware Acceleration in Google Chrome or Chromium. If the process goes smoothly, the resultant will look like this. output.pdf PDF Working with Puppeteer To verify GPU acceleration using the Node.js library Puppeteer, follow these steps: Installation Install Node.js, as this clean Ubuntu AMI does not include any unnecessary packages. Use the following commands: curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
NODE_MAJOR=18
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
sudo apt update
sudo apt install nodejs Testing After installing Node.js, we can install Puppeteer by running in our code directory. Here's a sample file for checking hardware acceleration: npm i puppeteer index.js const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({
    headless: true, 
    args: ['--use-gl=angle', '--use-angle=gl-egl'], 
  });
  const page = await browser.newPage();
  await page.goto('chrome://gpu');
  await page.waitForTimeout(2000);
  await page.screenshot({ path: 'output.png' });
  await browser.close();
})(); Run the script using . This will generate an file containing hardware acceleration information. node index.js output.png Note from the Creator has been immensely helpful in helping the “ ” community adopt GPU-accelerated rendering.He's researched and written about how to obtain, configure and run EC2 containers and headless Chrome correctly in order to take advantage of graphics acceleration.This area has been especially hard to crack, and without we would not have been able to unlock huge speed boosts. Mirza Remotion Mirza We are super grateful for that! -- Jonny Burger Creator / Remotion.dev Conclusion With hardware acceleration now readily available in headless mode, we can harness the power of GPUs for faster and more complex rendering tasks. Our next steps involve creating a custom AMI from this instance and streamlining the process using AWS Image Builder Pipelines for efficiency. Additionally, we plan to extend hardware capabilities to Docker containers, further extending our options. Resources Remotion - Make videos programmatically: https://remotion.dev Remotion - Using the GPU in the cloud: https://www.remotion.dev/docs/miscellaneous/cloud-gpu Cannot enable GPU acceleration: github.com/puppeteer/puppeteer/issues/3637 : Recommend a workflow for using the GPU on a AWS instance https://github.com/remotion-dev/remotion/issues/2889 Getting Started with Headless Chrome: developer.chrome.com/blog/headless-chrome Why Your AWS Deep Learning AMI is Holding You Back and How to Fix: mirzabilal.com/why-your-aws-deep-learning-a.. Deep Learning with “AWS Graviton2 + NVIDIA Tensor T4G” for as low as free* with CUDA 12.2: mirzabilal.com/deep-learning-with-aws-gravi.. How to install FFmpeg with Harware Accelaration on AWS: mirzabilal.com/how-to-install-ffmpeg-with-h.. CPU vs GPU for Video Transcoding - Challenging the Cost-Speed Myth: https://mirzabilal.com/cpu-vs-gpu-for-video-transcoding-challenging-the-cost-speed-myth Also published . here

Unlocking GPU-Accelerated Server-Side Rendering in Remotion With Chrome, Chromium & Puppeteer

Flaky automation tests can be a bane for QA engineers, introducing uncertainty and undermining the reliability of test suites. Drawing from my experience as an SDET and QA Automation mentor, this article offers practical advice to conquer flakiness. While I’ll provide JavaScript and Playwright examples, these universal tips are applicable in any language and framework, helping you write robust and dependable automation tests. Let’s dive in and ensure your tests stand firm. 1. Avoid the implicit wait methods Occasionally, you will encounter situations where you must wait for elements to appear in the DOM or for your application to reach a specific state to proceed with your test scenario. Even with modern, intelligent automation frameworks like Playwright’s auto-wait features, there will be instances where you need to implement custom wait methods. For instance, consider a scenario with text fields and a confirmation button. Due to specific server-side behavior, the confirmation button becomes visible only after approximately five seconds of completing the form. In such cases, resisting the temptation of inserting a five-second implicit wait between two test steps is essential. textField.fill('Some text')
waitForTime(5000)
confirmationButton.click() Use a smart approach instead and wait for the exact state. It can be the text element appearing after some loading or a loader spinning element disappearing after the current step is successfully done. You can check if you are ready for the next test step. button.click()
waitFor(textField.isVisible())
textField.fill() Of course, there are cases where you need to wait for something you cannot check smartly. I suggest adding comments in such places or even adding the reason explanation as a parameter in your wait methods or something like that: waitForTime(5000, {reason: "For unstable server processed something..."} An additional profit of using it is the test reports where you can store such explanations so it will be obvious for someone else or even for future you what and why such five-second waiting here. waitForTime(5000, {reason: "For unstable server processed something..."}
button.click()
waitFor(textField.isVisible())
textField.fill() 2. Use robust and reliable locators to select elements Locators are a crucial part of automation testing, and everyone knows it. However, despite this simple fact, many automation engineers are bailing on the stability and using something like this in their tests. //*[@id="editor_7"]/section/div[2]/div/h3[2] That is a silly approach because the DOM structure is not so static; some different teams can sometimes change it, and e this after your tests fail. So, use the robust locators. You’re welcome to my XPath-related story reading, by the way. https://hackernoon.com/getting-your-way-around-with-common-xpath-methods?embedable=true 3. Make tests independent from each other When running the test suite with multiple tests in a single thread, it’s crucial to ensure each test is independent. This means that the first test should return the system to its original state so the next test doesn’t fail due to unexpected conditions. For example, if one of your tests modifies user settings stored in LocalStorage, clearing the LocalStorage entry for User Settings after the first test runs is a good approach. This ensures that the second test will be executed with default user settings. You can also consider actions like resetting the page to the default entry page and clearing cookies and database entries so that each new test begins with identical initial conditions. For example, in Playwright, you can use the beforeAll(), beforeEach(), afterAll(), and afterEach() annotations to achieve it. Check the next test suite with a couple of tests. The first is changing the user's appearance settings, and the second is checking the default appearance. test.describe('Test suite', () => {
  test('TC101 - update appearance settings to dark', async ({page}) => {
    await user.goTo(views.settings);
    await user.setAppearance(scheme.dark);
  });

  test('TC102 - check if default appearance is light', async ({page}) => {
    await user.goTo(views.settings);
    await user.checkAppearance(scheme.light);
  });
}); If such a test suite runs in parallel, everything will go smoothly. However, if these tests are run one by one, you could face a failed test because one of them is interfering with another. The first test changed the appearance from light to dark. The second test checks if the current appearance is light. But it will fail since the first test changed the default appearance already. To solve such a problem, I’ve added the afterEach() hook executed after each test. In this case, it will navigate to the default view and clear the user's appearance by removing it from the Local Storage. test.afterEach(async ({ page }) => {
  await user.localStorage(appearanceSettings).clear()
  await user.goTo(views.home)
});

test.describe('Test suite', () => {
  test('TC101 - update appearance settings to dark', async ({page}) => {
    await user.goto(views.settings);
    await user.setAppearance(scheme.dark);
  });

  test('TC102 - check if default appearance is light', async ({page}) => {
    await user.goTo(views.settings);
    await user.checkAppearance(scheme.light);
  });
}); Using this approach, each test in this suite will be independent. 4. Use automatic retries wisely No one is immune to encountering flaky tests, and a popular remedy for this issue involves using retries. You can configure retries to occur automatically, even at the CI/CD configuration level. For instance, you can set up automatic retries for each failed test, specifying a maximum retry count of three times. Any test that initially fails will be retried up to three times until the test execution cycle is complete. The advantage is that if your test is only slightly flaky, it may pass after a couple of retries. However, the downside is that it could fail, resulting in extra runtime consumption. Moreover, this practice can inadvertently lead to an accumulation of flaky tests, as many tests may appear to “pass” on the second or third attempt, and you might incorrectly label them as stable. Therefore, I do not recommend setting auto-retries globally for your entire test project. Instead, use retries selectively in cases where you cannot promptly resolve the underlying issues in the test. For instance, pretend you have a test case where you must upload some files using the ‘filechooser’ event. You get the <Timeout exceeded while waiting for event ‘filechooser’> error, which indicates that the Playwright test is waiting for a ‘filechooser’ event to occur but is taking longer ied timeout. async function uploadFile(page: Page, file) {
  const fileChooserPromise = page.waitForEvent('filechooser');
  await clickOnElement(page, uploadButton);
  const fileChooser = await fileChooserPromise;
  await fileChooser.setFiles(file);
} This can be due to various reasons, such as unexpected behavior in the application or a slow environment. Because it is random behavior, the first thing you could think about is to use automatic retry for this test. However, if you retry the entire test, you risk wasting extra time and getting the same behavior you got with the first error in the uploadFile() method itself, so if the error occurs, you don’t need to retry the entire test. To do this, you can use the standard try…catch statement. async function uploadFile(page: Page, file) {
  const maxRetries = 3;
  let retryCount = 0;

  while (retryCount < maxRetries) {
    try {
      const fileChooserPromise = page.waitForEvent('filechooser');
      await clickOnElement(page, uploadButton);
      const fileChooser = await fileChooserPromise;
      await fileChooser.setFiles(file);
      break; // Success, exit the loop
    } catch (error) {
      console.error(`Attempt ${retryCount + 1} failed: ${error}`);
      retryCount++;
    }
  }
} Such an approach will save extra time and make your test less flaky. In closing, the path to reliable automation tests is straightforward. You can minimize or actively address common challenges and implement intelligent strategies. Remember, this journey is ongoing, and your tests will become more dependable with persistence. Say goodbye to flakiness and welcome stability. Your commitment to quality ensures project success. Happy testing! Also published . here

The Four Horsemen of Flaky Tests

ফ্ল্যাকি টেস্টের চার ঘোড়সওয়ার

Die vier Reiter der Flockentests

Los cuatro jinetes de las pruebas escamosas

Les quatre cavaliers des épreuves feuilletées

परतदार परीक्षणों के चार घुड़सवार

不安定なテストの四騎士

플래키 테스트의 네 기수

Os quatro cavaleiros dos testes esquisitos

Четыре всадника нестабильных тестов

Kesintisiz Testlerin Dört Atlısı

Bốn kỵ sĩ của những bài kiểm tra dễ vỡ

Flaky 测试的四骑士

About the issue I am a co-founder of a product called Alto's POS & Inventory. Every day, I review charts, logs, and numbers from multiple platforms to gain a complete understanding of whether everything is okay. On average, this process takes about 15 minutes per day, but sometimes it can extend to 30 minutes. After performing some basic calculations, I realized that over the course of one year, it amounts to more than 12 workdays. That's a substantial amount of time—12 workdays per year—that I spend on repetitive tasks. Consequently, I have decided to address this issue and allocate this time to more productive tasks. Requirements Now that we've identified the problem, let's outline the requirements for the solution: A single-page dashboard that displays all the crucial information. The ability to collect data from multiple platforms. Processing the collected data, mixing it if necessary, and extracting concise insights. Immediate display on the dashboard as a warning and notification if any action requires my attention. Easy accessibility for both me and my co-founders on any device and from anywhere to view this dashboard. About my solution The solution should be worth the effort. For instance, if I have to invest a significant amount of time in resolving this issue, it might be better to leave it as it is now. So, let's strive to find a solution that is easy to implement. Solutions are listed from the easiest to the most challenging: Notifications In my case, the easiest way to notifications is to use . I can quickly set up and start using it immediately. receive Telegram Bots Hosting To make the dashboard accessible on any device, at any time and from anywhere, it should be hosted on services like or . AWS IONOS Data collection Initially, I tried to find existing services that would allow me to easily gather data, but I couldn't find anything that suited my needs. So, I decided to explore the possibility of creating a custom solution. My approach involved collecting data from the platforms we use through their APIs, processing that data, and presenting it on a dashboard. However, I encountered a challenge as many of these platforms didn't have available APIs for data access. This forced me to explore alternative methods, and I found that Puppeteer was the most suitable choice for my specific requirements. Puppeteer Puppeteer is a Node.js which provides a high-level API to control Chrome/Chromium over the DevTools Protocol. Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples: library Generate screenshots and PDFs of pages. Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)). Automate form submission, UI testing, keyboard input, etc. Create an automated testing environment using the latest JavaScript and browser features. Capture a of your site to help diagnose performance issues. timeline trace Chrome Extensions. Test Here's a in action, which performs the following steps: simple example of Puppeteer Navigates to . Google.com Executes a search for " " HackerNoon Clicks on the first search result. Takes a screenshot. import puppeteer from 'puppeteer';

(async (searchValue) => {
  // Launch the browser and open a new blank page
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Search
  await page.goto('https://google.com');
  await page.locator('textarea').fill(searchValue);
  await page.$eval('form', form => form.submit());

  // Go to the first link
  await page.waitForNavigation();
  await page.click(`div[data-async-context^="query:"] a`);

  // Take a screenshot
  await page.waitForNavigation();
  await page.screenshot({path: './screenshot.png'});

  await browser.close();

})("HackerNoon"); Idea validation Let's validate the idea with a simple implementation that closely resembles a real scenario. Since we need to gather data from multiple platforms, let's choose a simpler scenario that is similar to this task, such as building a page that displays HackerNoon top stories and software engineering jobs. The following code collects data from and every hour, generates simple HTML content from this data, and then serves this HTML content when we receive an HTTP request. It's quite straightforward. HackerNoon Top Stories HackerNoon Jobs index.js: import http from 'http';
import * as scraper from './scraper.js';

(async () => {
  let scrapedHtml = 'Try again later...';
  http.createServer((req, res) => {
    res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8'})
    res.end(scrapedHtml);
  
  }).listen(8080);

  scrapedHtml = await scrapeAll();
  setInterval(async () => scrapedHtml = await scrapeAll(), 60*60*1000);
})();

async function scrapeAll() {
  const browser = await scraper.launchBrowser();
  const [stories, jobs] = await Promise.all([
    scraper.getTopStories(browser),
    scraper.getJobs('Software Engineer', browser)
  ]);
  await browser.close();
  return `
    <h2>Top Stories</h2>
    <ul>${stories.map(e => linkToHtml(e.title, e.url)).join('')}</ul>

    <h2>Jobs</h2>
    <ul>${jobs.map(e => linkToHtml(e.title, e.url)).join('')}</ul>
  `;
}

const linkToHtml = (title, url) => {
  return `<li>
    <a target="_blank" href="${url}">
      ${title}
    </a>
  </li>`;
} scraper.js: import puppeteer, {Browser} from 'puppeteer';

/**
 * 
 * @returns {Browser}
 */
export async function launchBrowser() {
  return await puppeteer.launch();
}

/**
 * 
 * @param {Browser} browser
 * @returns {[{title: String, url: String}]}
 */
export async function getTopStories(browser) {
  const page = await browser.newPage();
  await page.goto('https://hackernoon.com/tagged/hackernoon-top-story');

  // Wait for articles
  await page.waitForSelector('main .story-card');

  // Get articles
  const res = [];
  const articles = await page.$$('main .story-card h2 a');
  for (const article of articles) {
    res.push(
      await article.evaluate(el => ({
        "title": el.textContent,
        "url": el.href,
      }))
    );
  }
  return res;
}

/**
 * 
 * @param {String} keyword
 * @param {Browser} browser
 * @returns {[{title: String, url: String}]}
 */
export async function getJobs(keyword, browser) {
  const page = await browser.newPage();
  await page.goto('https://jobs.hackernoon.com');

  // Search
  await page.locator('#search-jobkeyword input').fill(keyword);
  await page.click('button[type=submit]');

  // Wait for result
  await page.waitForSelector('.job-list-item');

  // Get jobs
  const res = [];
  const items = await page.$$('.job-list-item');
  for (const item of items) {
    res.push(
      await item.evaluate(el => ({
        "title": [
          el.querySelector('.job-name'),
          ...el.querySelectorAll('.desktop-view span')
        ].map(e => e.textContent).join(', '),
        "url": el.href,
      }))
    );
  }
  return res;
} The result looks something like this: Advantages In my perspective, this solution offers the following advantages: Relatively quick implementation A single method of data extraction Ease of extension if I need to add an additional platform Disadvantages From my point of view, the disadvantages of this solution are as follows: Requires adjustment if the user interface of any platform changes It may be challenging or impossible to extract data from certain platforms, particularly those with enhanced security measures. Conclusion This solution proved effective for my situation, allowing me to resolve my problem quickly. If you believe there's a better approach to solving it, I would be delighted if you could share your insights. You can access the . source code for this example on GitHub

Streamlining Workflow with Puppeteer

Automation testing with Playwright is gaining popularity every day, and many tools are available to help you to be a master in it. Sometimes the hardest part is to choose the correct one. Automation testing is a great tool gaining popularity due to its flexibility and ease of use especially with JavaScript. Playwright considered to be one of the fastest to run UI-tests with. In this article, we will discuss  the process of writing your first test with Playwright using with code examples. Playwright JavaScript Installing Playwright Before we start writing our first test case, you need to install Playwright. You can do this by running the following command in your terminal inside IDE. I would also recommend installing (it’s free and great for creating your first tests) VScode npm install playwright Writing our first test 🍾 Test case Launch google.com Search Playwright Check Playwright in the title after the search Close browser Let’s start writing the architecture of our test first. const { chromium } = require('playwright');

describe('Google Search Test', () => {
  let browser;
  let page;

  beforeAll(async () => {
    
  });

  afterAll(async () => {
    
  });

  it('should return result search in google ', async () => {
    
  });
}); happens always once before all tests. It’s quite handy to use to set up the test requirements before actual tests. beforeAll For example, generate data for tests through API or initialize variables which you’ll use in the tests. has the same idea as beforeAll. afterAll It happens after all tests only once. You can use to close a browser or do some actions to clean data after running tests. We need these variables to work with a browser and page later. let browser;
let page; This is a test body, where we will write our actual test. it('should return result search in google ', async () => {
    
  }); Let’s add opening a browser inside beforeAll beforeAll(async () => {
    browser = await chromium.launch(); // here you can use any other browser if you want
    page = await browser.newPage();
    await page.goto('https://www.google.com');
  }); Inside we launched Chrome and navigated to google.com. beforeAll Let’s add a test body it('should return search results', async () => {
  await page.fill('id=input', 'Automation testing'); // you an search any other text here
  await page.press('id=input', 'Enter');
  await page.waitForNavigation();
  const title = await page.title();
  expect(title).toContain('Automation testing');
}); Our result ⬆️ We find input on a page and write Playwright there. is a method which you will usually use to enter the value into input you want to test. fill await page.fill('id=input', 'Automation testing'); We simulate pressing Enter while we are inside this input. await page.press('id=input', 'Enter'); We wait until the page loads. So we won’t continue testing before page is fully loaded. await page.waitForNavigation(); Finally, we use the function to retrieve the title of the search results page. title const title = await page.title(); Using assertion we check if the title contains Playwright(or another string which we were searching to confirm that we redirected to the correct page) expect(title).toContain('Automation testing'); Let’s add closing the browser inside afterAll afterAll(async () => {
    await browser.close();
  }); Inside we close browser after all test afterAll Running your test This will run all your test npx playwright test If you want to run a specific test file npx playwright test your-file.js When you run your tests Playwright also generates HTML reports, which are very useful for debugging. It breaks your test into steps, and if you got a failed test you can always see where the failure happened and debugged it faster. Run this to open your report npx playwright show-report The lead image for this article was generated by HackerNoon's via the prompt "a developer writing code". AI Image Generator

From Manual to Automated: Write Your First Test with Playwright

De manual a automatizado: escribe tu primera prueba con Playwright

Du manuel à l'automatisé : Rédigez votre premier test avec Playwright

मैनुअल से स्वचालित तक: नाटककार के साथ अपना पहला टेस्ट लिखें

手動から自動化へ: Playwright で初めてのテストを書く

Do manual ao automatizado: escreva seu primeiro teste com dramaturgo

Từ thủ công đến tự động: Viết bài kiểm tra đầu tiên của bạn với Playwright

从手动到自动化：用 Playwright 编写您的第一个测试

Introduction Google Maps Data is a crucial piece of data for review software companies, sentimental analysts, and data miners as it contains information like user ratings, user reviews, addresses, and images of a particular place. In this tutorial, we will learn how to scrape this valuable Google Maps information with the help of Node JS. And at the end, we will see how can help you scrape Google Maps Reviews without any extra effort, which we require in scraping Google. Serpdog's | Google Maps Reviews API Requirements: Web Parsing with CSS selectors Searching the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the for selecting the perfect tags to make your web scraping journey easier. CSS Selectors Gadget This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the , which will teach you to use this gadget for selecting the best CSS selectors according to your needs. tutorial User Agents User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user. You can also rotate User Agents, read more about this in this article: . How to fake and rotate User Agents using Python 3 If you want to further safeguard your IP from being blocked by Google, you can try these . 10 Tips to avoid getting Blocked while Scraping Websites Install Libraries Before we begin, install these libraries so we can move forward and prepare our scraper. Puppeteer JS Or you can type the below commands in your project terminal to install the libraries: npm i puppeteer Target: Process: Copy the below target URL to extract the HTML data: https://www.google.com/maps/search/coffee/@28.6559457,77.1404218,11z is our query. After that, we have our latitudes and longitudes. The number before at the end is nothing but the zooming intensity of Google Maps. You can decrease or increase it as per your choice. Its value ranges from 2.92, in which the map completely zooms out, to 21, in which the map completely zooms in. Coffee z Latitudes and longitudes are required to pass in the URL. But the zoom parameter is optional. Note: We will use Puppeteer Infinite Scrolling Method to scrape the Google Maps Results. So, let us start preparing our scraper. First, let us create the main function, which will launch the browser and navigate to the target URL. const getMapsData = async () => {     
        browser = await puppeteer.launch({
            headless: false,
            args: ["--disabled-setuid-sandbox", "--no-sandbox"],
        });
        const page = await browser.newPage();
        await page.setExtraHTTPHeaders({
            "User-Agent":
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4882.194 Safari/537.36",
        })
        
        
        await page.goto("https://www.google.com/maps/search/Starbucks/@26.8484046,75.7215344,12z/data=!3m1!4b1" , {
            waitUntil: 'domcontentloaded',
            timeout: 60000
        })
        
        await page.waitForTimeout(3000);
        
        let data =  await scrollPage(page,".m6QErb[aria-label]",2)
        
        console.log(data)
        await browser.close();
        }; Step-by-step explanation: - This will launch the Chromium browser with the options we have set in our code. In our case, we are launching our browser in non-headless mode. puppeteer.launch() - This will open a new page or tab in the browser. browser.newPage() - It is used to pass HTTP headers with every request the page initiates. page.setExtraHTTPHeaders() - This will navigate the page to the specified target URL. page.goto() - It will cause the page to wait for 3 seconds to do further operations. page.waitForTimeout() - At last, we called our infinite scroller to extract the data we need with the , the tag for the scroller , and the number of items we want as parameters. scrollPage() page div Now, let us prepare the infinite scroller. const scrollPage = async(page, scrollContainer, itemTargetCount) => {
        let items = [];
        let previousHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
        while (itemTargetCount > items.length) {
            items = await extractItems(page);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollTo(0, document.querySelector("${scrollContainer}").scrollHeight)`);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight > ${previousHeight}`);
            await page.waitForTimeout(2000);
        }
        return items;
    } Step-by-step explanation: - Scroll the height of the container. previousHeight - Function to parse the scraped HTML. extractItems() In the next step, we just scrolled down the container to a height equal to . previousHeight And in the last step, we waited for the container to scroll down until its height got greater than the previous height. And, at last, we will talk about our parser. const extractItems = async(page)  => {
        let maps_data = await page.evaluate(() => {
        return Array.from(document.querySelectorAll(".Nv2PK")).map((el) => {
            const link = el.querySelector("a.hfpxzc").getAttribute("href");
            return {
            title: el.querySelector(".qBF1Pd")?.textContent.trim(),
            avg_rating: el.querySelector(".MW4etd")?.textContent.trim(),
            reviews: el.querySelector(".UY7F9")?.textContent.replace("(", "").replace(")", "").trim(),
            address: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            description: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(2)")?.textContent.replace("·", "").trim(),
            website: el.querySelector("a.lcr4fd")?.getAttribute("href"),
            category: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            timings: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            phone_num: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            extra_services: el.querySelector(".qty3Ue")?.textContent.replaceAll("·", "").replaceAll("  ", " ").trim(),
            latitude: link.split("!8m2!3d")[1].split("!4d")[0],
            longitude: link.split("!4d")[1].split("!16s")[0],
            link,
            dataId: link.split("1s")[1].split("!8m")[0],
            };
        });
        });
        return maps_data;
        } Step-by-step explanation: - It will return all the elements that match the specified CSS selector. In our case, it is . document.querySelectorAll() Nv2PK -This will return the attribute value of the specified element. getAttribute() - It returns the text content inside the selected HTML element. textContent - Used to split a string into substrings with the help of a specified separator and return them as an array. split() - Removes the spaces from the starting and end of the string. trim() - Replaces the specified pattern from the whole string. replaceAll() Here is the full code: const puppeteer = require('puppeteer');

    const extractItems = async(page)  => {
        let maps_data = await page.evaluate(() => {
        return Array.from(document.querySelectorAll(".Nv2PK")).map((el) => {
            const link = el.querySelector("a.hfpxzc").getAttribute("href");
            return {
            title: el.querySelector(".qBF1Pd")?.textContent.trim(),
            avg_rating: el.querySelector(".MW4etd")?.textContent.trim(),
            reviews: el.querySelector(".UY7F9")?.textContent.replace("(", "").replace(")", "").trim(),
            address: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            description: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(2)")?.textContent.replace("·", "").trim(),
            website: el.querySelector("a.lcr4fd")?.getAttribute("href"),
            category: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            timings: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            phone_num: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            extra_services: el.querySelector(".qty3Ue")?.textContent.replaceAll("·", "").replaceAll("  ", " ").trim(),
            latitude: link.split("!8m2!3d")[1].split("!4d")[0],
            longitude: link.split("!4d")[1].split("!16s")[0],
            link,
            dataId: link.split("1s")[1].split("!8m")[0],
            };
        });
        });
        return maps_data;
        }
    
        const scrollPage = async(page, scrollContainer, itemTargetCount) => {
        let items = [];
        let previousHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
        while (itemTargetCount > items.length) {
            items = await extractItems(page);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollTo(0, document.querySelector("${scrollContainer}").scrollHeight)`);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight > ${previousHeight}`);
            await page.waitForTimeout(2000);
        }
        return items;
        }
    
    
    
    const getMapsData = async () => {
        browser = await puppeteer.launch({
        headless: false,
        args: ["--disabled-setuid-sandbox", "--no-sandbox"],
        });
        const [page] = await browser.pages();
        await page.setExtraHTTPHeaders({
            "User-Agent":
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4882.194 Safari/537.36",
        })
        
    
        await page.goto("https://www.google.com/maps/search/Starbucks/@26.8484046,75.7215344,12z/data=!3m1!4b1" , {
            waitUntil: 'domcontentloaded',
            timeout: 60000
        })
    
        await page.waitForTimeout(5000)  
    
    let data =  await scrollPage(page,".m6QErb[aria-label]",2)
    
    console.log(data)
    await browser.close();
    };
    
    getMapsData(); Results: Our result should look like this 👇🏻: [
    {
        title: 'The Coffee Bean & Tea Leaf',
        avg_rating: '4.7',
        reviews: '79',
        address: 'The Coffee Bean & Tea Lea,Ground Floor, Epicuria Food Court, Plot No-10 Shivaji Place, Najafgarh Rd',
        description: 'Chain coffee bar known for frozen drinks',
        category: 'Coffee shop',
        timings: 'Open ⋅ Closes 11PM',
        phone_num: 'Open ⋅ Closes 11PM',
        extra_services: 'Dine-in   Drive-through   No-contact delivery      Reserve a table',
        latitude: '28.6511983',
        longitude: '77.1215014',
        link: 'https://www.google.com/maps/place/The+Coffee+Bean+%26+Tea+Leaf/data=!4m7!3m6!1s0x390ce3a69997ad37:0xff83fd9a57a7a71e!8m2!3d28.6511983!4d77.1215014!16s%2Fg%2F11sgxr14tq!19sChIJN62XmabjDDkRHqenV5r9g_8?authuser=0&hl=en&rclk=1',
        dataId: '0x390ce3a69997ad37:0xff83fd9a57a7a71e'
    },
    {
        title: 'The Coffee Bean & Tea Leaf',
        avg_rating: '4.0',
        reviews: '271',
        address: 'T320, Ambience Mall, Gurgaon - Delhi Expy',
        description: 'Chain coffee bar known for frozen drinks',
        category: 'Coffee shop',
        timings: 'Open ⋅ Closes 11PM',
        phone_num: 'Open ⋅ Closes 11PM',
        extra_services: 'Dine-in   Takeaway   No-contact delivery',
        latitude: '28.5041789',
        longitude: '77.0970538',
        link: 'https://www.google.com/maps/place/The+Coffee+Bean+%26+Tea+Leaf/data=!4m7!3m6!1s0x390d194c1a223247:0x611f25bf4fddaf08!8m2!3d28.5041789!4d77.0970538!16s%2Fg%2F11cs6ch67r!19sChIJRzIiGkwZDTkRCK_dT78lH2E?authuser=0&hl=en&rclk=1',
        dataId: '0x390d194c1a223247:0x611f25bf4fddaf08'
    },
    ..... Serpdog's Google Maps Reviews API Scraping, in the long run, can become a time-consuming process as it requires you to maintain the scraper according to changing CSS Selectors. To solve this problem, we at also offer that returns the HTML and readymade structured JSON data to the users. Currently, we are working on Google Maps API, which we will launch after some time. Serpdog | Google Search API Google Maps Reviews API Scraping Google also requires solving captchas, a large pool of User agents, and proxies, but Serpdog solves all these problems on its behalf for a smooth scraping experience. Our users also get 100 free requests on the first . sign-up const axios = require('axios');

    axios.get('https://api.serpdog.io/reviews?api_key=APIKEY&data_id=0x89c25090129c363d:0x40c6a5770d25022b')
        .then(response => {
        console.log(response.data);
        })
        .catch(error => {
        console.log(error);
        }); Results: "location_info": {
    "title": "Statue of Liberty",
    "address": "New York, NY",
    "avgRating": "4.7",
    "totalReviews": "83,109 reviews"
    },
    "reviews": [
    {
        "user": {
        "name": "Vo Kien Thanh",
        "link": "https://www.google.com/maps/contrib/106465550436934504158?hl=en-US&sa=X&ved=2ahUKEwj7zY_J4cv4AhUID0QIHZCtC0cQvvQBegQIARAZ",
        "thumbnail": "https://lh3.googleusercontent.com/a/AATXAJxv5_uPnmyIeoARlf7gMWCduHV1cNI20UnwPicE=s40-c-c0x00000000-cc-rp-mo-ba4-br100",
        "localGuide": true,
        "reviews": "111",
        "photos": "329"
        },
        "rating": "Rated 5.0 out of 5,",
        "duration": "5 months ago",
        "snippet": "The icon of the U.S. 🗽🇺🇸. This is a must-see for everyone who visits New York City, you would never want to miss it.There’s only one cruise line that is allowed to enter the Liberty Island and Ellis Island, which is Statue Cruises. You can purchase tickets at the Battery Park but I’d recommend you purchase it in advance. For $23/adult it’s actually very reasonably priced. Make sure you go early because you will have to go through security at the port. Also take a look at the departure schedule available on the website to plan your trip accordingly.As for the Statue of Liberty, it was my first time seeing it in person so what I could say was a wow. It was absolutely amazing to see this monument. I also purchased the pedestal access so it was pretty cool to see the inside of the statue. They’re not doing the Crown Access due to Covid-19 concerns, but I hope it will be resumed soon.There are a gift shop, a cafeteria and a museum on the island. I would say it takes around 2-3 hours to do everything here because you would want to take as many photos as possible.I absolutely loved it here and I can’t wait to come back.The icon of the U.S. 🗽🇺🇸. This is a must-see for everyone who visits New York City, you would never want to miss it. …More",
        "visited": "",
        "likes": "91",
        "images": [
        "https://lh5.googleusercontent.com/p/AF1QipPOBhJtq17DAc9_ZTBnN2X4Nn-EwIEet61Y9JQo=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipPZ2ut1I7LnECqEB2vzrBk-PSXzBxaHEE4S54lk=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipM8nIogBhwcL-dUrd7KaIxZcc_SA6YnJpp50R0C=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipPQ-YP7uw_gHTNb1gGZSGRGRrzLMzOrvh98AmSN=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipOTqBzK30vQZi9lfuhpk5329bnx-twzgIVjwcI1=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipN0TWUE35ajoTdSKelspuUpK-ZTXlRRR9SfPbTa=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipPQH_4HtdXmSdkCiDTv2jO30LksCxpe9KQI4YKw=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipN_OfX2TgXVNry5fli5v-yExbyTAfV4K7SEy3T0=w100-h100-p-n-k-no",
        "https://lh5.googleusercontent.com/p/AF1QipNWKl0TeBmnzMaR_W4-7skitDwHjjJxPePbiSyd=w100-h100-p-n-k-no"
        ]
    },
        ........ Conclusion: This tutorial taught us to scrape Google Maps Results using Node JS. Feel free to if I missed something. Follow me on . Thanks for reading! message me Twitter Additional Resources Scrape Google Organic Search Results Web Scraping Google Images Scrape Google News Results Web Scraping Google With Node JS - A Complete Guide Originally published here.

How to Scrape Data from Google Maps

Bonus: if you like our content and this “Web automation test” article, you can join our web browser automation . Slack community The software development ecosystem is in a constant state of movement. This is especially true when it comes to building websites and other applications. Not only is the technology that you are using changing, but so are the requirements and acceptance criteria. This is where technologies like headless architecture can help with continuous integration and continuous delivery (CI/CD). In this article, we will be going over website test automation. Specifically, we will be talking about UI automation testing using Puppeteer and Browserless. There are other libraries that you can use for website test automation alongside Browserless (i.e. Playwright), but for the sake of this article, we will be using Puppeteer. and Puppeteer integrate seamlessly within your project and allow you to stay on top of testing as well as changing requirements. We’ll go through what both tools are, and do some simple examples. Browserless By the way, a lot of big brands use Browserless for their website test automation, for example, . here is how Samsara implemented it Headless Browser Testing First, let’s explain what headless browser testing is. Headless browser testing allows for automated control of a webpage without any graphical user interface (GUI). This allows a tester/user/QA/developer to do without having to manually test functionality on browsers that we visually see on our monitors. Imagine having a robot test your application for you! That’s pretty much what headless browser testing, is only often times it’s a simple script. Headless browser testing speeds up the testing process while also providing quick feedback during development. automated testing on a web application Puppeteer and Browserless So, you may be thinking how do Puppeteer and Browserless fit into this equation? Puppeteer provides a great high-level API to control Chrome (both headless and non-headless). Puppeteer is maintained by the Chrome DevTools team. Most things you can do manually in the browser can be accomplished using Puppeteer. This lets us easily perform actions to test our UI’s functionality. Puppeteer also allows us to generate screenshots and PDFs of the pages while performing actions on the application for testing. Puppeteer alone is a great tool to use for website test automation. However, Browserless can take your web browser automation to the next level. Let’s say you want to run tests in your production environment, but you don’t want to add Puppeteer as an extra dependency. This is where Browserless comes to the rescue. With Browserless there is no need to install extra packages or dependencies, and it’s compatible with Puppeteer. . This takes a huge load off your application. You can run your tests with Puppeteer in your dev environments, and meanwhile, run Browserless in your prod environments without having to worry about the extra baggage. All you need to do is plug and play Web automation testing setup Before we get to the examples let’s set up our environments. Luckily, there aren’t many dependencies we need to install. There is only one … Puppeteer. npm i puppeteer Once you have run that command you are good to go! The great part about the other tool we’re using, Browserless, is that there isn’t any setup involved other than creating an account. Let’s go to and create an account. Once you get your account set up you will be directed to your Browserless dashboard. www.browserless.io Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies. Keep your API key handy as we will use that for our examples. I should also mention, Browserless provides a great that lets you test out your scripts to see how they work. If you don’t want to go through the hassle of setting up your local machine, this is a great option to use. online debugger tool Website test automation example 1: In this example, we are going to test the functionality of a cool website created by Tim Holman (@twholman) and Tobias van Schneider (@vanschneider) called “The Passive Aggressive Password Machine”. Their website asks you to enter a password, and it will tell you how good your password is. This test is going to be super simple. We are going to test the functionality of the website by using Puppeteer to type a password in the input area. We will then take a screenshot to verify the website gave the user feedback. Taking a screenshot is the simplest form of validation. If you wanted to make this test even more robust, you could programmatically check to make sure the website was giving updated feedback on the password that was entered. See the code for example 1 below. Notice the commented section on line 5. This is where you would add your Browserless API token to run in your Browserless environment. const puppeteer = require("puppeteer");

const scrape = async () => {
    const browser = await puppeteer.launch({headless: false});
    // const browser = await puppeteer.connect({ browserWSEndpoint: 'wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]' })

    const page = await browser.newPage();

    await page.goto('https://trypap.com/');

    await page.setViewport({ width: 1920, height: 1080 });

    await page.type('input[placeholder="Please Enter a Password"]', 'BrowserlessIsCool1234');

    await page.waitForTimeout(2000);

    await page.screenshot({ path: "screenshot.png", fullPage: true });

    await browser.close();
};
scrape(); Once you have this script on your local machine, run in your terminal and you should get a screenshot outputted into the root of your directory. node [title of your script] Here is what it looks like: Congrats! You’ve officially done a simple UI test. Website test automation example 2: In this example, I want to show off the that Browserless provides to us. The /stats API is powered by Google’s Lighthouse project and gives you metrics about performance, accessibility, size, and much more. As mentioned in the documentation, the /stats API is pretty simple to use. To gather this data, send a request to with a simple JSON payload containing a valid URL. This will output a large amount of data, about 370kb. You can narrow down what gets outputted by adding a object with specific categories you want to see. This is similar to how you would use the Lighthouse node API. For our example, we will only be checking the performance category. /stats API POST https://chrome.browserless.io/stats config We are going to run a curl command to receive the output, but if you wanted to run this in JavaScript you could accomplish the same thing with the . For this example, we’ll run the Lighthouse tests on the we used in the first example. Below is the curl command. fetch API Passive Aggressive Password Machine website curl -X POST \
  https://chrome.browserless.io/stats?token=[ADD BROWSERLESS API TOKEN HERE] \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json' \
  -d '
{
  "url": "https://trypap.com/",
    "config": {
        "extends": "lighthouse:default",
        "settings": {
            "onlyCategories": ["performance"]
        }
    }
}' This should return a JSON object with the Lighthouse performance score. At the time of writing this, trypap.com received a 0.9 performance score (on a scale from 0-1). Pretty good! I hope this article helped open up a whole new world of website test automation for you. As always, make sure to check out the and for more exciting web browser automation content! Happy coding!❤️ Browserless blog Browserless YouTube channel If you like our content, you can check out how our clients use Browserless for different use cases: @IrishEnergyBot used web scraping to help create awareness around Green Energy Dropdeck automated slide deck exporting to PDF and generation of PNG thumbnails BigBlueButton, an open-source project, runs automated E2E tests with Browserless Also Published Here

How to Get Started with Website Test Automation

Puppeteer comes out of the box with a lot of great features and makes it dead simple to get started automating your web workflows. In fact, you can likely get everything up and . Because of its fairly simple API you can likely automate whatever it is you're doing with relatively few lines of code. running locally within minutes But, like any other piece of software, the devil is in the details. Since puppeteer (and playwright) are both heavily -heavy, there's a lot of opportunity for things to go wrong. If you're aggregating data or automating sites without APIs then the ever-changing nature of the web can easily show up to ruin the fun. To be clear: it's just as easy for things to go with puppeteer as it is for them to go right. A lot of times it's even easier 😢 async wrong I'm no stranger to puppeteer, or even web automation, and have even written my own headless web drivers in the past. It's been the longest standing passion of mine since I started programming over 10 years ago. With that said, today I'd like to go over a list of things every developer should do to make their puppeteer scripts more ironclad and graceful. It won't solve all your problems, but it'll give you enough situational awareness to make the issue(s) a lot easier to diagnose and fix. With that, let's jump in! Change your page.goto calls Puppeteer will default timeout after 30 seconds in a call when the specified "load" event doesn't happen. Even though there's some great loading events you can key off of (like events) we generally recommend using the option when you . Especially if you're after a particular page element or network call to wait for: page.goto networkidle2 domcontentloaded know what you're doing // DON'T do this:
// This will wait for network requests to be idle for 2 seconds before proceeding.
await page.goto('https://www.example.com', { waitUntil: 'networkidle2' });
await page.waitForSelector('h1');

// Do this instead:
// This will navigate to the page and *immediately* begin waiting for the h1 selector
// once the page's initial HTML is returned. Often much quicker.
await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1'); This essentially is just optimizing your puppeteer code to run as fast as possible and not let slow network requests get in the way. Often times we (browserless.io) will see a network request NOT respond, which causes your whole script to break as never completes successfully within . goto networkidle2 When shouldn't you use this? Well, the only time you really shouldn't is when you're dealing with the unknown. If you're not keenly aware of the pages layout, network calls, or DOM selectors, then doing a or is about as good as you can get to when the page is "interaction ready." networkidle0 networkidle2 Log your failed network calls Lots of times we see that it's a simple network call that fails an entire workflow. This can be differences between running puppeteer locally versus on a cloud server, a site being down, or even networking issues elsewhere. Though the internet generally works, it still relies on physical devices being online and running, which isn't always the case. The best thing you can do is to simply log failed requests and attempt to proceed on: // DON'T do this: there's nothing here catching _any_ network errors!
await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1');

// Do this instead:
page.on('response', (res) => {
  if (!res.ok()) {
    console.error(`Non-200 response from this request: [${req.status()}] "${req.url()}"`);
  }
});

await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1'); While we're at it, we might as well wire up some kind of page error handler as well and get a utility in place for these kinds of logs/warnings. // Drop in your favorite logging technology here. We're using console.error to illustrate.
const logWarning = (message) => console.error(message);

// Page errors here might be deal-breakers
page.on('pageerror', (err) => {
  logWarning(`Page error emitted: "${err.message}"`);
});

page.on('response', (res) => {
  if (!res.ok()) {
    logWarning(`Non-200 response from this request: [${req.status()}] "${req.url()}"`);
  }
});

await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1'); Monitor the browser process Depending on how you're connecting to the browser, there are a few things here that can help us make sure our scripts are resilient. The first thing is doing a listener on the event as it indicates that either the browser itself has crashed or we've been disconnected. Doing this is pretty straightforward: disconnected browser.once('disconnected', () => logWarning(`Browser has closed or crashed and we've been disconnected!`)); Now combine that with our prior code snippets for the following: const logWarning = (message) => console.error(message);

browser.once('disconnected', () => logWarning(`Browser has closed or crashed and we've been disconnected!`));

const page = await browser.newPage();

page.on('pageerror', (err) => {
  logWarning(`Page error emitted: "${err.message}"`);
});

page.on('response', (res) => {
  if (!res.ok()) {
    logWarning(`Non-200 response from this request: [${req.status()}] "${req.url()}"`);
  }
});

await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1'); Intermission As you might have noticed, we've started to add a lot of listeners and other handlers! It's starting to become a bit un-wieldy from a best-practices standpoint, and for the most part, all of this code is essentially just boilerplate. To say this another way: all of our code here has almost nothing to do with the automation side of puppeteer. Because of this, we should start thinking about making a higher-order class that abstracts all of this logic away and lets our business-oriented code (the code that's doing the actual automation) do its thing. Let's define two modules: one to handle the events and logging/monitoring, and the second that just holds our business logic and consumes the first module. This also makes it easier for us later if we need to add more scripts as we've abstracted away all the management aspects of puppeteer into a reusable module. I'll illustrate this in TypeScript since it better aligns with our goal of having a more deterministic puppeteer experience. // puppeteer-helper.ts
import { Browser, Page } from 'puppeteer';

export class PuppeteerHelper {
  private page?: Page;

  static log(...messages: string[]) {
    // Use your logging library here
    console.warn(...messages);
  }

  private disconnectListener = () => {
    PuppeteerHelper.log(
      `Browser has closed or crashed and we've been disconnected!`
    );
  };

  constructor(private browser: Browser) {
    browser.once('disconnected', this.disconnectListener);
  }

  public close = async () => {
    this.browser.off('disconnect', this.disconnectListener);
    this.browser.close();
  }

  public newPage = async () => {
    const page = await this.browser.newPage();

    page.on('pageerror', (err) => {
      PuppeteerHelper.log(`Page error emitted: "${err.message}"`);
    });

    page.on('response', (res) => {
      if (!res.ok()) {
        PuppeteerHelper.log(`[${req.status()}]: "${req.url()}"`);
      }
    });

    // Capture the page so we can track stuff later
    this.page = page;

    return page;
  }
} Now, with that out of the way we can simply incorporate this easily into our simple script: import puppeteer from 'puppeteer';
import { PuppeteerHelper } from './puppeteer-helper.ts';

(async() => {
  const browser = new PuppeteerHelper(await puppeteer.launch());
  const page = await browser.newPage();

  await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('h1');
  await browser.close();
})(); Perfect! Now that we've got a bit of a management layer in place, let's move on. Avoiding 'file://' requests More of a security precaution here, but we really (ideally) want to avoid chrome making calls to the local file system if we can help it. That, plus there's a slew of IP addresses that most cloud providers use to read hosting metadata that we also want to firewall away from Chrome. I want to reiterate how important it is that we NEVER allow file-system access as it's crucial that things like remain hidden. Unless you want someone taking over your cloud machines, it's best to add this right away. /etc/passwd static: string[] urlDeny = [
  'file://',
  'http://169.254',
  'https://169.254',
  '169.254'
];

static: string[] ipDeny = [
  '169.254',
  '192.168',
  '0.0.0.0',
];

// Later, let's use these and make sure they get reject right away:
public newPage = async () => {
  const page = await this.browser.newPage();

  page.on('request', (req) => {
    if (PuppeteerHelper.urlDeny.some((url) => req.url().startsWith(url))) {
      PuppeteerHelper.log(`Blocking request and exiting: "${req.url()}"`);
      this.close();
    }
  });

  page.on('response', (res) => {
    const responseUrl = res.url();
    const remoteAddressIP = res.remoteAddress().ip;

    if (!res.ok()) {
      PuppeteerHelper.log(`[${res.status()}]: "${res.url()}"`);
    }

    if (responseUrl && PuppeteerHelper.urlDeny.some((url) => responseUrl.startsWith(url))) {
      PuppeteerHelper.log(`Blocking request URL and exiting: "${responseUrl}"`);
      this.close();
    }

    if (remoteAddressIP && PuppeteerHelper.ipDeny.some((ip) => remoteAddressIP.startsWith(ip))) {
      PuppeteerHelper.log(`Blocking request IP and exiting: "${responseUrl}"`);
      this.close();
    }
  })

  page.on('pageerror', (err) => {
    PuppeteerHelper.log(`Page error emitted: "${err.message}"`);
  });

  this.page = page;

  return page;
}; With any project there can be security things we'll need to come up along the way, so it's best to always treat any technology's security as a permanent work-in-progress. Here we can have some assurances that things will get killed before network responses back if the resulting IP address or URL is sensitive and we're dealing with a bad actor. While we're on the subject of security, let's do one more thing... Separate Chrome from your application It's a common best-practice to generally separate concerns in software. A good example is ensuring your application code runs on separate hardware than, say, your database. This gives the programmer a lot of flexibility on how to scale but also allows tighter controls on things like firewalls and connection pools for your database. Since we're dealing with a blackbox (Chrome) we should also exercise caution in working with it. Instead of launching Chrome locally with puppeteer let's start Chrome elsewhere and connect to it. browserless makes this trivial as it's a sandboxing layer of sorts for Chrome, meaning you can run it in an ephemeral fashion totally segregated from your app code and data. This, combined with the network helper up above, makes it easy to have more confidence in your deployment. I'll leave the docker aspects of running browserless as an exercise to you ( ), but once you have it running (we assume localhost and port 3000 below), simply connect to it: we've got a lot of resources on that // DON'T launch locally!
const browser = new PuppeteerHelper(await puppeteer.launch());

// DO replace launch with connect:
const browser = new PuppeteerHelper(await puppeteer.connect({ browserWSEndpoint: 'ws://localhost:3000' })); Now that we've got Chrome all situated and away from our codebase, let's do one last thing in the case of failures. Take a picture, it lasts longer! One of the MOST helpful things you can do is screenshot the webpage in the case of a failure. This, combined with network logging, can more easily give you clues as to what's going on. Scraping and automating the web isn't as deterministic as we'd like, however giving yourself as much context and evidence can make experience a whole lot better. In order to do that and save on same network bandwidth, let's just do a quick JPEG image at 50% quality. Setting dimensions at 1024x768 should also keep sizing matters down as well, but you can do whatever you'd like. We'll adjust our 'close' method to capture a last screenshot so we can also keep track of page changes over time as well. This will help you immensely if you get notified at 2AM that something's wrong since you'll have a photo in your mind of the page as it should be public close() {
  this.browser.off('disconnect', this.disconnectListener);
  if (this.page) {
    // screenshot here is a base64-encoded string of the jpeg file.
    // We don't do anything here with it since this greatly depends on your tools and tooling.
    // We'll catch issues here and just return `null` in cases of total failure
    const screenshot = await this.page.screenshot({ type: 'jpeg', quality: 50, fullPage: true }).catch(() => null);
  }
  this.browser.close();
} In order for this to take effect, we'll need to update our business-level code to catch errors and attempt this screenshot. This, too, is pretty straightforward: import puppeteer from 'puppeteer';
import { PuppeteerHelper } from './puppeteer-helper.ts';

(async() => {
  const browser = new PuppeteerHelper(await puppeteer.connect({ browserWSEndpoint: 'ws://localhost:3000' }));

  try {
    const page = await browser.newPage();
    await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
    await page.waitForSelector('h1');
    await browser.close();
  } catch (err) {
    PuppeteerHelper.log(`Error running script: ${err.message}`);
  } finally {
    browser.close();
  }
})(); Now that there's a visual history of what the page looks like over time, we can begin to paint a better understanding of how things will change over time and feel more confident in our automation. Putting it all together, here's the module we've written: // puppeteer-helper.ts
import { Browser, Page } from 'puppeteer-core';

export class PuppeteerHelper {
  private page?: Page;

  constructor(private browser: Browser) {
    browser.once('disconnected', this.disconnectListener);
  }

  // Add some static url's and IPs to our class:
  static urlDeny: string[] = [
    'file://',
    'http://169.254',
    'https://169.254',
    '169.254'
  ];

  static ipDeny: string[] = [
    '169.254',
    '192.168',
    '0.0.0.0',
  ];

  static log(...messages: string[]) {
    // Use your logging library here
    console.warn(...messages);
  }

  private disconnectListener = () => {
    PuppeteerHelper.log(
      `Browser has closed or crashed and we've been disconnected!`
    );
  };

  public close = async () => {
    this.browser.off('disconnect', this.disconnectListener);
    if (this.page) {
      // screenshot here is a base64-encoded string of the jpeg file.
      // We don't do anything here with it since this greatly depends on your tools and tooling.
      // We also catch and return null here in case something fails
      const screenshot = await this.page.screenshot({ type: 'jpeg', quality: 50, fullPage: true }).catch(() => null);
    }
    this.browser.close();
  };

  public newPage = async () => {
    const page = await this.browser.newPage();

    page.on('request', (req) => {
      if (PuppeteerHelper.urlDeny.some((url) => req.url().startsWith(url))) {
        PuppeteerHelper.log(`Blocking request and exiting: "${req.url()}"`);
        this.close();
      }
    });

    page.on('response', (res) => {
      const responseUrl = res.url();
      const remoteAddressIP = res.remoteAddress().ip;

      if (responseUrl && PuppeteerHelper.urlDeny.some((url) => responseUrl.startsWith(url))) {
        PuppeteerHelper.log(`Blocking request URL and exiting: "${responseUrl}"`);
        this.close();
      }

      if (remoteAddressIP && PuppeteerHelper.ipDeny.some((ip) => remoteAddressIP.startsWith(ip))) {
        PuppeteerHelper.log(`Blocking request IP and exiting: "${responseUrl}"`);
        this.close();
      }
    })

    page.on('pageerror', (err) => {
      PuppeteerHelper.log(`Page error emitted: "${err.message}"`);
    });

    page.on('response', (res) => {
      if (!res.ok()) {
        PuppeteerHelper.log(`[${res.status()}]: "${res.url()}"`);
      }
    });

    this.page = page;

    return page;
  };
} Feel free to treat this as a starting place of sorts to contain your management code for puppeteer. There's a lot you can do here, for instance tracking DOM changes over time or even network requests. While these are definitely suggestions we make it's entirely up to your use-case on what is or isn't interesting to note. all Where do we go from here? This is just the starting point! We haven't talked about a lot of things you'll likely need to consider when running puppeteer: How do we ensure we're not running too much traffic? Can we make our scripts more reliable? Tools and other things to consider?! Be sure to follow us on our blog to get the best practices and more. If you're thinking about running your own Chrome cluster of instances, ! we'd love to hear your thoughts and offer some help

7 Tips to Making Your Puppeteer Scripts More Bulletproof

Web Scraping Sites With Session Cookie Authentication Using NodeJS Request

Sitios de raspado web con autenticación de cookies de sesión mediante solicitud de NodeJS

In this article, we are going to completely automate the process of searching for a product on Google Shopping. Specifically, we are going to search for board games and scrape the Google Shopping results page for the product name and price. Below is a visual of what we are going to be targeting on the page. Here's the kicker, all we are going to use is one script and exactly 50 lines of code. How are we going to do this you may ask? The magic of JavaScript. Furthermore, a node library called and a web service called . If you don't know much about these tools I'd highly recommend you check them out further. They offer a lot of neat capabilities! We will go over some of them in this article. Puppeteer Browserless For further info on Puppeteer, I'd encourage watching in 2018. Also, if you're hungry for more Browserless material after reading this article check out the Browserless YouTube channel and . Both the channel and blog offer great tips and tricks to use in your Browserless and Puppeteer environments. Eric Bidelman's talk at the Google I/O conference the Browserless blog Why Puppeteer? Next, let's very quickly explain why we are using Puppeteer to accomplish our task of automating a Google Shopping search. First, Puppeteer provides a great high-level API to control Chrome (both headless and non-headless). Puppeteer is maintained by the Chrome DevTools team. These engineers know what they are doing. Lastly, most things you can do manually in the browser can be accomplished using Puppeteer. This will allow us to easily manipulate our script to perform the actions we want in the browser. Initial setup Alrighty, let's get to it! We are going to start with the initial setup. Luckily for us, there aren't many dependencies we need to install. There's only one... Puppeteer. npm i puppeteer Once you've run that command you are good to go! 👍 Browserless Technically you are all set to start coding, but I want to use Browserless in tandem with Puppeteer. Why am I going to use this tool you're wondering? Let's get into that. The script that we are going to be coding is one of many things that can be done with Puppeteer and headless Chrome. Let's say you want to run this script in one of your production applications, but you don't want to have to add Puppeteer as an extra dependency. What can you do? This is one scenario where Browserless comes to the rescue. With Browserless there is no need to install extra packages or dependencies, and it's compatible with Puppeteer. All you need to do is plug and play (We'll get more into the details of setting up Browserless below). This takes a huge load off of your application. You can test and run Puppeteer scripts in your dev environments and meanwhile run Browserless in your prod environments without having to worry about the extra baggage. Talk about a performance boost! Another area where Browserless shines is in remote development environments (i.e. , gitpod, codepen, codespaces, etc.). Since there is no need to install any software you can easily hook up to the Browserless API and run your Puppeteer scripts in all your remote environments. repl.it Before we get to the actual script, let's look at how to set up Browserless real quick. For starters, you'll need to set up an account. Once you get your account set up, you'll be directed to your Browserless dashboard. Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies. Keep your API key handy as we'll use that once we start writing our script. Other than that we are ready to start coding! The code Ok now for the exciting part! I like to show the final product at the beginning and then dissect each piece from top to bottom. Here is our script: const puppeteer = require("puppeteer");
const fs = require("fs");

const scrape = async () => {
  try {
    const browser = await puppeteer.launch({headless: false});
    // const browser = await puppeteer.connect({
    //   browserWSEndpoint:
    //     "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
    // });

    const page = await browser.newPage();
    await page.goto("https://shopping.google.com/");
    await page.type(
      "input[placeholder='What are you looking for?']",
      "board game"
    );
    await page.click("button[aria-label='Google Search']");
    await page.waitForSelector("#pnnext");

    // Go to next results page
    await page.click("#pnnext");
    await page.waitForSelector("#pnnext");

    // Gather product title
    const title = await page.$$eval("div.sh-dgr__grid-result h4", (nodes) =>
      nodes.map((n) => n.innerText)
    );

    // Gather price
    const price = await page.$$eval(
      "div.sh-dgr__grid-result a.shntl div span span[aria-hidden='true'] span:nth-child(1)",
      (nodes) => nodes.map((n) => n.innerText)
    );

    // Consolidate product search data
    const googleShoppingSearchArray = title.slice(0, 10).map((value, index) => {
      return {
        title: title[index],
        price: price[index],
      };
    });
    const jsonData = JSON.stringify(googleShoppingSearchArray, null, 2);
    fs.writeFileSync("googleShoppingSearchResults.json", jsonData);
    // await browser.close();
  } catch (err) {
    console.error(err);
  }
};
scrape(); Okie doke, let's dive into this. The first 2 lines are pulling in JavaScript modules. We'll want to pull in Puppeteer for obvious reasons. The second module is what we will use to export our scraped data to a JSON file. const puppeteer = require("puppeteer");
const fs = require("fs"); The next section is the start of our actual function. At the very beginning of this arrow function, we hook up to the Browserless API using the API key you obtained from the Browserless scrape() dashboard. const browser = await puppeteer.connect({
    browserWSEndpoint:
      "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
  }); Alternatively, you could replace this section of code with like so: puppeteer.launch const browser = await puppeteer.launch({ headless: false }); This is good to do in your dev environment to test out the script. That will save your Browserless resources and help with debugging if there is any issue with your code. One thing I wanted to point out was the property that is being passed in. This tells Puppeteer to launch Chrome on the screen so you can visually see how the code is interacting with the webpage. Pretty cool to watch, and extra useful when building out scripts! {headless: false} In the next section, we create a new browser page, navigate to Google Shopping's website, find the search bar, type in whatever we want to search for (in this case we are searching for a board game), click the search button, and finally waiting for the page to load. More specifically, waiting for the pagination at the bottom of the page to load. You'll see why in a minute. const page = await browser.newPage();
    await page.goto("https://shopping.google.com/");
    await page.type(
      "input[placeholder='What are you looking for?']",
      "board game"
    );
    await page.click("button[aria-label='Google Search']");
    await page.waitForSelector("#pnnext"); The following section is kind of a bonus. What if you wanted to navigate a couple of pages deep into the search results? That is what we accomplish in the next code block. Remember how we waited for the pagination to load in the last part of the code? In this section, we click on that "next page" button that we were waiting on. This will navigate our browser to the next page of board games. await page.click("#pnnext");
    await page.waitForSelector("#pnnext"); Alrighty, now to gather the data we want from the page! The following piece of code is what finds the product title and the price of the board game. I do want to add a little disclaimer here. When inspecting Google Shopping's search page, you'll notice their compiled code is a little convoluted. If these specific attributes aren't working for you, make sure to inspect the search results page yourself and find the HTML attributes that work for your scenario. This is where using that property can help. puppeteer.launch({ headless: false }) // Gather product title
    const title = await page.$$eval("div.sh-dgr__grid-result h4", (nodes) =>
      nodes.map((n) => n.innerText)
    );

    // Gather price
    const price = await page.$$eval(
      "div.sh-dgr__grid-result a.shntl div span span[aria-hidden='true'] span:nth-child(1)",
      (nodes) => nodes.map((n) => n.innerText)
    ); Ok, so once we've gotten our product titles and prices it's time to consolidate our data. For this I'm only going to take the first 10 results I gathered. // Consolidate product search data
    const googleShoppingSearchArray = title.slice(0, 10).map((value, index) => {
      return {
        title: title[index],
        price: price[index],
      };
    }); Lastly, we want to display or save our data. If you simply want to see what results get returned, you can . But let's say you want to export this data to a JSON file and maybe use that JSON to make a chart of the results. To save your data into a JSON file you'll use the code in our example. console.log(googleShoppingSearchArray); const jsonData = JSON.stringify(googleShoppingSearchArray, null, 2);
    fs.writeFileSync("googleShoppingSearchResults.json", jsonData); I added extra parameters to the method for formatting purposes. The last param is a parameter. stringify space And to finish off our script, we close the browser. await browser.close(); Below is a visual display of what our script is doing in the browser. As you can see, Puppeteer is a pretty cool API. Pair that with Browserless and you have a super scalable, web scraping, automation tool. Did this article spark an idea or thought? Let me know in the comments, or on Twitter at @tylerreicks. Happy coding ❤️! Also Published Here

Scraping Google Shopping Using Puppeteer and Browserless

Quality is critical when it comes to releasing any software, whether it’s in order to ensure an excellent experience of your product with minimal bugs, or even for security reasons, and it’s been well established that good code comes coupled with good tests. The hierarchy of tests as they relate to quality assurance are from unit tests through E2E testing, so just so we’re on the same page––I’ll briefly dive into each, and why the ultimate goal is to reach E2E testing of your product. Unit tests are at the bottom of the pyramid, as they’re just what they sound like, the smallest self-contained unit of testing. This means that you take the component or piece of code that you wrote, and test it as a standalone unit. This code is completely disconnected from anything else in the software stack and system it is going to live in, and the tests check if it performs as expected, as a standalone entity. Once we complete this basic testing of our code, we need to dive a little deeper, and test the code with more context of where it is actually going to exist in the stack, and this is where integration testing comes in. Integration testing is when we start integrating our code with other modules and services to see how it performs and functions in a more realistic ecosystem. While these can provide some context, they still need to be fast, and so they really don’t cover all scenarios, and don’t cover how users interact with your product at all. While some would argue that unit tests are the simplest unit of testing, sometimes you need to actually write mock modules to ensure the code actually works, whereas with integration tests you plug into actual existing modules. At this point you verify that your assumptions when mocking and injecting dependencies for unit testing do hold. So it’s a toss up - but both of these test types usually present fairly different challenges than end-to-end (E2E) testing. E2E testing is likely the most important testing you will need to perform on your product. It’s mission-critical to the company to ensure that the core business and value you are trying to provide the end-user is fulfilled. It involves simulating the end user as closely as possible, and ensuring the product works as expected––with many considerations in mind: usability, performance, user experience, and much more. E2E testing usually involves the impersonation of an end-user journey, such as logging in, integrating it with the user's existing environment, interacting with it and much more. Since there is so much involved with getting this right, and extracting realistic results, many companies choose to do this manually. Companies usually employ QA teams whose sole purpose is to spin up the product in testing environments, and use the product like a regular user. However, what the company will quickly discover is that they need to do this at a greater velocity and scale that doesn’t keep up, with repeated ticket back-and-forths between the QA and Engineering teams and this is a tedious and tiresome process that largely just frustrates engineers. So they decide to automate the tests, by creating a test automation team. At Diversion, as an early stage startup, we needed all our engineers focused on writing great code to reach our product release targets quickly. However, we knew we couldn’t compromise quality, so we went to work designing a solution for automating our E2E tests from day one. We’ll walk you through how we built and automated our E2E testing framework with Docker Compose, RPyC, and Puppeteer. Automate All the Tests It’s core to our philosophy that developers should be happy, and enjoy their day to day work. Yet, as engineering managers, one of the most common sources of pain for developers is testing which we can’t compromise on. Automation is a great way to remove human error from all processes, as well as streamline repeatable processes, and remove the pain of manual testing. We understood that if we invest in the technical design of our test automation, we would be able to offload this from our engineers. In addition, by automating our E2E we created greater satisfaction and ownership of our product by our engineers, who focus on building an excellent product, with built-in quality guardrails while minimizing friction. However, automating the end-to-end user experience of your product is quite challenging. We needed to ensure the testing covers all of the critical parts in the chain of the E2E user flow and the typical operations they would perform, such as: signing in and entering credentials in a web page, checking file system operations such as creating a file, writing to it and then moving it somewhere, make this happen on several machines in parallel, and enabling these machines to communicate with each other. On top of this, we needed to be able to trust the accuracy of the results. The Stack After some research and understanding of what we need to design and build, we chose , , and for our test automation suite. Docker Compose RPyC Puppeteer The tech stack for our E2E testing is completely independent of the stack we chose for our product of course (we’ll dive into this in a separate post), and it can run with any product that is able to run Docker and Docker Compose in their environment. We chose Docker Compose because the service requires several hosts to be spun up and communicate with each other. Docker Compose provides a convenient solution for this, as it sets up a for your application by default. single flat network This means that each container for a service joins a default network and is reachable by other containers on that network at a hostname identical to the container name. Docker Compose is also a very popular tool in the ecosystem, so it was easy to integrate and run it with other popular CI tools like Github Actions. Making the automation really easy to set up. RPyC enabled us to run the test scenario from a single script, as one host can control other hosts that participate in the test script. Much of our backend code is Python, so this was a sensible choice for us, as the entire team is comfortable with it. RPyC also enables us to do any random operations we need from within the testing scenario, such as executing any binary. Puppeteer is an automation tool for Chrome that enables you to take control of web pages, and is able to click buttons, fill in forms, and perform other web-based actions. When you start using the , the sign-in process takes you from the terminal to a webpage for authentication. It then uses a Proof Key for Code Exchange flow ( ) to fetch the credentials from the server. Puppeteer enabled us to emulate a user performing the web sign-in process. Diversion CLI RFC7636 Show Me the Code We launch three services, one for Puppeteer, and the other two are the mock users/participants in the test scenario (we create two machines for the same user). Next we dive into the different parts of this test scenario setup. # docker-compose.yml
version: '3'

services:
 cli-1:
   image: cli_user_image
   build:
     context: .
     dockerfile: Dockerfile.cli
   volumes:
     - ../:/repos/diversion
   environment:
     - AWS_REGION  # forward AWS credentials, for accessing AWS Secrets Manager
     - AWS_ACCESS_KEY_ID
     - AWS_SECRET_ACCESS_KEY
   command: >
     sh -c "REPOS_PATH=/repos/dv_repos &&
           /repos/diversion/system-tests/prepare_test_env.sh &&
           cd /repos/diversion/system-tests/tests/ &&
           su -c 'dv login' cliuser &&
           su -c 'rpyc_classic.py --host 0.0.0.0' cliuser"
 cli-2:
   image: cli_user_image
   volumes:
     - ../:/repos/diversion
   environment:
     ...  # same vars
   command: >
     sh -c "REPOS_PATH=/repos/dv_repos &&
           /repos/diversion/system-tests/prepare_test_env.sh &&
           cd /repos/diversion/system-tests/tests/ &&
           su -c 'dv login' cliuser &&
           REPOS_PATH=$$REPOS_PATH PLAYER2_RPYC=cli-2 su -c 'python3 -m unittest discover .' cliuser"
 puppeteer:
   build:
     context: .
     dockerfile: Dockerfile.puppeteer
   stdin_open: true
   cap_add:
     - SYS_ADMIN
   depends_on:
     - 'cli-1'
     - 'cli-2'
   volumes:
     - ../:/repos/diversion
   environment:
     ...  # same vars
   command: >
     sh -c 'cd /usr/app &&
           /repos/diversion/system-tests/wait_for_it.sh --strict --timeout=240 cli-2:5002 &&
           LOGIN_PW=$$(aws secretsmanager get-secret-value --secret-id test/e2e/login | jq -r .SecretString | jq -r .password) &&
           node --unhandled-rejections=strict ./loginscript.js $$(nc -w 1 cli-1 5001) "$$LOGIN_PW" &&
           /repos/diversion/system-tests/wait_for_it.sh --strict --timeout=120 cli-1:5002 &&
           node --unhandled-rejections=strict ./loginscript.js $$(nc -w 1 cli-2 5001) "$$LOGIN_PW"' The first CLI instance signs in, then opens an RPyC socket, and is then controlled by the test scenario on the other CLI instance. For signing in, our CLI calls on Linux machines with a preconfigured URL. xdg-open To automate it, we placed the following script at : /usr/bin/xdg-open #!/usr/bin/env bash
# xdg-open override
nohup sh -c "echo '""$1""' | nc -l 5001" >/dev/null 2>&1 &
nohup sh -c "echo ready | nc -l 5002" >/dev/null 2>&1 & When the CLI calls , this script publishes the sign-in URL on port 5001 using , then announces that the address is available via port 5002: xdg-open <url> netcat This is then used in the puppeteer service, as we see in the docker-compose file: it first waits for the URL to be available by calling (project on Github) on port 5002 of host cli-1, then reads the address that should be opened from port 5001 ( ) and passes it to . wait_for_it.sh $$(nc -w 1 cli-2 5001) loginscript.js The puppeteer login script is pretty straightforward, feeding the sign-in form with the credentials. This is the gist of it: // loginscript.js
const signInUrl = myArgs[0];
await page.goto(signInUrl);
try {
 await page.waitForSelector('input[name=username]', {timeout: 1000});
 await page.type('input[name=username]', 'testuser');
 await page.waitForSelector('input[name=password]');
 const pw = myArgs[1];
 await page.type('input[name=password]', pw);
 await page.click('input[name=signInSubmitButton]');
} catch (error) { // User is already signed in and all we're getting is a "Sign in as <user>" button
 await page.waitForSelector('input[type=Submit]');
 await page.click('input[type=Submit]');
} After both clients have signed in, it’s time to run the actual scenarios. Below is a snippet of the system tests. It starts by connecting to the other machine in , then runs a test involving two clients for a single user. setUpClass Note how uses the RPyC socket to execute CLI commands on the other machine: dv_runner_remote dv # test_system.py

class SystemTest(unittest.TestCase):
   TEST_REPO_NAME = 'test-repo'
   TEST_REPO_DIR = f'./{TEST_REPO_NAME}'

   @classmethod
   def setUpClass(cls) -> None:
       cls.player2 = rpyc.classic.connect(os.environ['PLAYER2_RPYC'])

   def test_create_move(self):
       dv_runner_local = DvRunner(subprocess.run)
       dv_runner_remote = DvRunner(self.player2.modules.subprocess.run)

       # clone the repo
       resp = dv_runner_local.run('clone', SystemTest.TEST_REPO_NAME)
       self.assertTrue(os.path.exists(f'{SystemTest.TEST_REPO_DIR}'),
                       f'stdout: {resp.stdout}, stderr: {resp.stderr}')

       # create a file, then move it to a new dir
       os.mkdir(f'{SystemTest.TEST_REPO_DIR}/test-dir/')
       with open(f'{SystemTest.TEST_REPO_DIR}/test-dir/test-file', 'w') as f:
           f.write('test-content')
       os.mkdir(f'{SystemTest.TEST_REPO_DIR}/test-dir2')
       shutil.move(f'{SystemTest.TEST_REPO_DIR}/test-dir/test-file',
                   f'{SystemTest.TEST_REPO_DIR}/test-dir2/test-file')

       # clone the repo on the other machine
       resp = dv_runner_remote.run('clone', SystemTest.TEST_REPO_NAME)
       self.assertTrue(self.player2.modules.os.path.exists(f'{SystemTest.TEST_REPO_DIR}'), resp.stdout)
       self.player2.modules.os.chdir(SystemTest.TEST_REPO_DIR)

       # ensure the file is in the right location
       self.assertTrue(self.player2.modules.os.path.exists(f'{SystemTest.TEST_REPO_DIR}/test-dir2/test-file'))

       # delete the repo
       player2_status = dv_runner_remote.run('status')
       repo_id = re.findall(r'(dv.repo.\S)', player2_status.stdout)[0]
       resp = dv_runner_remote.run('repo', '-d', repo_id) With this stack we were able to simulate E2E testing of our CLI with a web-based login, and this is applicable for any similar application. This general framework can be leveraged to solve many other use cases as well, for example if you need to perform an action from a CLI and verify it propagated in a webpage, synchronize actions across multiple machines from one central script, etc. This framework manages to reduce the daunting task of E2E tests to a nearly self-contained unit, very similar to unit testing. With all of the relevant services and components connected and communicating with each other, you can write these tests quickly and simply just like unit tests. This is a sweet spot for developers, and the testing realm they feel most comfortable with, as it is much more native to their workflows, and with the automation it is much more scalable than manual QA. The E2E testing also provides as close a replication of the product journey as possible. This ensures that our developer experience and product quality are always at the core of our engineering. Special thanks to Alon Regev and Avi Mastov for their feedback on this article. Previously published here.

How We Automated our End-to-End Testing from the First Line of Code 

In this article, we are going to completely automate the process of searching for a product on Amazon. Specifically, we are going to search for office chairs and scrape the Amazon results page for the product name and price. Below is a visual of what we are going to be targeting on the page. Here's the kicker, all we are going to use is one script and under 50 lines of code. How are we going to do this you may ask? The magic of JavaScript. Furthermore, a node library called and a web service called . If you don't know much about these tools I'd highly recommend you check them out further. They offer a lot of neat capabilities! We will go over some of them in this article. Puppeteer Browserless For further info on Puppeteer, I'd highly recommend watching in 2018. Also, but if you're hungry for more Browserless after reading this article checkout . Joel is the CEO of Browserless and offers great tips and tricks to use in your Browserless and Puppeteer environments. Eric Bidelman's talk at the Google I/O conference Joel Griffith's webinar about cool ways to optimize web scraping Why Puppeteer? Next, let's very quickly explain why we are using Puppeteer to accomplish our task of automating an Amazon search. First, Puppeteer provides a great high-level API to control Chrome (both headless and non-headless). Puppeteer is maintained by the Chrome DevTools team. These engineers know what they are doing. Lastly, most things you can do manually in the browser can be accomplished using Puppeteer. This will allow us to easily manipulate our script to perform the actions we want in the browser. Initial setup Alrighty, let's get to it! We are going to start with the initial setup. Luckily for us, there aren't many dependencies we need to install. There's only one... Puppeteer. npm i puppeteer Once you've run that command you are good to go! 👍 Browserless Technically you are all set to start coding, but there is another tool I want to use in tandem with Puppeteer. That tool is a web service called Browserless. Why am I going to use this tool you're wondering? Let's get into that. The script that we are going to be coding is one of many things that can be done with Puppeteer and headless Chrome. Let's say you want to run this script in one of your production applications, but you don't want to have to add Puppeteer as an extra dependency. What can you do? This is one scenario where Browserless comes to the rescue. With Browserless there is no need to install extra packages or dependencies, and it's compatible with Puppeteer. All you need to do is plug and play (We'll get more into the details of setting up Browserless below). This takes a huge load off of your application. You can test and run Puppeteer scripts in your dev environments and meanwhile run Browserless in your prod environments without having to worry about the extra baggage. Talk about a performance boost! Another area where Browserless shines is in remote development environments (i.e. , gitpod, codepen, codespaces, etc.). Since there is no need to install any software you can easily hook up to the Browserless API and run your Puppeteer scripts in all your remote environments. repl.it Before we get to the actual script, let's look at how to set up Browserless real quick. For starters, you'll need to set up an account. Once you get your account set up, you'll be directed to your Browserless dashboard. Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies. Keep your API key handy as we'll use that once we start writing our script. Other than that we are ready to start coding! The code Ok now for the exciting part! I like to show the final product at the beginning and then dissect each piece from top to bottom. Here is our script: const puppeteer = require("puppeteer");
const fs = require("fs");

const scrape = async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint:
      "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
  });

  const page = await browser.newPage();
  await page.goto("https://www.amazon.com");
  await page.type("#twotabsearchtextbox", "office chair");
  await page.click("#nav-search-submit-button");
  await page.waitForSelector(".s-pagination-next");

  // Go to next results page
  await page.click(".s-pagination-next");
  await page.waitForSelector(".s-pagination-next");

  // Gather product title
  const title = await page.$$eval("h2 span.a-color-base", (nodes) =>
    nodes.map((n) => n.innerText)
  );

  // Gather price
  const price = await page.$$eval(
    "[data-component-type='s-search-result'] span.a-price[data-a-color='base'] span.a-offscreen",
    (nodes) => nodes.map((n) => n.innerText)
  );

  // Consolidate product search data
  const amazonSearchArray = title.slice(0, 5).map((value, index) => {
    return {
      title: title[index],
      price: price[index],
    };
  });

  const jsonData = JSON.stringify(amazonSearchArray, null, 2);
  fs.writeFileSync("amazonSearchResults.json", jsonData);
  await browser.close();
};
scrape(); Okie doke, let's dive into this. The first 2 lines are pulling in JavaScript modules. We'll want to pull in Puppeteer for obvious reasons. The second module is what we will use to export our scraped data to a JSON file. const puppeteer = require("puppeteer");
const fs = require("fs"); The next section is the start of our actual function. At the very beginning of this arrow function, we hook up to the Browserless API using the API key you obtained from the Browserless dashboard. scrape() const browser = await puppeteer.connect({
    browserWSEndpoint:
      "wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]",
  }); Alternatively, you could replace this section of code with like so: puppeteer.launch const browser = await puppeteer.launch({ headless: false }); This is good to do in your dev environment to test out the script. That will save your Browserless resources and help with debugging if there is any issue with your code. One thing I wanted to point out was the property that is being passed in. This tells Puppeteer to launch Chrome on the screen so you can visually see how the code is interacting with the webpage. Pretty cool to watch, and extra useful when building out scripts! {headless: false} In the next section, we create a new browser page, navigate to Amazon's website, find the search bar, type in whatever we want to search for (in this case we are searching for an office chair), click the search button, and finally waiting for the page to load. More specifically, waiting for the pagination at the bottom of the page to load. You'll see why in a minute. const page = await browser.newPage();
  await page.goto("https://www.amazon.com");
  await page.type("#twotabsearchtextbox", "office chair");
  await page.click("#nav-search-submit-button");
  await page.waitForSelector(".s-pagination-next"); This following section is kind of a bonus. What if you wanted to navigate a couple of pages deep into the search results? That is what we accomplish in the next code block. Remember how we waited for the pagination to load in the last part of the code? In this section, we click on that "next page" button that we were waiting on. This will navigate our browser to the next page of office chairs. await page.click(".s-pagination-next");
  await page.waitForSelector(".s-pagination-next"); Alrighty, now to gather the data we want from the page! The following piece of code is what finds the product title and the price of the office chair. I do want to add a little disclaimer here. When inspecting Amazon's search page, you'll notice their compiled code is a little convoluted. If these specific attributes aren't working for you, make sure to inspect the search results page yourself and find the HTML attributes that work for your scenario. This is where using that property can help. puppeteer.launch({ headless: false }) // Gather product title
  const title = await page.$$eval("h2 span.a-color-base", (nodes) =>
    nodes.map((n) => n.innerText)
  );
  
  // Gather price
  const price = await page.$$eval(
    "[data-component-type='s-search-result'] span.a-price[data-a-color='base'] span.a-offscreen",
    (nodes) => nodes.map((n) => n.innerText)
  ); Ok, so once we've gotten our product titles and prices it's time to consolidate our data. For this I'm only going to take the first 5 results I gathered. // Consolidate product search data
  const amazonSearchArray = title.slice(0, 5).map((value, index) => {
    return {
      title: title[index],
      price: price[index],
    };
  }); Lastly, we want to display or save our data. If you simply want to see what results get returned, you can . But let's say you want to export this data to a JSON file and maybe use that JSON to make a chart of the results. To save your data into a JSON file you'll use the code in our example. console.log(amazonSearchArray); const jsonData = JSON.stringify(amazonSearchArray, null, 2);
  fs.writeFileSync("amazonSearchResults.json", jsonData); I added extra parameters to the method for formatting purposes. The last param is a parameter. stringify space And to finish off our script, we close the browser. await browser.close(); Below is a visual display of what our script is doing in the browser. As you can see, Puppeteer is a pretty cool API. Pair that with Browserless and you have a super scalable, web scraping, automation tool. Did this article spark an idea or thought? Let me know in the comments, or on Twitter at @tylerreicks. Happy coding ❤️! First Published here

Scraping Amazon using Puppeteer and Browserless

End-to-end testing is a Quality Assurance technique that involves exercising and validating an application's workflow from start to finish. This technique aims to reproduce real user scenarios and validate expectations over application output. Unfortunately, end-to-end testing has problems that make tests difficult to maintain: The testing process must generally assume that the system is in a . This might require seeding or wiping test data before/after the test is run. consistent starting state When are made, there may need to be changes to the test sequence to handle the application update. application changes Even when nothing changes, sometimes tests will fail anyway. Such tests are denoted as . flaky Over the last 18 months, I’ve been bootstrapping : a SaaS tool that allows non-technical QA personnel to implement and maintain end-to-end tests using a no-code recording UI. I’ve accrued a toolkit of strategies to fix tests, and automatically heal the test sequence when there are . reflow.io flaky application changes This article is a summary of those techniques, and how to apply them into your test suite, regardless of whether you use reflow or not. Why does this matter? Time is a precious resource for all product teams. It must be protected and spent wisely. . Flaky tests cost time Every time a test flakes, generally someone will investigate whether the failure is a true-failure, or a false-positive. The time cost of this can become very significant. A test may have downstream dependencies that cannot run until the test is re-run, or fixed. This can often escalate the time cost. The failed test may place your system into a non-deterministic state, which may then require manual effort to fix. Emotional Damage The very existence of an automated tests implies that someone cared enough to automate away some manual effort. When that manual effort returns through a test flake, there is an emotional toll. If the flake cannot be removed, the team will be very aware that they are stuck maintaining them in perpetuity. Automation Trust If the tests are flaky, the tests will not be trusted. If the tests are not trusted, they will become gradually deprecated and eventually removed. If the product team does not replace these tests, the codebase will lose test coverage, and may become less trusted. If the codebase is less trusted, productivity will suffer. Mitigations Strategy 1: Generic pre-action stability The most common flake reason that we see is caused by interacting with the page too quickly. Page elements often get rendered before they can be successfully interacted with. This means just checking for their existence is often not enough. There are a few generic actions that can be done to increase the likelihood that the page is fully loaded before it is interacted with. In playwright, these stability method are available at the api. waitForLoadState await page.waitForLoadState('domcontentloaded', { timeout: 15000 });
await page.waitForLoadState('load', { timeout: 30000 });
await page.waitForLoadState('networkidle', { timeout: 5000 }); These 3 load events are: - wait for the DOMContentLoaded event to be fired. This fires when the initial page document is loaded and parsed. For SPAs, stylesheets, images and javacript will generally not be loaded when this event fires. Hence it is usually preferable to rely on the event instead. domcontentloaded load - wait for the load event to be fired. This event fires when all markup, stylesheet, javascript and all static assets like images and audio have been loaded. For many SPAs, which dynamically load data after the page renders its initial document, the event may not be enough. load load - wait until there are no network connections for at least 500 ms. This is a useful event to hook into for SPAs which load data after they have their initial render. However, it may either be too early or too late for page interaction. networkidle In reflow, there is an optional 4th event: . This event is fired when the page: screenshotstable Has not changed within 5s. Looks like the page in its most recent successful test. We introduced this event as most applications, when loading, either show a loading animation or continuously re-render whilst they load data. It introduces a minor delay to test actions, but we believe the decrease in flakiness makes it worthwhile. Similarly, for a deterministic end-to-end test sequence, the page almost always looks the same between test runs. Hence we can save a screenshot in S3 to track how the page looks like before an action execution, and compare it during the test execution to the previous successful run of the same device/browser/operating system. As always, a timeout needs to be set to avoid this waiting forever when the application has changed, or is not expected to stop rendering. In reflow, these timeouts are automatically configured based on how the page behaved when it was originally recorded or, if it exists, the most recent successful test execution. Strategy 2: Intelligent Waiting If an action expects to be on a given page, we can wait until the navigation to that page is complete with . This can be important, as multiple events may fire during a navigation sequence: hence a will not always be enough. page.waitForNavigation load page.waitForLoadState If an action affects a specific element, we can do further action-specific steps. For example: Wait for the element to be Attached to the DOM. Wait for the element to be Visible. Wait for the element to be Stable, as in not animating or completed animation Wait for the element to be able to receive events If the element is a clickable element, wait for the element to be . enabled If the element is a text-entry element, wait for the element to be . editable In playwright, these . pre-action stability statements are automatically applied based on the interaction type In reflow, stability is further enhanced with: Waiting for the element to look the same as it did in the last successful run. I.e. continuously take screenshots of the element, and wait until they look the same (or a timeout). Apply a specific timeout to the element based on how long it took for the last DOM attribute update to be applied during the last successful run. E.g. if the button turned green via a attribute change after 7s, wait at least 7s for the element to get DOM attribute changes, before further timeouts. class Strategy 3: Pick Good Selectors When an application changes, the locators used to identify elements often also change. These locators are generally known as . selectors A smart development team will combat this with a strongly consistent element attribute, such as . E.g. data-test-id <button
  data-test-id={`test-actions-${testId}`}
/> To click such a button we can use the attribute to locate the element. data-test-id await page.click(`[data-test-id="test-actions-${testId}"]`); Other good options, for scenarios where adding a specific element selector is undesirable, are: Selector Description placeholder="..." Placeholders are often unique to the element [aria-label="..."] An attribute used by assistive technologies to help identify an element (e.g. for a screen reader) img[alt="..."] An attribute used to denote alternate text for an image, if the image cannot be displayed role="..." A role attribute is used to add semantic meaning to an element for assistive technologies (e.g. screen readers) input[type="..." An input often has a unique/unchanging type that references it for short forms nodeName If an element node type is only used once, the (e.g. [ ; ; ]) is often a good choice nodeName a input button #... Elements are often assigned unique attributes to aid in locating them by javascript. id Reflow will pick up such selectors automatically during test recording, and score them by uniqueness and type. It will also: Combine parents selectors with their children to reduce ambiguity. E.g. . [data-test-id="foo"] >> [data-test-id="bar"] Calculate all the possible sets of selectors, rank them, and inspect the page for the most likely element based on how close they are to the most recent successful run's element. Compare all page elements made from partial selector matches to a screenshot of the most recent successful run's element, to enable auto-healing the locator when it cannot be found or is ambiguous. We call this a Visual Selector Strategy 4: "Wait Until" checkpoints This strategy relies on writing application-specific logic to be built to ensure the application is in a specific state. For instance, if a page element represents a calculation, and we want to wait for that calculation to be completed before moving on, a developer could write an assertion like: await page.waitForSelector('[aria-label="calculation"] >> text=29.76') In reflow, such an assertion can be made with as long a timeout as desired. For instance, in most of reflow's internal test suite (powered by itself) it starts by: Making a new reflow test Navigating to the recording UI of that test Waiting until the recording UI looks like a pre-recorded screenshot, up to 5 minutes. The on step [3] removes the flakiness of both server startup times, and DNS propagation. It doesn't require any code, and works very reliably. When the starting URL has any visual changes, reflow will fail the test and offer an option to quickly replace it with the new visual snapshot. Visual Assertion auto-healing Strategy 5: Zero-dependency Data Seeding This strategy helps combat the problem "The testing process must generally assume that the system is in a consistent starting state", by breaking out an initial test suite that resets application data. In reflow, this is done by adding a user to a team, accepting the invite, then removing them. After this sequence all the user's end-to-end tests are associated with the team that they just left, and they are in an empty team. In other applications, this might be done via invoking a REST endpoint directly in the test suite to reset application data. E.g. invoking a REST handler that resets the database: await page.request.post(`${event.variables.url}/reset/scenario/empty?token=${event.variables.secret}`); Conclusion: Is there a way to eliminate all flaky tests forever? No. An engineering team can be smart, and drastically reduce the amount of time spent maintaining end-to-end tests by applying these strategies. However that time will never go to zero whilst an application is being actively developed. Test Automation helps ensure that QA effort is placed on the boundaries of new feature development, rather than endlessly covering existing features on every application change. Commercial low-code automation tools like can allow product teams to work more by reducing the . They are not a magic pill, but can be a very valuable tool for teams where QA and development personnel want to work closer together on test automation, regardless of coding ability. reflow.io effectively cost of automation

How to Fix Flaky End-to-End Tests with Playwright and Reflow

Web scraping: Web Scraping is a way to collect all sorts of publicly available data like prices, text, images, contact information and much more from the world-wide-web. This can be useful when trying to collect data that might take a person a lot of time to collect and organize manually. Some of the most useful use cases of web scraping include: Scraping product prices from e-commerce websites such as Amazon, eBay or Alibaba. Scraping social media posts, likes, comments, followers or bios. Scraping contacts from websites like Yellowpages or Linkedin. Puppeteer While there are a few different libraries for scraping the web with Node.js, in this tutorial, i'll be using the puppeteer library. Puppeteer is a popular and easy-to-use npm package used for web automation and web scraping purposes. Some of puppeteer's most useful features include: Being able to extract a scraped element's text content. Being able to interact with a webpage by filling out forms, clicking on buttons or running searches inside a search bar. Being able to scrape and download images from the web. Being able to see the web scraping in progress using headless mode. You can read more about puppeteer here Installation For this tutorial, I will suppose you already have and installed, as well as a and file. npm node_modules package.json package-lock.json If you don't, here's a great guide on how to do so: Setup To install puppeteer, run one of the following commands in your project's terminal: npm i puppeteer Or yarn add puppeteer Once puppeteer is installed, it will appear as a directory inside your node_modules. Let's make a simple web scraping script in Node.js The web scraping script will get the first synonym of "smart" from the web thesaurus by: Getting the HTML contents of the web thesaurus' webpage. Finding the element that we want to scrape through it's selector. Displaying the text contents of the scraped element. Let's get started! Before scraping, and then extracting this element's text through it's selector in Node.js, we need to setup a few things first: Create or open an empty javascript file, you can name it whatever you want, but I'll name mine " " for this tutorial. Then, require puppeteer on the first line and create the async function inside which we will be writing our web scraping code: index.js index.js const puppeteer = require('puppeteer')

async function scrape() {
}
scrape() Next, initiate a new browser instance and define the "page" variable, which is going to be used for navigating to webpages and scraping elements within a webpage's HTML contents: index.js const puppeteer = require('puppeteer')

async function scrape() {
   const browser = await puppeteer.launch({})
   const page = await browser.newPage()
}
scrape() Scraping the first synonym of "smart" To locate and copy the selector of the first synonym of "smart", which is what we're going to use to locate the synonym inside of the web thesaurus' webpage, first go to the , right click on the first synonym and click on "inspect". This will make this webpage's DOM pop-up at the right of your screen: web thesaurus' synonyms of "smart" Next, right-click on the highlighted HTML element containing the first synonym and click on "copy selector": Finally, to navigate to the web thesaurus, scrape and display the first synonym of "smart" through the selector we copied earlier: First, make the "page" variable navigate to inside the newly created browser instance. https://www.thesaurus.com/browse/smart Next, we define the "element" variable by making the page wait for our desired element's selector to appear in the webpage's DOM. The text content of the element is then extracted using the evaluate() function, and displayed inside the "text" variable. Finally, we close the browser instance. index.js const puppeteer = require('puppeteer')

async function scrape() {
   const browser = await puppeteer.launch({})
   const page = await browser.newPage()
   
   await page.goto('https://www.thesaurus.com/browse/smart')
   var element = await page.waitFor("#meanings > div.css-ixatld.e15rdun50 > ul > li:nth-child(1) > a")
   var text = await page.evaluate(element => element.textContent, element)
   console.log(text)
   browser.close()
}
scrape() Time to test Now if you run your index.js script using "node index.js", you will see that it has displayed the first synonym of the word "smart": Scraping the top 5 synonyms of smart We can implement the same code to scrape the top 5 synonyms of smart instead of 1: index.js const puppeteer = require('puppeteer')

async function scrape() {
   const browser = await puppeteer.launch({})
   const page = await browser.newPage()
   
   await page.goto('https://www.thesaurus.com/browse/smart')
   for(i = 1; i < 6; i++){
    var element = await page.waitFor("#meanings > div.css-ixatld.e15rdun50 > ul > li:nth-child(" + i + ") > a")
    var text = await page.evaluate(element => element.textContent, element)
    console.log(text)
   }
   browser.close()
}
scrape() The " " variable will be: " " on the first iteration, " " on the second, and so on until it reaches the last iteration where the " " variable will be " ". element #meanings > div.css-ixatld.e15rdun50 > ul > li:nth-child(1) > a #meanings > div.css-ixatld.e15rdun50 > ul > li:nth-child(2) > a element #meanings > div.css-ixatld.e15rdun50 > ul > li:nth-child(5) > a As you can see, the only thing that is altered in the "element" variable throughout the iterations is the "li:nth-child()" value. This is because in our case, the elements that we are trying to scrape are all "li" elements inside a "ul" element, so we can easily scrape them in order by increasing the value inside "li:nth-child()": li:nth-child(1) for the first synonym. li:nth-child(2) for the second synonym. li:nth-child(3) for the third synonym. li:nth-child(4) for the fourth synonym. And li:nth-child(5) for the fifth synonym. Final notes While web scraping has many advantages like: Saving time on manually collecting data. Being able to programmatically aggregate pieces of data scraped from the web. Creating a dataset of data that might be useful for machine learning, data visualization or data analytics purposes. It also has 2 disadvantages: Some websites don't allow for scraping their data, one popular example is craigslist. Some people consider it to be a gray area since some use cases of web scraping practice user or entity data collection and storage. Hopefully, this article gave you some insight into web scraping in Node.js, it's practical applications, pros and cons, and how to extract specific elements and their text contents from webpages using the puppeteer library. Also Published Here

Web Scraping Using Node.js

Getting browser automation scripts to do exactly what you want can feel like a never-ending battle between you and your code. I still remember vividly when I was up late one night years ago writing a Selenium script for a change I was making. I was banging my head against the table as I was running out of Selenium incantations to cast on my test to make it work. Fast forward to today, I’m writing up this guide on a few tricks I wish I knew at the time, to quickly craft reliable E2E tests, instead of crying over error logs and wrangling clueless automation scripts. Use a Recorder A recorder is a tool that can record the browser actions you take, and turn those actions into a fully-working automation script. So you can start coding out your test, by simply stepping through your user flow as a user would. I know there are people who might be skeptical, but hear me out. Just as automation frameworks have dramatically improved over time, test recorders have gotten some serious love over time as well. I find test recorders pretty useful for a few reasons: - There’s no reason to be hunting for selectors yourself, when you can automate the work away. The best recorders use selector logic that will give you stable selectors, rather than trying to enumerate every div and class name on the way to your element. (We’ll also talk more about selectors later!) No more hunting for selectors - Just like selectors, there’s no need to manually type out every “click” “fill” and “waitFor” command, when you can get all of that for free by just simply walking through your web pages. No more manually writing commands - While recorders won’t generate perfect code, they will have best-practices built-in. Instead of needing to worry about using “fill” instead of “type”, or remembering how to await for text in Puppeteer, you can just let the recorder generate code that’s aligned with best practices, so you can focus on the rest of your work. Best practices are built-in Ultimately, using a test recorder can give you a head start, save you from doing the boring parts of testing, and let you focus on the harder parts of perfecting a test automation script. Here’s a few recorders that can help you get started: : A Chrome/Firefox Extension that generates Cypress/Playwright/Puppeteer scripts (Disclaimer: I am one of the authors, but only because I wanted to build the most accurate recorder available) DeploySentinel Recorder : A CLI tool included in Playwright. Playwright Codegen : A recorder built into Chrome for Puppeteer. Puppeteer Chrome DevTool Recorder : A Chrome Extension that can automate some simpler actions. Headless Recorder Use Stable Selectors As dynamic CSS classes and elements are the norm today, it can be difficult to try to pick selectors that are robust to a rapidly changing code base, or even a rapidly changing web page due to async logic. Especially if your project uses something like Tailwinds, styled-components, or even a JS framework like React or Vue that use async logic to render elements, you might be struggling to figure out how to target the right element reliably. First Solution: Test IDs The best way to combat these issues is to explicitly introduce a stable selector to elements you’re looking to test, and nothing beats stability like introducing test IDs to your application. The concept is very simple: for elements you need to interact with, append a attribute to the HTML element. In your automation scripts, you can easily target the element with data-test-id=”my-element” await page.click('[data-test-id="my-element"]'); Now you’ll never need to worry about your selectors breaking the next time your team decides to change button colors or re-builds the application with a new minified class name. However, this requires adding new attributes to elements you need to target. Depending on how open the application owners are to adding these new attributes, it may be difficult to rely on them. If test IDs aren’t possible, I’d fall back to the next best solution. Second Solution: Accessibility Selectors Luckily with accessibility becoming a higher priority for web applications, more and more critical elements that need to be interacted with might already have a machine-friendly label attached to it. Usually you’ll see attributes such as , , or for elements you want to interact with. Those attributes tend to be more stable than CSS classes and can serve as a good stop-gap measure until you’re able to implement test IDs for the elements you need to test. aria-label alt title A script that utilizes these attributes might look like await page.click('[alt="Main Logo"]'); Last Solution: Text Content Selectors If you’re testing an application that might not have all the accessibility selectors built out yet, and haven’t had time to implement test IDs, the last solution you can look towards is targeting elements by text content. At first glance, it might sound like an incredibly fragile proposition. Indeed it can be for certain elements, but for others, it may be the best stable solution available. Can you remember the last time your team updated the “Sign In” button text on your application? For elements with non-dynamic text content, usually buttons or input placeholders, text content can be a fairly reliable way to target elements. Luckily in Playwright, it’s incredibly easy to target elements by text like so: page.click('text=Sign In'); In Puppeteer, you’ll need to dip into XPaths to target elements by text: await page.waitForXPath('//*[contains(., "Sign In")]');
const [element] = await page.$x('//*[contains(., "Sign In")]');
await element.click(); Automate Selector Picking These best practices of stable selectors mentioned here and more are already built-in to selector picking logic. So you don’t have to hunt for a specific test ID or accessibility selector. DeploySentinel Recorder’s Flip On the Debug Features If you’re banging your head against a test script trying to figure out why it isn’t working, whipping out the debugging modes is probably the fastest way to find out why your script isn’t doing what you want. Playwright’s Inspector and Trace Viewer With Playwright, it’s incredibly easy to append in front of your script to pull up , where it’ll be able to step through everything Playwright is doing in great detail during the test. If there’s a step you have issues with, you can add to pause the test run so you can inspect the page at that point in time. PWDEBUG=1 Playwright Inspector await page.pause() If you’re executing the script in a remote environment, you can take advantage of Playwright’s which records detailed logs and DOM snapshots before and after every action. Trace Viewer If you’re using DeploySentinel to run your test - Playwright traces are captured by default and viewable at any time to debug test runs. Headed Mode and Slow Mo In general, you can also enable headed mode with slow-motion enabled to visually see what your script is doing. Both Playwright and Puppeteer support this with just two extra lines of code. See the docs for and here. Playwright Puppeteer The will always have these two options commented out but inserted as part of every script generated to make it easy to debug locally. DeploySentinel Recorder Enable Chrome Dev Tools Lastly, if there’s an issue that requires you to look at network requests or browser logs, you can have Playwright and Puppeteer open the Chrome dev tools panel on browser launch so all logs and network requests are captured from the start automatically for you. See the here or the dev tools section of . Playwright docs Puppeteer’s debugging docs You’re All Set! I hope this list of tips helps you out in creating testing scripts for Puppeteer or Playwright. If you’re looking to run tests easily and affordably - you can , with many of the above best practices included for you with no extra effort. try DeploySentinel for free First Published here

Write your own E2E Test Scripts for Playwright & Puppeteer

Stories, News & Mentions about PAGE2API

Page2api’s Stories on HackerNoon