Page2API is the Ultimate Web Scraping API

Page2API

How to Use NextJS, TypeScript, Tailwind, and OneEntry CMS to Build an eCommerce Store 🛒👨‍💻

কিভাবে NextJS, TypeScript, Tailwind, এবং OneEntry CMS একটি ইকমার্স স্টোর তৈরি করতে ব্যবহার করবেন 🛒👨‍💻

So verwenden Sie NextJS, TypeScript, Tailwind und OneEntry CMS zum Aufbau eines E-Commerce-Shops 🛒👨‍💻

Cómo utilizar NextJS, TypeScript, Tailwind y OneEntry CMS para crear una tienda de comercio electrónico 🛒👨‍💻

Comment utiliser NextJS, TypeScript, Tailwind et OneEntry CMS pour créer une boutique de commerce électronique 🛒👨‍💻

ईकॉमर्स स्टोर बनाने के लिए नेक्स्टजेएस, टाइपस्क्रिप्ट, टेलविंड और वनएंट्री सीएमएस का उपयोग कैसे करें 🛒👨‍💻

NextJS、TypeScript、Tailwind、OneEntry CMS を使用して e コマース ストアを構築する方法 🛒👨‍💻

NextJS, TypeScript, Tailwind 및 OneEntry CMS를 사용하여 전자상거래 상점을 구축하는 방법 🛒👨‍💻

Como usar NextJS, TypeScript, Tailwind e OneEntry CMS para construir uma loja de comércio eletrônico 🛒👨‍💻

Как использовать CMS NextJS, TypeScript, Tailwind и OneEntry для создания магазина электронной коммерции 🛒👨‍💻

Bir e-Ticaret Mağazası Oluşturmak için NextJS, TypeScript, Tailwind ve OneEntry CMS Nasıl Kullanılır 🛒👨‍💻

Cách sử dụng NextJS, TypeScript, Tailwind và OneEntry CMS để xây dựng cửa hàng thương mại điện tử 🛒👨‍💻

如何使用 NextJS、TypeScript、Tailwind 和 OneEntry CMS 构建电子商务商店 🛒👨‍💻

Mastering Dynamic Web Scraping

ডায়নামিক ওয়েব স্ক্র্যাপিং আয়ত্ত করা

Dynamisches Web Scraping beherrschen

Dominar el raspado web dinámico

Maîtriser le Web Scraping dynamique

गतिशील वेब स्क्रैपिंग में निपुणता

動的ウェブスクレイピングをマスターする

동적 웹 스크래핑 마스터하기

Dominando o Web Scraping Dinâmico

Освоение динамического парсинга веб-страниц

Dinamik Web Scraping'de Uzmanlaşmak

Làm chủ việc quét web động

掌握动态网页抓取

Are you a QA tester eager to dive into performance testing without the need for extensive programming expertise? In this article, we'll explore an approachable way for non-programmers to conduct sort of load testing on cloud apps APIs using . Load testing without the need for complex coding - discover how even regular QA testers can use Python to find serious bugs and uncover potential performance bottlenecks. Python Performance testing is a critical aspect of ensuring your applications can handle real-world demands. I will try to explain my approach and Python scripts designed for load testing a cloud service managing browsers. Imagine a cloud service responsible for managing browser profiles (browsers for web scrapping). Users interact via API with the service to create, start, stop, delete, etc profiles. My Python script simulates this scenario, applying load to the cloud service by repeatedly performing these actions. Load Testing Scenario #  Dependencies
import asyncio
import httpx

#  Configuration
API_HOST = 'https://cloud.io'
API_KEY = 'qatest'
API_HEADERS = {
    "x-cloud-api-token": API_KEY,
    "Content-Type": "application/json"
}
CYCLES_COUNT = 3

#  Browser profile configuration
data_start = {
    "proxy": "http://127.0.0.1:8080",
    "browser_settings": {"inactive_kill_timeout": 120}
} Asynchronous functions: The heart of my load tests The function retrieves existing browser profiles from the service, simulating a scenario where users request information. Fetching Browser Profiles: get_profiles async def get_profiles(cl: httpx.AsyncClient):
    resp = await cl.get(f'{API_HOST}/profiles', params={'page_len': 10, 'page': 0}, headers=API_HEADERS)
    return resp.json() The script initiates cycles, each involving starting browser profiles asynchronously. This emulates a scenario where a user creates multiple browsers concurrently and uses browser profiles. Starting Browsers: async def start_profile(cl: httpx.AsyncClient, uuid):
    resp = await cl.post(f'{API_HOST}/profiles/{id}/start', json=data_start, headers=API_HEADERS)
    if error := resp.json().get('error'):
        print(f'Profile {id} not started with error {error}') After starting profiles, the script fetches active profiles and stops them, followed by deleting all profiles. This load scenario assesses the cloud service's responsiveness to dynamic changes and its efficiency in resource cleanup. Stopping browsers and deleting profiles: async def stop_profile(cl: httpx.AsyncClient, uuid):
    resp = await cl.post(f'{API_HOST}/profiles/{id}/stop', headers=API_HEADERS)
    if error := resp.json().get('error'):
        print(f'Profile {id} not stopped with error {error}')

async def delete_profile(cl: httpx.AsyncClient, uuid):
    resp = await cl.delete(f'{API_HOST}/profiles/{id}', headers=API_HEADERS)
    if error := resp.json().get('error'):
        print(f'Profile {id} not stopped with error {error}') The script concludes by checking and reporting on active connections. This step is crucial for understanding the load impact and identifying potential issues related to monitoring. Monitoring connections: understanding load impact for conn in cl._transport._pool.connections:
    if conn._connection._state.value != 1:
        continue

    print(f'Connection in progress: {conn}') Load testing in action The main function orchestrates the load testing cycles, iterating through profiles and executing asynchronous tasks. Each cycle represents a simulated user interaction, creating, using, and deleting browser profiles. async def main():
    async with httpx.AsyncClient(timeout=httpx.Timeout(timeout=300)) as cl:
        for _ in range(CYCLES_COUNT):
            profiles = await get_profiles(cl)
            start_tasks = [asyncio.create_task(start_profile(cl, profile['id'])) for profile in profiles]
            await asyncio.gather(*start_tasks)

            active_browsers = await get_active_profiles(cl)
            stop_tasks = [asyncio.create_task(stop_profile(cl, active_browser['id'])) for active_browser in active_browsers['data']]
            await asyncio.gather(*stop_tasks)

            profiles = await get_profiles(cl)
            del_tasks = [asyncio.create_task(delete_profile(cl, profile['id'])) for profile in profiles]
            await asyncio.gather(*del_tasks)

# Monitor active connections for insights into load impact Tailoring load tests to your needs This script shows a foundation for QAs to tailor load-testing scenarios to their applications. By customizing the number of cycles, adjusting user interactions, and modifying the script to fit specific API endpoints, testers can gain valuable insights into their application's performance under different loads. Here, you'll need essential monitoring tools to gain info on server states, assess the server load, and track resource utilization and logs. Utilize tools like Grafana, Kibana, Prometheus, etc, for comprehensive monitoring. Additionally, keep a close eye on the responses your script receives, ensuring a thorough evaluation of your application's performance. This approach is invaluable in your effective load testing and performance analysis. Additionally, for a more realistic load simulation, consider opening specific pages in your browser. While I personally used a start page in my browsers, you can also explore options like Pyppeteer or Playwright to open multiple tabs and navigate through various pages. This approach enhances the authenticity of your load-testing scenario, closely resembling user interactions with your application. # Attempt to connect to the browser using the provided profile URL
try:
    browser = await connect({'browserWSEndpoint': browser_url, 'defaultViewport': None})
except Exception as e:
    # Handle connection errors and print a message
    print(f'Error occurred when connecting to the browser: {str(e)}')
    return

# Create a new page in the connected browser
page = await browser.newPage()

# Introduce a brief delay to ensure the page is ready
await asyncio.sleep(2)

# Set the viewport dimensions for the page
width, height = 1920, 1080
await page.setViewport({'width': width, 'height': height})

# Try to navigate to a specific URL 
try:
    await page.goto('https://{your_website}')
    # Wait for 10 seconds to simulate user interaction
    await page.waitFor(10000)
    # Introduce another delay for additional stability
    await asyncio.sleep(5)
except pyppeteer.errors.PageError as e:
    # Handle page navigation errors and print a message
    print(f'Error occurred during page navigation: {str(e)}')

# Attempt to take a screenshot of the page
try:
    await page.screenshot(path='screen.png', fullPage=True)
    # Print a success message if the screenshot is captured successfully
    print('Screenshot taken successfully.')
except Exception as e:
    # Handle screenshot capture errors and print a message
    print(f'Error occurred during taking a screenshot: {str(e)}') Python's asynchronous capabilities, coupled with HTTP libraries, make it a versatile tool for load-testing cloud-based systems. This example serves as a starting point for QA engineers looking to learn Python's power in their load-testing attempts. NOTE In my scenario, the described script proved to be robust and impactful. It served as a useful tool in identifying and addressing numerous issues. The script's aggressive nature was okay in pinpointing critical issues, facilitating effective debugging, and leading the way for a seamless and improved user experience, which is pretty good for a QA. In continuation, I will briefly discuss another script utilizing Python's multiprocessing module. This approach aims to enhance load generation by concurrently executing multiple instances of the testing script. The primary goal of multiprocessing is to parallelize the execution of a script, enabling simultaneous interactions with the service. This approach contrasts with the asynchronous approach discussed earlier, where tasks are executed sequentially but concurrently managed. but it also might be very useful. This is more like spam/ddos with the same requests, Key Components Similar to the asynchronous script, we still need to fetch existing browser profiles from the cloud service. Function to retrieve profiles: def get_profiles():
    response = requests.get(url=f"{api}", params=PARAMS, headers=headers)
    return response The core of the script revolves around two functions: start_profiles and stop_profiles. These functions initiate and terminate browser profiles, respectively. Functions: def start_profiles(list_of_profiles_uuids):
    for uuid in list_of_profiles_uuids:
        # ... (API request to start profile)


def stop_profiles(internal_uuids):
    for uuid in internal_uuids:
        # ... (API request to stop profile)


def run_script():
    start_profiles(get_profile_ids())
    stop_profiles(list_of_ids) The script utilizes the multiprocessing module to run the load testing cycle multiple times in parallel. For each cycle, a new process is spawned, and the run_script function is executed concurrently. Multiprocessing execution: if __name__ == "__main__":
    for runs in range(0, 5):
        processes = []
        for i in range(20):
            p = multiprocessing.Process(target=run_script)
            processes.append(p)
            p.start()

        for p in processes:
            p.join() How to generate load with multiprocessing Each process spawned by the multiprocessing module operates independently, executing the run_script function concurrently. This parallelization significantly increases the number of requests sent to the service simultaneously. Parallel execution: By running multiple instances of the script in parallel, we simulate a higher volume of users interacting with the cloud service concurrently. This approach is particularly useful for scenarios where real-world usage involves a large number of simultaneous users. Simulating user interactions: Choosing the right approach The multiprocessing provides a strategy for load-testing applications. It allows QA engineers to experiment with different methodologies based on the unique characteristics of their applications. While asynchronous testing offers efficiency in managing concurrent tasks, multiprocessing excels in parallelizing the entire testing process. You can choose the approach that aligns best with their specific load-testing goals and application requirements. A quick reminder: This demo aims to introduce basic concepts in a beginner-friendly format, highlighting Python's simplicity for QA testers venturing into performance testing. If you have programming challenges, don't hesitate to google stuff and ask colleagues, use ChatGPT or similar tools, and use GitHub Copilot for extra assistance in writing your load-testing scripts. Also published . here

Python-Powered Performance Testing for QA Testers: A Beginner's Guide to Cloud API Load Testing

QA পরীক্ষকদের জন্য পাইথন-চালিত পারফরম্যান্স টেস্টিং: ক্লাউড API লোড পরীক্ষার জন্য একটি শিক্ষানবিস গাইড

Python-basierte Leistungstests für QA-Tester: Ein Leitfaden für Einsteiger in Cloud-API-Lasttests

Pruebas de rendimiento basadas en Python para probadores de control de calidad: una guía para principiantes sobre pruebas de carga de API en la nube

Tests de performances basés sur Python pour les testeurs d'assurance qualité : guide du débutant sur les tests de charge des API Cloud

क्यूए परीक्षकों के लिए पायथन-संचालित प्रदर्शन परीक्षण: क्लाउड एपीआई लोड परीक्षण के लिए एक शुरुआती मार्गदर्शिका

QA テスターのための Python を活用したパフォーマンス テスト: クラウド API 負荷テストの初心者ガイド

QA 테스터를 위한 Python 기반 성능 테스트: Cloud API 부하 테스트 초보자 가이드

Teste de desempenho baseado em Python para testadores de controle de qualidade: um guia para iniciantes em testes de carga de API em nuvem

Тестирование производительности на базе Python для тестировщиков QA: руководство для начинающих по нагрузочному тестированию облачных API

QA Test Uzmanları için Python Destekli Performans Testi: Yeni Başlayanlar İçin Bulut API Yük Testi Kılavuzu

Kiểm tra hiệu suất dựa trên Python dành cho người kiểm tra QA: Hướng dẫn cho người mới bắt đầu về kiểm tra tải API trên đám mây

面向 QA 测试人员的 Python 支持的性能测试：云 API 负载测试初学者指南

Playwright on Steroids: Overcoming Limits With Object-Oriented Programming

The Amazon product reviews page shows important information to customers because it provides insights into the quality and performance of products. However, it is difficult to scrape when you need login access. But tools like are useful to scrap data behind a login. Puppeteer This tutorial will show you how to scrape an Amazon product review behind login, parse the raw data, and export the reviews as CSV with a screenshot using and . Node.js Puppeteer Prerequisites Before you begin, make sure to install the following: Ensure you have Nodejs installed on your computer NodeJs: This Node library allows us to navigate the Amazon website, log in, and extract review information from the product page. Puppeteer: This library exports the data to a CSV file. Papaparse: You need an Amazon account to access the product review page. Amazon account: 1: Get Access to the Public Page This step involves retrieving the full HTML of the Aamazon public page. This HTML contains all the data displayed on the page, and you can access it by using Puppeteer to automate web interaction, which means you can automate the actions of a Chrome browser without having to launch it. Amazon public product page URL: https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/ Now that you have the URL, the next thing to do is write a Nodejs script that will retrieve the HTML content of the page above. Create a new folder for your project and initialize a Node project. Open your terminal and run the command below. Mkdri amazon-review
cd amazon-review
npm init -y This will create a package.json file for your project. Import the Puppeteer library. const puppeteer = require('puppeteer'); Set up a headless browser instance. (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); This starts an asynchronous and await javascript method, launches a headless Chromium browser using Puppeteer, and assigns it to a variable. browser The browser variable will call the page function will create a new browser page (This means a single tab in the browser) Navigate to the product page using the page.goto() function await page.goto('https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/',{ timeout: 60000 }
); This navigates to the URL added as an argument in the goto method. Sometimes, if the page is big and has a lot of information, it takes longer to load. For this tutorial, you will add an option to the method to increase the timeout limit and give the page more time to load. page.goto() { timeout: 60000 } You increased the timeout to 60 seconds (60000 milliseconds). Extract the full HTML content of the page const pageContent = await page.content(); This method returns the full html source code of the specified page and assigns it to the variable. page.content pageContent . Output the HTML content in your console console.log(pageContent); This logs the HTML content to the console, which allows you to view the raw HTML. Close the browser This closes the browser instance to free up space and resources. await browser.close(); Complete code (async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();

// Navigate to the product page
await page.goto(
"https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/",{ timeout: 60000 }

);

// Retrieve the full HTML content of the page
const pageContent = await page.content();

// Output the HTML to the console
console.log(pageContent);

// Close the browser
await browser.close();
})(); Example HTML output 2: Scrape Behind the login This web scraping method allows you to extract data from a website that requires login access. In this section, you will use to automate filling in your Amazon login details ( and ) and click the login button. Follow the steps below: Puppeteer email password Login into your Amazon account This instructs the Puppeteer to navigate to the Amazon login URL await page.goto(
"https://www.amazon.com/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fcart%2Fadd-to-cart%2Fref%3Dnav_custrec_signin&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0"
); Fill in the login details and click the login button After navigating to the login page, this await page.waitForSelector("#ap_email") This ensures that it proceeds only when the element is visible and present in the DOM. #ap_email Once the email input field is present, the script uses the method to simulate typing the Amazon email address into the input field and then clicking the button using the method await ("#continue-announce") page.type continue page.click() page.click Now, the script uses this await page.waitForSelector("#ap_password") to wait for the element to appear; once the password input field is present, the script uses to simulate typing the Amazon password ap_password page.type Replace with your email and password. After entering the password, the script finds the element with the id and simulates a click. auth-signin-button After clicking on the sign-in button, the script uses this await page.waitForNavigation(); wait for the login to complete. await page.waitForSelector("#ap_password");
await page.type("#ap_password", ""); // Replace with your Amazon password
await page.click("#auth-signin-button");
// Wait for the login to complete
await page.waitForNavigation(); . Navigate to the product review page After logging in, the scripts navigate to the product page using this method "https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/",
{ timeout: 60000 }
); 3: Parse Review Data This allows you to extract data from the Amazon product review page by doing the following. Wait for reviews to load This uses ; to wait for the element with the name “ ” to be present. Until review elements are found. page.waitForSelector(".review") review 10 These elements represent individual product reviews on the Amazon product page. Extract information from review elements. After waiting for the information to load, this code block extracts information from the review elements. It retrieves the author's name, review text, and review date. const reviews = await page.$$eval(".review", (reviewElements) => {
return reviewElements.slice(0, 10).map((review) => {
const author = review.querySelector(".a-profile-name").textContent;
const text = review.querySelector(".a-row.review-data").textContent;
const date = review.querySelector(".review-date").textContent;
return { author, text, date };
});
}); 4: Export Reviews to CSV This allows you to save them in a format that is easy to analyze; in this tutorial, you will export them to CSV. Export the data to a CSV file. This is where Papaparse comes in because it exports data to CSV; the data it extracted from the review is passed to method, which converts the review data to CSV format. Papa.unparse() After creating the CSV file, this code writes the CSV data to using Node.js inbuilt file system (fs). amazon_reviews.csv const csvData = Papa.unparse(reviews);
fs.writeFileSync("amazon_reviews.csv", csvData); Output of CSV author,text,date
Whitney Brown,Verified Purchase,"Reviewed in the United States on July 7, 2021"
Eiskatze,Verified Purchase,"Reviewed in the United States on March 4, 2019"
Billy Johnson,Verified Purchase,"Reviewed in the United States on May 24, 2019"
Katelyn goering,Verified Purchase,"Reviewed in the United States on March 24, 2021"
Timothy A. Taylor,Verified Purchase,"Reviewed in the United States on March 19, 2022"
Sandra G.,Verified Purchase,"Reviewed in the United States on March 4, 2019"
Amazon Customer,Verified Purchase,"Reviewed in the United States on March 29, 2021"
Amra,Verified Purchase,"Reviewed in the United States on February 17, 2019"
pietro,Verified Purchase,"Reviewed in Italy on December 28, 2022"
OLIVIER DETRY,Verified Purchase,"Reviewed in France on December 26, 2020" Then use method to take screenshots of the review page and give it a name. For example, you can use . page.screenshot screenshot.png await page.screenshot({ path: "screenshot.png" }); Output of the screenshot The complete codebase You can check out the complete codebase on my Github Repo: https://github.com/wise4rmgod/Amazon-webscrapping Conclusion Web scraping a website that requires a login is easy with tools like Puppeteer, and it's useful for businesses that want to scrap customer reviews, especially from a site like Amazon. This tutorial covers scraping an Amazon product page behind a login using Puppeteer.

How to Scrape Amazon Product Reviews Behind a Login

How We Enhanced Remotion Performance through GPU-Accelerated Server-Side Rendering Running with hardware acceleration in headless mode can be more challenging than it appears. We embarked on this journey with , which is an excellent framework that enables developers to "Make Videos Programmatically.” On our way, explore various Nvidia driver versions, including those from Nvidia's website and Ubuntu's official repositories. Google Chrome Remotion We collaborated with the vibrant Open-source community to find the answer for using . After encountering some setbacks on our way, we were able to make it work for Remotion eventually. We consolidated our findings in the Remotion Docs at and simplified the instructions for Remotion developers to make the most out of it. Although the whole research was done for , the same process will work for any headless application that requires GPU power, especially in headless mode. Remotion GPU with Remotion for server-side rendering Using the GPU in the Cloud Remotion Hardware Acceleration in Browser Rendering Hardware Acceleration is to your browser what " " is for a gasoline car. It uses the power of specialized hardware like GPUs to make your web pages load faster, and videos play smoothly. Hardware acceleration improves the part where the web page is displayed and shown to you, known as rendering. This results in a smoother and snappier browsing experience. Nitrogen Fuel DLAMI from Marketplace? One may wonder why we didn't opt for the AWS Deep Learning AMI or DLAMI. But we needed fine-grained control over the GPU drivers and utilities and the outdated DLAMIs despite all the bloatware was not the answer. Such tasks are often challenging to achieve with pre-packaged AMIs, as discussed in " .” Tailoring the environment to our exact requirements also helped us troubleshoot issues effectively. Why Deep Learning AMI is Holding You Back We experimented with multiple packages to identify the key ones needed for Google Chrome to recognize the GPU. Despite these efforts, GPU recognition remained elusive, and in some instances, GPU drivers were crashing even at initialization. The long and arduous journey has brought us to the end of the tunnel, yet the darkness persists, and the solution remains as elusive as ever. As we decided to call it a day, We stumbled upon on 's Github issues, which did not give us any hope but did give us a new perspective and drive to achieve our goal. this thread Puppeteer The Fresh Start It was time to return to the drawing board and start afresh. We started with a clean base Ubuntu AMI on AWS EC2 Instance. After that, we downloaded and installed the Nvidia Tesla T4G drivers from Nvidia's official website. Once it was confirmed that the GPU was working properly using , we installed Google Chrome and tested it for hardware acceleration. To our relief, Google Chrome finally gave a nod to work with Hardware Acceleration. nvidia-smi Launching an EC2 Instance Launching an EC2 instance can be achieved in various ways, with the AWS Web Console and AWS CLI being two notable options. Regardless of the method, the important parameters are instance type and AMI selection. At the time of writing, the AMI ID for our chosen instance is . For launching the EC2 instance via , the following command can be used after replacing the placeholders with actual values. g4dn.xlarge ami-053b0d53c279acc90 aws-cli aws ec2 run-instances \
  --instance-initiated-shutdown-behavior "terminate" \
  --image-id "ami-053b0d53c279acc90" \
  --instance-type "g4dn.xlarge" \
  --key-name "<KEY_NAME>" \
  --subnet-id "<SUBNET_ID>" \
  --security-group-ids "<SECURITY_GROUP_ID>" \
  --iam-instance-profile Arn="<ARN>" \
  --block-device-mappings '[
  {
    "DeviceName": "/dev/sda1",
    "Ebs": {
      "VolumeSize": 8,
      "VolumeType": "gp3"
    }
  }
]' Ubuntu and Kernel Upgrade Ubuntu 22.04 AMI on AWS comes with Linux Kernel (specifically ). At this point, we have two options either we stick with Kernel or Upgrade to ( ), we decided to upgrade the Kernel. This decision is crucial as the Nvidia driver compiled for one version will not work for the other. ami-053b0d53c279acc90 v5 5.19.0-1025-aws v5 v6 v6.2.0-1013-aws After launching the instance, SSH into the instance and initiate the Ubuntu packages upgrade process. To update all the installed packages, including Linux Kernel, execute the following one-liner. sudo bash -c "apt update && export DEBIAN_FRONTEND=noninteractive && export NEEDRESTART_MODE=a && apt upgrade -y && reboot" We can also split , and . But the reason behind doing it like this is to install these updates in non-interactive mode. apt update apt upgrade -y reboot After the execution, the system will reboot with Linux Kernel Version 6. We can confirm the kernel update with . Rebooting is essential when upgrading the Ubuntu system because it loads the new Kernel on the next boot, and Nvidia drivers will subsequently be built for the updated version. uname -r Nvidia GPU Driver Installation The process of installing GPU drivers is straightforward when we have the correct driver for the GPU and the corresponding compatible version. Before starting the installation, we need to install a couple of packages: and . The former is a bundle with a variety of compilation tools required to compile the Nvidia driver. The latter, while not mandatory, is required by Google Chrome. Therefore, it's a good idea to install it beforehand to enable support for Vulkan ICD loader for Nvidia, should we require it in the future. build-essential libvulkan1 sudo apt install -y build-essential libvulkan1 After build tools are installed, we can proceed to download and install the GPU drivers using the following commands: DRIVER_URL="https://us.download.nvidia.com/tesla/535.104.12/NVIDIA-Linux-x86_64-535.104.12.run"
DRIVER_NAME="NVIDIA-Linux-driver.run"
wget -O "$DRIVER_NAME" "$DRIVER_URL"
sudo sh "$DRIVER_NAME" --disable-nouveau --silent
rm "$DRIVER_NAME" Now we can run to confirm the GPU driver installation. The output will look similar to this: nvidia-smi Configuring the Startup Service We learned this the hard way: running once is needed for the proper initialization of and . Google Chrome and Chromium fail to initialize without this preliminary setup. To automate this process, we will create a service that runs at boot time using the following commands: nvidia-smi EGL ANGLE EGL nvidia-smi echo '[Unit]
Description=Run nvidia-smi at system startup

[Service]
ExecStart=/usr/bin/nvidia-smi
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target' | sudo tee /etc/systemd/system/nvidia-smi.service
sudo systemctl enable nvidia-smi.service
sudo systemctl start nvidia-smi.service Testing the Installation To test hardware acceleration with Google Chrome or Chromium, we can follow these steps. It's worth noting that Google Chrome has more dependencies, making it an easier choice if we plan to use the Node.js library, Puppeteer. On the other hand, Chromium is a lightweight option with a smaller footprint of dependencies, but we can still use Puppeteer with Chromium by installing a few extra dependencies. Installing Chromium Installation Installing Chromium is straightforward, as it is readily available with Ubuntu's official software repositories. We can install it using with the following command. apt sudo apt install -y chromium-browser Testing Chromium is fully prepared to handle Hardware Acceleration tasks. We can verify its readiness by launching Chromium with the specified parameters. chromium-browser --headless --use-gl=angle \
   --use-angle=gl-egl --use-cmd-decoder=passthrough \
   --print-to-pdf=output.pdf 'chrome://gpu' Installing Puppeteer Dependencies To use Puppeteer with Chromium instead of Google Chrome, there are a few extra dependencies that should be installed. sudo apt install ca-certificates fonts-liberation \
    libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 \
    libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 \
    libgbm1 libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 \
    libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 \
    libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 \
    libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
    libxtst6 lsb-release wget xdg-utils Installing Google Chrome Installation Google Chrome can be installed from multiple options, but we will use the Google Chrome official repository for Debian. We need to add it to Ubuntu's repo list along with the public signing key. curl -fsSL https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/googlechrom-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/googlechrom-keyring.gpg] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt update
sudo apt install -y google-chrome-stable Testing Google Chrome is ready for our Hardware Acceleration tasks. We can test it by launching Google Chrome with the following parameters. google-chrome-stable --headless --use-gl=angle \
    --use-angle=gl-egl --use-cmd-decoder=passthrough \
    --print-to-pdf=output.pdf 'chrome://gpu' The above commands will generate , and transfer it to your local machine to check the status of Hardware Acceleration in Google Chrome or Chromium. If the process goes smoothly, the resultant will look like this. output.pdf PDF Working with Puppeteer To verify GPU acceleration using the Node.js library Puppeteer, follow these steps: Installation Install Node.js, as this clean Ubuntu AMI does not include any unnecessary packages. Use the following commands: curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
NODE_MAJOR=18
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
sudo apt update
sudo apt install nodejs Testing After installing Node.js, we can install Puppeteer by running in our code directory. Here's a sample file for checking hardware acceleration: npm i puppeteer index.js const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({
    headless: true, 
    args: ['--use-gl=angle', '--use-angle=gl-egl'], 
  });
  const page = await browser.newPage();
  await page.goto('chrome://gpu');
  await page.waitForTimeout(2000);
  await page.screenshot({ path: 'output.png' });
  await browser.close();
})(); Run the script using . This will generate an file containing hardware acceleration information. node index.js output.png Note from the Creator has been immensely helpful in helping the “ ” community adopt GPU-accelerated rendering.He's researched and written about how to obtain, configure and run EC2 containers and headless Chrome correctly in order to take advantage of graphics acceleration.This area has been especially hard to crack, and without we would not have been able to unlock huge speed boosts. Mirza Remotion Mirza We are super grateful for that! -- Jonny Burger Creator / Remotion.dev Conclusion With hardware acceleration now readily available in headless mode, we can harness the power of GPUs for faster and more complex rendering tasks. Our next steps involve creating a custom AMI from this instance and streamlining the process using AWS Image Builder Pipelines for efficiency. Additionally, we plan to extend hardware capabilities to Docker containers, further extending our options. Resources Remotion - Make videos programmatically: https://remotion.dev Remotion - Using the GPU in the cloud: https://www.remotion.dev/docs/miscellaneous/cloud-gpu Cannot enable GPU acceleration: github.com/puppeteer/puppeteer/issues/3637 : Recommend a workflow for using the GPU on a AWS instance https://github.com/remotion-dev/remotion/issues/2889 Getting Started with Headless Chrome: developer.chrome.com/blog/headless-chrome Why Your AWS Deep Learning AMI is Holding You Back and How to Fix: mirzabilal.com/why-your-aws-deep-learning-a.. Deep Learning with “AWS Graviton2 + NVIDIA Tensor T4G” for as low as free* with CUDA 12.2: mirzabilal.com/deep-learning-with-aws-gravi.. How to install FFmpeg with Harware Accelaration on AWS: mirzabilal.com/how-to-install-ffmpeg-with-h.. CPU vs GPU for Video Transcoding - Challenging the Cost-Speed Myth: https://mirzabilal.com/cpu-vs-gpu-for-video-transcoding-challenging-the-cost-speed-myth Also published . here

Unlocking GPU-Accelerated Server-Side Rendering in Remotion With Chrome, Chromium & Puppeteer

Flaky automation tests can be a bane for QA engineers, introducing uncertainty and undermining the reliability of test suites. Drawing from my experience as an SDET and QA Automation mentor, this article offers practical advice to conquer flakiness. While I’ll provide JavaScript and Playwright examples, these universal tips are applicable in any language and framework, helping you write robust and dependable automation tests. Let’s dive in and ensure your tests stand firm. 1. Avoid the implicit wait methods Occasionally, you will encounter situations where you must wait for elements to appear in the DOM or for your application to reach a specific state to proceed with your test scenario. Even with modern, intelligent automation frameworks like Playwright’s auto-wait features, there will be instances where you need to implement custom wait methods. For instance, consider a scenario with text fields and a confirmation button. Due to specific server-side behavior, the confirmation button becomes visible only after approximately five seconds of completing the form. In such cases, resisting the temptation of inserting a five-second implicit wait between two test steps is essential. textField.fill('Some text')
waitForTime(5000)
confirmationButton.click() Use a smart approach instead and wait for the exact state. It can be the text element appearing after some loading or a loader spinning element disappearing after the current step is successfully done. You can check if you are ready for the next test step. button.click()
waitFor(textField.isVisible())
textField.fill() Of course, there are cases where you need to wait for something you cannot check smartly. I suggest adding comments in such places or even adding the reason explanation as a parameter in your wait methods or something like that: waitForTime(5000, {reason: "For unstable server processed something..."} An additional profit of using it is the test reports where you can store such explanations so it will be obvious for someone else or even for future you what and why such five-second waiting here. waitForTime(5000, {reason: "For unstable server processed something..."}
button.click()
waitFor(textField.isVisible())
textField.fill() 2. Use robust and reliable locators to select elements Locators are a crucial part of automation testing, and everyone knows it. However, despite this simple fact, many automation engineers are bailing on the stability and using something like this in their tests. //*[@id="editor_7"]/section/div[2]/div/h3[2] That is a silly approach because the DOM structure is not so static; some different teams can sometimes change it, and e this after your tests fail. So, use the robust locators. You’re welcome to my XPath-related story reading, by the way. https://hackernoon.com/getting-your-way-around-with-common-xpath-methods?embedable=true 3. Make tests independent from each other When running the test suite with multiple tests in a single thread, it’s crucial to ensure each test is independent. This means that the first test should return the system to its original state so the next test doesn’t fail due to unexpected conditions. For example, if one of your tests modifies user settings stored in LocalStorage, clearing the LocalStorage entry for User Settings after the first test runs is a good approach. This ensures that the second test will be executed with default user settings. You can also consider actions like resetting the page to the default entry page and clearing cookies and database entries so that each new test begins with identical initial conditions. For example, in Playwright, you can use the beforeAll(), beforeEach(), afterAll(), and afterEach() annotations to achieve it. Check the next test suite with a couple of tests. The first is changing the user's appearance settings, and the second is checking the default appearance. test.describe('Test suite', () => {
  test('TC101 - update appearance settings to dark', async ({page}) => {
    await user.goTo(views.settings);
    await user.setAppearance(scheme.dark);
  });

  test('TC102 - check if default appearance is light', async ({page}) => {
    await user.goTo(views.settings);
    await user.checkAppearance(scheme.light);
  });
}); If such a test suite runs in parallel, everything will go smoothly. However, if these tests are run one by one, you could face a failed test because one of them is interfering with another. The first test changed the appearance from light to dark. The second test checks if the current appearance is light. But it will fail since the first test changed the default appearance already. To solve such a problem, I’ve added the afterEach() hook executed after each test. In this case, it will navigate to the default view and clear the user's appearance by removing it from the Local Storage. test.afterEach(async ({ page }) => {
  await user.localStorage(appearanceSettings).clear()
  await user.goTo(views.home)
});

test.describe('Test suite', () => {
  test('TC101 - update appearance settings to dark', async ({page}) => {
    await user.goto(views.settings);
    await user.setAppearance(scheme.dark);
  });

  test('TC102 - check if default appearance is light', async ({page}) => {
    await user.goTo(views.settings);
    await user.checkAppearance(scheme.light);
  });
}); Using this approach, each test in this suite will be independent. 4. Use automatic retries wisely No one is immune to encountering flaky tests, and a popular remedy for this issue involves using retries. You can configure retries to occur automatically, even at the CI/CD configuration level. For instance, you can set up automatic retries for each failed test, specifying a maximum retry count of three times. Any test that initially fails will be retried up to three times until the test execution cycle is complete. The advantage is that if your test is only slightly flaky, it may pass after a couple of retries. However, the downside is that it could fail, resulting in extra runtime consumption. Moreover, this practice can inadvertently lead to an accumulation of flaky tests, as many tests may appear to “pass” on the second or third attempt, and you might incorrectly label them as stable. Therefore, I do not recommend setting auto-retries globally for your entire test project. Instead, use retries selectively in cases where you cannot promptly resolve the underlying issues in the test. For instance, pretend you have a test case where you must upload some files using the ‘filechooser’ event. You get the <Timeout exceeded while waiting for event ‘filechooser’> error, which indicates that the Playwright test is waiting for a ‘filechooser’ event to occur but is taking longer ied timeout. async function uploadFile(page: Page, file) {
  const fileChooserPromise = page.waitForEvent('filechooser');
  await clickOnElement(page, uploadButton);
  const fileChooser = await fileChooserPromise;
  await fileChooser.setFiles(file);
} This can be due to various reasons, such as unexpected behavior in the application or a slow environment. Because it is random behavior, the first thing you could think about is to use automatic retry for this test. However, if you retry the entire test, you risk wasting extra time and getting the same behavior you got with the first error in the uploadFile() method itself, so if the error occurs, you don’t need to retry the entire test. To do this, you can use the standard try…catch statement. async function uploadFile(page: Page, file) {
  const maxRetries = 3;
  let retryCount = 0;

  while (retryCount < maxRetries) {
    try {
      const fileChooserPromise = page.waitForEvent('filechooser');
      await clickOnElement(page, uploadButton);
      const fileChooser = await fileChooserPromise;
      await fileChooser.setFiles(file);
      break; // Success, exit the loop
    } catch (error) {
      console.error(`Attempt ${retryCount + 1} failed: ${error}`);
      retryCount++;
    }
  }
} Such an approach will save extra time and make your test less flaky. In closing, the path to reliable automation tests is straightforward. You can minimize or actively address common challenges and implement intelligent strategies. Remember, this journey is ongoing, and your tests will become more dependable with persistence. Say goodbye to flakiness and welcome stability. Your commitment to quality ensures project success. Happy testing! Also published . here

The Four Horsemen of Flaky Tests

ফ্ল্যাকি টেস্টের চার ঘোড়সওয়ার

Die vier Reiter der Flockentests

Los cuatro jinetes de las pruebas escamosas

Les quatre cavaliers des épreuves feuilletées

परतदार परीक्षणों के चार घुड़सवार

不安定なテストの四騎士

플래키 테스트의 네 기수

Os quatro cavaleiros dos testes esquisitos

Четыре всадника нестабильных тестов

Kesintisiz Testlerin Dört Atlısı

Bốn kỵ sĩ của những bài kiểm tra dễ vỡ

Flaky 测试的四骑士

About the issue I am a co-founder of a product called Alto's POS & Inventory. Every day, I review charts, logs, and numbers from multiple platforms to gain a complete understanding of whether everything is okay. On average, this process takes about 15 minutes per day, but sometimes it can extend to 30 minutes. After performing some basic calculations, I realized that over the course of one year, it amounts to more than 12 workdays. That's a substantial amount of time—12 workdays per year—that I spend on repetitive tasks. Consequently, I have decided to address this issue and allocate this time to more productive tasks. Requirements Now that we've identified the problem, let's outline the requirements for the solution: A single-page dashboard that displays all the crucial information. The ability to collect data from multiple platforms. Processing the collected data, mixing it if necessary, and extracting concise insights. Immediate display on the dashboard as a warning and notification if any action requires my attention. Easy accessibility for both me and my co-founders on any device and from anywhere to view this dashboard. About my solution The solution should be worth the effort. For instance, if I have to invest a significant amount of time in resolving this issue, it might be better to leave it as it is now. So, let's strive to find a solution that is easy to implement. Solutions are listed from the easiest to the most challenging: Notifications In my case, the easiest way to notifications is to use . I can quickly set up and start using it immediately. receive Telegram Bots Hosting To make the dashboard accessible on any device, at any time and from anywhere, it should be hosted on services like or . AWS IONOS Data collection Initially, I tried to find existing services that would allow me to easily gather data, but I couldn't find anything that suited my needs. So, I decided to explore the possibility of creating a custom solution. My approach involved collecting data from the platforms we use through their APIs, processing that data, and presenting it on a dashboard. However, I encountered a challenge as many of these platforms didn't have available APIs for data access. This forced me to explore alternative methods, and I found that Puppeteer was the most suitable choice for my specific requirements. Puppeteer Puppeteer is a Node.js which provides a high-level API to control Chrome/Chromium over the DevTools Protocol. Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples: library Generate screenshots and PDFs of pages. Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)). Automate form submission, UI testing, keyboard input, etc. Create an automated testing environment using the latest JavaScript and browser features. Capture a of your site to help diagnose performance issues. timeline trace Chrome Extensions. Test Here's a in action, which performs the following steps: simple example of Puppeteer Navigates to . Google.com Executes a search for " " HackerNoon Clicks on the first search result. Takes a screenshot. import puppeteer from 'puppeteer';

(async (searchValue) => {
  // Launch the browser and open a new blank page
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Search
  await page.goto('https://google.com');
  await page.locator('textarea').fill(searchValue);
  await page.$eval('form', form => form.submit());

  // Go to the first link
  await page.waitForNavigation();
  await page.click(`div[data-async-context^="query:"] a`);

  // Take a screenshot
  await page.waitForNavigation();
  await page.screenshot({path: './screenshot.png'});

  await browser.close();

})("HackerNoon"); Idea validation Let's validate the idea with a simple implementation that closely resembles a real scenario. Since we need to gather data from multiple platforms, let's choose a simpler scenario that is similar to this task, such as building a page that displays HackerNoon top stories and software engineering jobs. The following code collects data from and every hour, generates simple HTML content from this data, and then serves this HTML content when we receive an HTTP request. It's quite straightforward. HackerNoon Top Stories HackerNoon Jobs index.js: import http from 'http';
import * as scraper from './scraper.js';

(async () => {
  let scrapedHtml = 'Try again later...';
  http.createServer((req, res) => {
    res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8'})
    res.end(scrapedHtml);
  
  }).listen(8080);

  scrapedHtml = await scrapeAll();
  setInterval(async () => scrapedHtml = await scrapeAll(), 60*60*1000);
})();

async function scrapeAll() {
  const browser = await scraper.launchBrowser();
  const [stories, jobs] = await Promise.all([
    scraper.getTopStories(browser),
    scraper.getJobs('Software Engineer', browser)
  ]);
  await browser.close();
  return `
    <h2>Top Stories</h2>
    <ul>${stories.map(e => linkToHtml(e.title, e.url)).join('')}</ul>

    <h2>Jobs</h2>
    <ul>${jobs.map(e => linkToHtml(e.title, e.url)).join('')}</ul>
  `;
}

const linkToHtml = (title, url) => {
  return `<li>
    <a target="_blank" href="${url}">
      ${title}
    </a>
  </li>`;
} scraper.js: import puppeteer, {Browser} from 'puppeteer';

/**
 * 
 * @returns {Browser}
 */
export async function launchBrowser() {
  return await puppeteer.launch();
}

/**
 * 
 * @param {Browser} browser
 * @returns {[{title: String, url: String}]}
 */
export async function getTopStories(browser) {
  const page = await browser.newPage();
  await page.goto('https://hackernoon.com/tagged/hackernoon-top-story');

  // Wait for articles
  await page.waitForSelector('main .story-card');

  // Get articles
  const res = [];
  const articles = await page.$$('main .story-card h2 a');
  for (const article of articles) {
    res.push(
      await article.evaluate(el => ({
        "title": el.textContent,
        "url": el.href,
      }))
    );
  }
  return res;
}

/**
 * 
 * @param {String} keyword
 * @param {Browser} browser
 * @returns {[{title: String, url: String}]}
 */
export async function getJobs(keyword, browser) {
  const page = await browser.newPage();
  await page.goto('https://jobs.hackernoon.com');

  // Search
  await page.locator('#search-jobkeyword input').fill(keyword);
  await page.click('button[type=submit]');

  // Wait for result
  await page.waitForSelector('.job-list-item');

  // Get jobs
  const res = [];
  const items = await page.$$('.job-list-item');
  for (const item of items) {
    res.push(
      await item.evaluate(el => ({
        "title": [
          el.querySelector('.job-name'),
          ...el.querySelectorAll('.desktop-view span')
        ].map(e => e.textContent).join(', '),
        "url": el.href,
      }))
    );
  }
  return res;
} The result looks something like this: Advantages In my perspective, this solution offers the following advantages: Relatively quick implementation A single method of data extraction Ease of extension if I need to add an additional platform Disadvantages From my point of view, the disadvantages of this solution are as follows: Requires adjustment if the user interface of any platform changes It may be challenging or impossible to extract data from certain platforms, particularly those with enhanced security measures. Conclusion This solution proved effective for my situation, allowing me to resolve my problem quickly. If you believe there's a better approach to solving it, I would be delighted if you could share your insights. You can access the . source code for this example on GitHub

Streamlining Workflow with Puppeteer

Page2API

page2api.com

-- employees

Since n.d.

COMPANY RANKING

PAGE2API

EVERGREEN INDEX #7500

Page2API WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Page2API

page2api.com

-- employees

Since n.d.

COMPANY RANKING

PAGE2API

EVERGREEN INDEX #7500

RELATED COMPANIESRANK

STORIES ABOUT PAGE2API

Page2API WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!