Building a data scraping application is common for many developers, especially in fields where data collection, analysis, and storage are crucial. Using the Puppeteer, Node.js, and PostgreSQL trio is an excellent approach. is a great choice for this task. It allows you to control a headless browser, which is crucial for scraping dynamic websites. Many websites now use JavaScript to load content, and Puppeteer enables you to interact with and scrape this dynamic content. Puppeteer is well-suited for building scalable and efficient server-side applications. It's non-blocking and event-driven, making it a good fit for handling concurrent requests in a scraping application. Node.js will handle the backend of the data-scraping application. Node.js Storing the scraped data for the app will be handled by because it supports complex queries and indexing, which makes it suitable for storing and retrieving scraped data efficiently. PostgreSQL is a PaaS provider that aids the secure deployment of applications. It also provides database management services, with PostgreSQL being one of its supported databases. Hence, this project will use the managed PostgreSQL service provided by Aptible. Aptible In this tutorial, you'll discover how to build a data-scraping application using Puppeteer, Node.js, PostgreSQL, and Aptible. The application will scrape data from a popular movie streaming app, Netflix, and store it in a PostgreSQL database managed by Aptible. Prerequisites To follow along with this tutorial, you will need the following: A node version of 16.13.2 or greater is installed on your machine. You can install it by following the instructions on the . official Node.js website An Aptible account. Visit the to sign up. Aptible website A fundamental understanding of JavaScript and Node.js. Setting up the project To get started, create a new directory for the project and initialize a new Node.js project in it by running the following commands: mkdir web-scraper cd web-scraper npm init -y This will create a file for you with the following output: package.json Wrote to /home/user/web-scraper/package.json: { "name": "web-scraper", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC" } Installing dependencies To install the dependencies for the project, run the following command: npm install puppeteer pg dotenv This will install the necessary packages, which are Puppeteer, Node-postgres, and dotenv. Creating the database For the PostgreSQL database, you will use the managed database service on Aptible. To do this, follow the steps below: Log in to your Aptible account and create a new environment. Navigate to the new environment and click on > . Databases New Database Click on the dropdown menu and select . This will create a new PostgreSQL database for you. PostgreSQL 16 Input the Database Handle and click on to create the database. New Database This will provision a new PostgreSQL database for you. You can view the database by clicking the tab on the left menu. Databases Creating a database endpoint To connect to the PostgreSQL database, you will need to create a database endpoint. To do this, follow the steps below: Navigate to the database you created and click on > . Endpoints New Endpoint Include an IP address in the field. This will allow you to connect to the database from your local machine. IP Allowlist After the endpoint has finished provisioning, click on the tab. ConnectionURL Click on the button to copy the connection URL. You will need this to connect to the database. show Create the scraper script. This scraper script will scrape data from Netflix and store it in the PostgreSQL database. To create the script, create a new file named in the root of the project directory and add the following code to it: scraper.js const puppeteer = require("puppeteer"); const { Client } = require("pg"); require("dotenv").config(); The code above imports the necessary packages and loads the environment variables from the file. .env Below is a breakdown of the packages imported: : A headless browser automation library for Node.js used for web scraping. puppeteer : Part of the pg library, a PostgreSQL client for Node.js. Client : A library for loading environment variables from a file, used to load database credentials. dotenv Next, add the following code to the file: scraper.js async function run() { const browser = await puppeteer.launch({ headless: "new" }); const page = await browser.newPage(); // Navigate to the website you want to scrape await page.goto("https://netflix.com/"); // Extract data from the page const data = await page.evaluate(() => { const title = document.querySelector("h1").innerText; const url = window.location.href; return { title, url }; }); // Close the browser await browser.close(); return data; } The code above creates a function named which will be used to scrape data from the Netflix website. run A brief explanation of the code above: is an asynchronous function that launches a headless browser using Puppeteer. run It opens a new page, navigates to , extracts data (title and URL) from the page using a function provided to , and then close the browser. https://netflix.com/ page.evaluate The extracted data is returned as an object. Next, add the following code to the file: scraper.js async function saveToDatabase(data) { const client = new Client({ connectionString: process.env.DATABASE_URL, ssl: { rejectUnauthorized: false }, }); try { await client.connect(); // Check if the table exists, and create it if not const createTableQuery = ` CREATE TABLE IF NOT EXISTS scraped_data ( id SERIAL PRIMARY KEY, title TEXT, url TEXT ); `; await client.query(createTableQuery); // Insert data into the database const insertQuery = "INSERT INTO scraped_data (title, url) VALUES ($1, $2)"; const values = [data.title, data.url]; await client.query(insertQuery, values); } finally { await client.end(); } } The code above creates a function named which will be used to save the scraped data to the PostgreSQL database. saveToDatabase A brief explanation of the code above: is an asynchronous function that connects to the PostgreSQL database using the credentials in the environment variable. saveToDatabase DATABASE_URL It checks if the table exists and creates it if it doesn't. scraped_data It then inserts the scraped data into the database using an SQL query. Next, add the following code to the file: scraper.js (async () => { try { const scrapedData = await run(); console.log("Scraped Data:", scrapedData); // Save data to the database await saveToDatabase(scrapedData); console.log("Data saved to the database."); } catch (error) { console.error("Error during scraping:", error); } })(); The code above calls the function to scrape data from the Netflix website and then call the function to save the scraped data to the PostgreSQL database. run saveToDatabase A brief explanation of the code above: This immediately invoked function expression (IIFE) executes the scraping and database saving process. It catches any errors that occur during scraping or database operations and logs them. If successful, it logs the scraped data and a success message. All together, the file should look like this: scraper.js const puppeteer = require("puppeteer"); const { Client } = require("pg"); require("dotenv").config(); async function run() { const browser = await puppeteer.launch({ headless: "new" }); const page = await browser.newPage(); // Navigate to the website you want to scrape await page.goto("https://netflix.com/"); // Extract data from the page const data = await page.evaluate(() => { const title = document.querySelector("h1").innerText; const url = window.location.href; return { title, url }; }); // Close the browser await browser.close(); return data; } async function saveToDatabase(data) { const client = new Client({ connectionString: process.env.DATABASE_URL, ssl: { rejectUnauthorized: false }, }); try { await client.connect(); // Check if the table exists, and create it if not const createTableQuery = ` CREATE TABLE IF NOT EXISTS scraped_data ( id SERIAL PRIMARY KEY, title TEXT, url TEXT ); `; await client.query(createTableQuery); // Insert data into the database const insertQuery = "INSERT INTO scraped_data (title, url) VALUES ($1, $2)"; const values = [data.title, data.url]; await client.query(insertQuery, values); } finally { await client.end(); } } (async () => { try { const scrapedData = await run(); console.log("Scraped Data:", scrapedData); // Save data to the database await saveToDatabase(scrapedData); console.log("Data saved to the database."); } catch (error) { console.error("Error during scraping:", error); } })(); Creating a file .env The file will contain the database credentials. Create a new file named in the root of the project directory and add the following code to it: .env .env DATABASE_URL=postgres://<username>:<password>@<host>:<port>/<database> Replace the placeholders with the database credentials from the database endpoint you created earlier. Note: Running the scraper script To run the scraper script, run the following command: node scraper.js This will launch a headless browser, navigate to the Netflix website, scrape data from the page, and save the data to the PostgreSQL database on Aptible. You should see the following output: Scraped Data: { title: 'Unlimited movies, TV shows, and more', url: 'https://www.netflix.com/' } Data saved to the database. You can verify that the data was saved to the database by logging into the PostgreSQL database using the credentials from the database endpoint. You can do this by running the following command: psql <connection_url> Replace with the connection URL from the database endpoint. Note: <connection_url> Once you're logged in, you can list the tables in the database by running the following command: \dt You should see the table listed in the output: scraped_data List of relations Schema | Name | Type | Owner --------+--------------+-------+--------- public | scraped_data | table | aptible (1 row) You can view the data in the table by running the following command: scraped_data SELECT * FROM scraped_data; You should see the scraped data listed in the output: id | title | url ----+--------------------------------------+-------------------------- 1 | Unlimited movies, TV shows, and more | https://www.netflix.com/ (1 row) Conclusion In this tutorial, you learned how to create a data scrapping application using Puppeteer, Node.js, PostgreSQL, and Aptible. To enhance this project, consider adding more functionality to the scraper script. For instance, you can expand it to scrape data from multiple websites and store it in the database. Furthermore, you can extend the application by incorporating a frontend. This addition lets you visualize the scraped data in a web browser, enhancing the user experience.
