What is end-to-end testing? If you’re new to end-to-end testing, it is a type of automated software testing in which the entire application is being tested by simulating actions that a real user would do. is an open-source library for writing and executing automated and end-to-end tests for websites and web applications. It was published in 2014, and in 2021, it was transferred to the open-source program office at BrowserStack, where it is currently being developed. Written in Node.js, it has support for all major web browsers, and also it can run tests on mobile devices as well. Nightwatch.js This tutorial will look at how to develop a Nightwatch.js plugin that sends the test failure and associated errors to a service that integrates with the to analyze the errors and get some actionable feedback. By default, in recent versions, Nightwatch already has pretty good feedback for test failures and provides some degree of actionable feedback, so we’ll attempt to extend that by using the GPT-4 model to add a bit more flair to the output messages, provide a slightly better context and learn how to develop services that incorporate the assistance of AI. OpenAI platform Step 1 — create the error analysis service Our exercise here consists of mainly two parts, both of them relatively straightforward: Building the backend service, which makes the call to the OpenAI service Writing the Nightwatch.js plugin, which will receive the actual test failure and send it to the backend service for analysis. We’ll start with part 1. — the analysis service is only a basic express.js API service that accepts POST requests and makes a specific call to the OpenAI platform using the SDK for Node.js. You’ll need to get yourself a developer key from OpenAI and then configure a model to be used. For the purpose of this article, I used but that requires a paid plan. If you'd like to try it out on a free plan, you can use the . here gpt-4-1106-preview gpt-3.5-turbo 1.1 Project structure Set up the new project with: mkdir nightwatch-openai-service 
cd nightwatch-openai-service 
touch index.js npm 
init -y Next, edit the file and set , e.g.: package.json type=module { "name": "openai-nightwatch-service", "type": "module", ... } Then go ahead and install the required dependencies: npm i dotenv express openai 1.2 Add the service In the newly created project, create two new files: – should contain the openai api key, e.g.: .env OPENAI_API_KEY=xxxxxx 
PORT=4001 – paste the code below index.js import dotenv from 'dotenv';
import express from 'express';
import { OpenAI } from 'openai';

dotenv.config();
const app = express();
app.use(express.json());
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
app.post('/analyze-error', async (req, res) => {
  try {
    const { errorMessage, codeSnippet, additionalDetails } = req.body;
 const details = `Additional details: Nightwatch version: ${additionalDetails.nightwatchVersion}, config file: ${additionalDetails.configFile}, platform: ${additionalDetails.platform}, browser: ${additionalDetails.browser}, headless mode: ${additionalDetails.headless}.`;
    const messages = [
      {
        role: "system",
        content: "You are an expert in web development using Node.js, automated testing with Selenium WebDriver, and the Nightwatch.js framework."
      },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: `Investigate and explain why the tests failed. Error message: ${errorMessage}\n.Code snippet from test case where the error occurred: ${codeSnippet}. ${details}`
          }
        ]
      }
    ];
    const response = await openai.chat.completions.create({
      model: "gpt-4-1106-preview",
      messages,
      max_tokens: 600,
    });
    res.json({ analyzedResult: response.choices[0].message.content });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running on port ${PORT}`)); That’s about it for the service. Simply run it with: node index.js We will be sending in the error message from the Nightwatch plugin, along with a small code snippet that contains the line of code where the error or assertion failure occurred. The only thing which remains is to think with the prompt. There’s an entire section on the OpenAI docs called — how to write better prompts to improve your results, which is where we are now when it comes to product innovation. prompt engineering The code is available . Go ahead and fork it and run it locally. We’ll need it for the next part of the tutorial. on GitHub Step 2 — write the Nightwatch.js reporting plugin Besides the built-in test reporters which are included by default (junit-xml, json, html), Nightwatch supports functionality to load custom reporters, which is what we’re going to develop next. The complete code is and the package is published in NPM as , so you can actually use it directly and skip ahead to Step 3 if you prefer. available on Github nightwatch-openai-plugin https://www.npmjs.com/package/nightwatch-openai-plugin?embedable=true The role of the custom reporter plugin is to send the error data to the AI-assisted analysis service which we developed at Step 1. To do that, we need to create a new Node.js project that follows a specific structure which Nightwatch will understand. 2.1 Project structure First, set up the new project with: mkdir my-nightwatch-ai-reporter 
cd my-nightwatch-ai-reporter 
touch index.js 
npm init -y 
git init Basically, the plugin needs to be wrapped as an NPM package and export a module that looks like below: // index.js
module.exports = {
  async reporter(results) {
console.log('do some reporting here...');
  }
} We also need to add an file which will populate with the url for our AI Analysis service. If you’re running the service as per Step 1 of this article, the file will look like so: .env .env SERVICE_URL=http://localhost:4001/analyze-error 2.2 Write the custom reporter Now we just have to populate the function inside the file with some logic to send the reports to Analysis service from step 1 and display the results. The Analysis service will call the OpenAI platform with the prompt we have defined. reporter() index.js When the test run finishes, Nightwatch calls the reporter with the argument, which contains the failure results and any other associated errors. Here's how it will look like: results module.exports = {
  async reporter(results) {
    const errors = getErrorMessages(results);
    if (!errors) {
      return;
    }
    const outputs = makeOutputs(errors);
    for (const output of outputs) {
      try {
        const response = sendErrorAnalysisRequest(output);
        const terminalOutput = marked.parse(response.data.analyzedResult);
        console.log('Error analysis completed:', terminalOutput);
      } catch (err) {
        console.error('Error analysis failed:', error.response?.data || error.message);
      }
    }
  }
} The simply will issue a POST request with the test data. sendErrorAnalysisRequest The complete code is available on GitHub: https://github.com/pineviewlabs/nightwatch-openai-plugin?embedable=true Step 3 — putting it all together Now that we have the plugin and the service, it’s time to put them together in a test project. We’re going to build a small end-to-end testing project which contains a few very basic tests for an example website. The project will use Nightwatch to run the tests and our newly created plugin. 3.1 Setup a test project First, create a test project with the commands below: mkdir nightwatch-testing 
cd nightwatch-testing 
npm init -y 3.2 Install nightwatch from NPM Nightwatch can be installed from NPM with simply running the command below and it is ready to go: npm i nightwatch You can verify if nightwatch has been installed with: npx nightwatch --info 3.3 Add the Nightwatch reporting plugin Now it’s t time to add the AI analysis plugin which we’ve developed at Step 2 to our test project, so Nightwatch can discover it and use it. You can either install the package directly from NPM or you can use the local version if you have completed Step 2 in full. To install it from NPM run: npm i nightwatch-openai-plugin To install it from your local folder (update the path accordingly, relative locations also work): npm i /path/to/my-nightwatch-ai-reporter 3.4 Configure Nightwatch to load the plugin In order for Nightwatch to be able to load the plugin, we need to define it in the nightwatch config file — . nightwatch.conf.js First, let’s examine the file. It should have the plugin it the list of dependencies. Assuming the plugin was installed from NPM, it will look like so: package.json {
  "name": "nightwatch-testing",
  ...
  "dependencies": {
    "nightwatch": "^3.3.1",
 "nightwatch-openai-plugin": "^0.1.0"
  }
} Now open the file and add to the array, as below: nightwatch.conf.js nightwatch-openai-plugin plugins // nightwatch.conf.js
module.exports = {
  // ... other settings
  plugins: ['nightwatch-openai-plugin'],
  // continued...  
} You can verify that Nightwatch was installed and working properly by running an example test, which is bundled with the library, using Chrome: npx nightwatch examples/tests/duckDuckGo.js --chrome The output will look like: ℹ Connected to ChromeDriver on port 9515 (1001ms).
Using: chrome (119.0.6045.123) on MAC.

Running Search Nightwatch.js and check results:
───────────────────────────────────────────────────────────────
  ✔ Element <body> was visible after 15 milliseconds.
  ✔ Testing if element <input[name="q"]> is visible (17ms)
  ✔ Testing if element <button[type=submit]> is visible (14ms)
  ✔ Testing if element <.react-results--main> contains text 'Nightwatch.js' (1545ms)
  ✨ PASSED. 4 assertions. (2.534s) You can also specify , , or depending on which of these browsers you have installed on your machine. --firefox --safari --edge Step 4 — run the tests and check the analysis report Now that we have Nightwatch installed and configured using the plugin for the AI-assisted analysis service we developed in Step 1, we can run some more end-to-end tests and see it in action. If you haven’t completed Step 1 or if you want to get a quick bite before you start, I have prepared an example project for you, including a demo backend service that is up and running so you can see it in action: https://github.com/pineviewlabs/nightwatch-ai-testproject Feel free to fork it and run it on your local machine. Please do note that the Analysis service is running in a for Demo purposes and should not be used in real-life testing scenarios. limited capacity 4.1 Adding some end-to-end tests For those of you who have completed the previous steps and are neck deep in it, we just need to add a few basic tests so we can run everything locally. Go to the folder and create a new folder: nightwatch-testing test Then add these 2 tests inside the folder: test Then add these 2 tests inside the test folder: 1) homepage.js describe('Homepage End-to-end Test', () => {
  it('tests if homepage is loaded', browser => {
    browser
      .navigateTo('https://middlemarch.netlify.app/')
      .assert.visible('#app .new-arrivals-panel')
      .expect.elements('#app .new-arrivals-panel .col-md-6').count.toEqual(4)
  });
}); 2) addtocart.js describe('add to cart test', () => {
  before(browser => browser.navigateTo('https://middlemarch.netlify.app/'));

  it('adds 2 volumes of "Rhinoceros and Other Plays" to cart', browser => {
    const addToCartEl = browser.element.findByText('Rhinoceros and Other Plays').getParentElement().find('button');
    addToCartEl.click()
    addToCartEl.click()
    browser.assert.textEquals('.shopping-cart .badge', '2');
  });
  after(browser => browser.end());
}); Both tests are written against an example bookstore app, part of an earlier tutorial I wrote about Vite and Vue 3. The first test simply opens up the website and verifies if the content is there, while the second test adds a book to cart and performs a basic assertion. here To run the tests, use the command as below, optionally passing the argument if you don't like to see the browser popping up during the tests: --headless npx nightwatch test --chrome

# or use --fireox, --safari, or --edge 4.2 Intentionally fail the tests In order to test the AI Analysis service we’ll need to intentionally fail at least one of the tests. Then, the plugin reporter will come into effect and send the test failure to the backend service, and then print the result. Thankfully, there are many available options to fail a test. One of the simplest ways is to rename one of the elements and then wait for the test to fail with an “element not found” error. Simulate “element not found” errors Edit the file from the test folder and simply change the line that starts with to something like: homepage.js .assert.visible browser
 .navigateTo('https://middlemarch.netlify.app/')
 .assert.visible('#xapp') Then the test will simply fail with an error message that the element with the selector could not be found and will print something similar to the following output: #xapp TEST FAILURE (12.844s):  
 - 1 assertions failed; 1 passed

✖ 1) homepage
 – tests if homepage is loaded (7.903s)
 → ✖ NightwatchAssertError
 Testing if element <#xapp .new-arrivals-panel> is visible in 5000ms - expected "is visible" but got: "element could not be located" (5131ms)
    Error location:
    /Users/andrei/pineviewlabs/nightwatch-openai/test/homepage.js:6
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     4 |     browser
     5 |       .navigateTo('https://middlemarch.netlify.app/')
     6 |       .assert.visible('#xapp .new-arrivals-panel') 
     7 |       .expect.elements('#app .new-arrivals-panel .col-md-6').count.toEqual(4)
     8 |   });
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– Then we can actually see the report from the AI Analysis service: Error analysis completed: The failure indicates that the element with CSS selector #xapp .new-arrivals-panel was not found within the time frame of 5000ms (5 seconds). Here's what you can do to debug this:

1. Update the test code:
    .debug({selector: "#xapp .new-arrivals-panel"}) // Add this line
    .browser.expect.element('#xapp .new-arrivals-panel').to.be.visible.before(5000);

2. Run Nightwatch with the flags for debugging:
    nightwatch --debug --devtools

This will open the Chrome Developer Tools where you can inspect the page and the console.

Probable reasons for the error could include:
    * The element does not exist on the page at the time of testing.
    * The element is not visible within the 5 seconds because the page hasn't finished loading, there are network delays, or JavaScript that displays the element runs late.
    * The selector is incorrect or has changed.
    * There's an error in the page's JavaScript that prevents the element from being displayed correctly. The report might be a bit too long and overly generic, but it is now only a matter of tuning the prompt so it generates desired results, which is not a task for this article. 4.3 Configure the analysis backend service The uses a default HTTP API service to interact with the OpenAI API, provided for demo purposes. You can host your own service by cloning the repository and running it with your own OpenAI API key. nightwatch-openai-plugin openai-nightwatch-service When running the , you need to define the environment variable to point to the service URL. You can also use files. openai-nightwatch-service NIGHTWATCH_ANALYSIS_SERVICE_URL .env For example, assuming you have the service running at , you can create a file with the following content in the root of your Nightwatch project: http://localhost:4001 .env NIGHTWATCH_ANALYSIS_SERVICE_URL=http://localhost:4001/analyze-error Conclusion So there you have it. We have managed to build (I hope) an AI-assisted analysis plugin for Nightwatch tests, and also we’ve seen it in action. You can simulate now try to simulate all kinds of errors and see what the response is. Remember, this is just an experiment, and the service is only available for Demo purposes. I haven’t attempted to try out different types of errors and test failures yet, and I haven’t tuned the prompt for too long, so I can’t guarantee it will scale up to a large collection of tests where you might have different types of failures. So you’ll have to try this at your own risk, but you’re welcome to perform your own experiments and report back your findings. Also published . here

Build a Single Page Web3 App with Vue 3, Vite and Pinia

Building an Analysis Service for Automated Test Failures Using OpenAI

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Look at End-to-end Testing in Nightwatch v2.0

11 Best Automation Testing Tools to Try in 2021

145 Stories To Learn About Test Automation

Python-Powered Performance Testing for QA Testers: A Beginner's Guide to Cloud API Load Testing

57 Stories To Learn About Automated Testing

A Look at End-to-end Testing in Nightwatch v2.0

11 Best Automation Testing Tools to Try in 2021

145 Stories To Learn About Test Automation

Python-Powered Performance Testing for QA Testers: A Beginner's Guide to Cloud API Load Testing

57 Stories To Learn About Automated Testing

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps