Automated testing has become an essential part of software development.
However, I still very much dislike writing tests and recently, I found out I’m not alone. I posted a poll on Reddit to see if other devs think like me and, thankfully, I’m not alone.
Imagine choosing between:
- writing automated tests for a feature you created
- or improving the feature / starting the implementation of another feature
Most of us would prefer not to write tests but rather work on the core codebase but we know that a disaster awaits us if we don’t have any written tests. So, we just bite the bullet and write them.
All of this got me thinking if all of this is really necessary. Having tests definitely is necessary but do we really have to write them manually?
So, I got to work with my friend and now, 2 months later, we released Pythagora – an open source npm package that creates automated integration tests by recording and analyzing server activity without you having to write a single line of code. Within 1 hour of recording, it can generate tests with 90% code coverage.
In this blog post, I’ll show you how easy it is to use Pythagora compared to traditional ways of writing tests.
If you’re a video person, here’s a quick 2 minute video of how Pythagora works:
https://www.youtube.com/watch?v=Be9ed-JHuQg&embedable=true
The traditional way of writing integration tests
Imagine you want to test an endpoint that returns products that a user has viewed before but hasn’t purchased.
When testing that kind of an endpoint it’s important to first set up the necessary data for the test. This includes having a user in the database along with associated products that the user has viewed but hasn’t purchased.
You want these tests to run from any machine (local machines from other developers or staging/testing environments) so you need to make sure that the data we want to test is actually in the database. Since we don’t know what is the state of the database, usually, the best practice is to set up database conditions before the test is run (using the before function) and clear the database state after each test (using the after function).
let userId, productIds;
before(async () => {
const products = await Product.create([
{ name: 'Product 1' },
{ name: 'Product 2' }
]);
productIds = products.map(p => p._id);
const user = await User.create({
name: 'Bruce Wayne',
email: '[email protected]',
viewedProducts: productIds
});
userId = user._id;
});
In this example, we first create two new products using the Product.create method. Next, we create a new user with the name “Bruce Wayne“, and email “[email protected]” and we associate the created products to this user by adding product ids in the viewedProducts parameter.
Ok, once the setup is done, we can then proceed to test the endpoint that returns the list of products.
test('GET /purchased-products', async () => {
const response = await request(app)
.get(`/viewed-products/${userId}`);
expect(response.statusCode).toBe(200);
expect(response.body).toEqual([
{ name: 'Product 1' },
{ name: 'Product 2' }
]);
});
In this example, the test makes a GET request to the /purchased-products/:userId endpoint with the userId that we created in the setup method. And it uses Jest’s expect function to check that the response has a status code of 200 and that the response body contains the products we created in the setup method.
Test that updates the database
Another very common test scenario is checking if a database update happened. For example, once a user purchases a product, we want to modify its document in the database to reflect that. To make sure that the user collection was updated, we need to query the database and inside the test, compare if the database reflects the wanted change. Here is a full example of such a test.
const request = require("supertest");
const app = require("../app");
const User = require("../models/user");
const mongoose = require("mongoose");
describe("POST /purchase", () => {
let user;
before(async () => {
await mongoose.connection.dropDatabase();
const viewedProductIds = ["productId1", "productId2"];
user = new User({
name: "testuser",
viewedProducts: viewedProductIds,
});
await user.save();
});
test("should purchase a product and update the user", async () => {
const productId = "productId2";
const response = await request(app)
.post("/purchase")
.send({ productId });
expect(response.statusCode).toBe(200);
expect(response.body).toEqual({ message: "Product purchased successfully" });
const updatedUser = await User.findOne({ name: "testuser" });
expect(updatedUser.purchasedProducts).toContain(productId);
});
});
To cover a server’s codebase with tests as much as possible, we need to code a test for each of the different cases for each of the endpoints. This can easily scale to hundreds or even thousands of different tests.
If we estimate that an average dev needs an hour to create a test, it becomes a quite daunting task to cover an entire codebase with tests from scratch.
Using Pythagora
Now imagine if you could create hundreds of tests like this, with setting up the database, making a request, and checking all of the values from the database and the response in only 1 hour!
🏗️ How it works
Pythagora works by capturing API requests (with responses) and server activity during the processing of the request. From that data, it creates automated tests that you can run from any environment/machine.
When an API request is being captured, Pythagora saves all database documents used during the request (before and after each db query).
When you run the test, Pythagora first connects to a temporary pythagoraDb database and restores all saved documents. This way, the database state is the same during the test as it was during the capture so the test can run on any environment while NOT changing your local database. Then, Pythagora makes an API request tracking all db queries and checks if the API response and db documents are the same as they were during the capture.
For example, if the request updates the database – then, after the API returns the response, Pythagora checks the database to see if it was updated correctly.
⚙️ Setup
It’s super simple to set up Pythagora for a Node.js server. Just install the npm package with
npm install pythagora
and add one line of code right after you initialize Express. For example, if you initialize it with const app = express(); – then you add the following line of code.
if (global.Pythagora) global.Pythagora.setApp(app);
That’s it, you’re ready now to start capturing requests.
🎥 Capturing requests
Now, you just need to run the pythagora command and pass the .js file you’re usually running when starting your server. For example, if you run your server with node ./path/to/your/server.js, you would run Pythagora like this:
npx pythagora --initScript ./path/to/your/server.js
This command will start your server but wrapped with Pythagora. Now, to get Pythagora to create automated tests, you just need to send API requests to your server.
You can do this in any way you’re usually sending requests while developing or testing. You can use your browser and click around your frontend, send requests through Postman or cURL or any other way you’re used to.
Pythagora records all requests to endpoints of the app with the response and everything that’s happening during the request. Currently, that means all Mongo and Redis queries with responses (in the future 3rd party API requests, disk IO operations, etc.).
You can see all captured tests with metadata in the pythagora_data folder in the root of your project. By having this folder present, anyone from your team can run these tests regardless of the database they are connected to.
🚦Running tests
When running tests, it doesn’t matter what database is your Node.js connected to or what is the state of that database. Actually, that database is never touched or used —> instead, Pythagora creates a special, ephemeral pythagoraDb database, which it uses to restore the data before each test is executed, which was present at the time when the test was recorded. Because of this, tests can be run on any machine or environment.
If a test does an update to the database, Pythagora also checks the database to see if it was updated correctly.
So, after you captured all requests you want, you just need to add the mode parameter --mode test to the Pythagora command to run the captured requests.
npx pythagora --initScript ./path/to/your/server.js --mode test
📝 Code coverage report
Code coverage is a great metric while building automated tests as it shows us which lines of code are covered by the tests. Pythagora uses nyc to generate a report about code that was covered with Pythagora tests. By default, Pythagora will show you the basic code coverage report summary when you run tests.
If you want to generate a more detailed report, you can do so by running Pythagora with --full-code-coverage-report flag. For example:
npx pythagora --initScript ./path/to/your/server.js --mode test --full-code-coverage-report
You can find the code coverage report inside pythagora_data folder in the root of your repository. You can open the HTML view of the report by opening pythagora_data/code_coverage_report/lcov-report/index.html.
In case you don’t want the code coverage to be shown at all while running tests, you can run the tests with --no-code-coverage parameter.
🎚 Current limitations
Right now, Pythagora is quite limited and it supports only Node.js apps with Mongoose and Express but we’re working hard to extend this support to other technologies and languages. In case Mongoose is not used in the codebase, Pythagora will capture API requests and responses without testing the database.
To support the most important technologies, it would mean a lot to us if you leave a comment or send us an email at [email protected] with technologies you use that you would like to be supported.
Conclusion
Most developers, including me, don’t enjoy the task of creating and maintaining tests. This is the problem we’re trying to solve with Pythagora.
Pythagora is an open-source npm package that can record and analyze server activity to create tests that mimic the behavior of your codebase. Using it, you can save time and effort and generate up to 90% code coverage in as little as one hour. This means you can spend more time working on the core codebase and less time worrying about writing and maintaining tests.
You don’t have to worry about what database your server is connected to, or the state of that database. Pythagora creates a special, ephemeral pythagoraDb database, which it uses to restore the data before each test is executed, which was present at the time when the test was recorded. This means that tests can be run on any machine or environment, which makes testing much more efficient.
Anyway, that’s it. I hope you liked it. We’re still in a super early stage so any feedback is much appreciated.
How do you write integration tests for your API server? Would you consider using Pythagora instead/along with your system?
If not, I’d love to hear what are your concerns and why this wouldn’t work for you?
Also, if you like Pythagora, it would mean a lot if you could give us a star on Github – https://github.com/Pythagora-io/pythagora
To get an update about the beta release or to give a suggestion on tech (framework / database) you want Pythagora to support you can 👉 add your email / comment here 👈