Testing the code you write is an important step in the process of software engineering. It ensures that your software works as expected and reduces the risk of shipping bugs and vulnerabilities to production.
Automated tests, in particular, play an important role when it comes to testing frequently and consistently. Continuous integration makes them even more powerful.
In this blog post, I'll show you an architecture for testing Node.js REST APIs that use a database in the background. There are some things you should consider in this scenario that we'll talk about.
You'll see how to separate and organize your application's components in a way that you can test them independently. Therefore, we'll use two different approaches. On the one hand, we set up a test environment in which we run our tests against a test database.
On the other hand, we mock the database layer using mock functions so that we can run them in an environment in which we have no access to a database.
We'll start by writing unit tests for testing single components of our application. In the next step, we combine those components and test them using integration tests. Last but not least, we set up a CI/CD pipeline using GitHub actions and run the tests on each push that is made.
Note that this is not a guide on how testing in general works. There are thousands of articles about frameworks like Jest, Mocha, Supertest, and so on.
This is more of a guide on how to prepare your Node application and test environment in a way that you can write tests effortlessly and efficiently with or without a database connection. There's also an example repo on GitHub. You should definitely check it out.
Disclamer: I know there really is no right or wrong when it comes to the architecture. The following is my prefered one.
Let's start with the tools we'll use. Most of them should be familiar to you:
- Language: Typescript
- Server: Express
- Database: Postgres
- Testing: Jest & Supertest & Chai
- CI/CD: GitHub Actions
- Containerization: Docker
One benefit of this architecture is that you can use it with other databases than Postgres too, like MySQL for example. In this architecture, we don't use any kind of ORM. Moreover, you can replace Jest with Mocha if that is your desired testing framework.
Application Architecture
The architecture of our application looks roughly like this:
node-api
├── api
│ ├── components
│ │ ├── user
| | │ ├── tests // Tests for each component
| | | │ ├── http.spec.ts
| | | │ ├── mock.spec.ts
| | │ | └── repo.spec.ts
| | │ ├── controller.ts
| | │ ├── dto.ts
| | │ ├── repository.ts
| | │ └── routes.ts
│ └── server.ts
├── factories // Factories to setup tests
| ├── abs.factory.ts
| ├── http.factory.ts
| └── repo.factory.ts
└── app.ts
Note: The example repo contains some more code.
Each component consists of the following four files:
- controller.ts: HTTP Handler
- dto.ts: Data Transfer Object (more)
- repository.ts: Database Layer
- routes.ts: HTTP Routing
The tests directory includes the tests of the according component. If you want to read more about this architecture, check out this article of mine.
Config
We'll start by creating an .env.test file that contains the secret environment variables for testing. The npm Postgres package uses them automatically when establishing a new database connection. All we have to do is to make sure that they are loaded using dotenv.
NODE_PORT=0
NODE_ENV=test
PGHOST=localhost
PGUSER=root
PGPASSWORD=mypassword
PGDATABASE=nodejs_test
PGPORT=5432
Setting NODE_PORT=0 lets Node choose the first randomly available port that it finds. This can be useful if you run multiple instances of an HTTP server during testing. You can also set a fixed value other than 0 here. Using PGDATABASE we provide the name of our test database.
Next, we set up Jest. The config in jest.config.js looks as follows:
module.exports = {
preset: "ts-jest",
testEnvironment: "node",
roots: ["src"],
setupFiles: ["<rootDir>/setup-jest.js"],
}
And setup-jest.js like this:
require("dotenv").config({
path: ".env.test",
})
This snippet ensures that the appropriate environment variables are loaded from the provided .env file before running the tests.
Testing With Database
Let's start with the assumption that we have a test database that we can use. This might be one in a GitHub Actions CI/CD pipeline for example. Later on, I'll show you how to test your application without a database connection.
3 rules
In the beginning, I said that there are some important things we want to consider that make life much easier when testing Node APIs:
- Separate the database layer
- Outsource the database connection initialization
- Outsource the HTTP server initialization
What do I mean by that?
Separate the database layer.
You should have your own layer, separated from your business logic, that takes care of the communication with the database. In the example Git repo, you can find this layer in a component's repository.ts file.
This allows us to easily mock a layer's methods when we have no database available for testing.
Moreover, it's easier to replace your database system with another one.
export class UserRepository {
readAll(): Promise<IUser[]> {
return new Promise((resolve, reject) => {
client.query<IUser>("SELECT * FROM users", (err, res) => {
if (err) {
Logger.error(err.message)
reject("Failed to fetch users!")
} else resolve(res.rows)
})
})
}
Outsource the database connection initialization.
You should already know that when writing tests against a database, you do not run them against the production one. Instead, you set up a test database. Otherwise, you run the risk of messing up your production data.
Most of the time, your application connects to the database in a startup script, like index.js. After the connection is established, you start the HTTP server.
That's what we want to do in our test setup too. This way, we can connect to the database and disconnect from it gracefully before and after each test case.
Outsource the HTTP server initialization.
It's a good practice, whether you use a database or not, to start the HTTP server from inside your tests. Just as we do for the database connection, we create a new HTTP server before and stop it after each test case.
This might look as follows: (you'll see the concrete implementation later on)
describe("Component Test", () => {
beforeEach(() => {
// Connect to db pool && start Express Server
});
afterEach(() => {
// Release db pool client && stop Express Server
});
afterAll(() => {
// End db pool
});
In particular, the execution order is:
- Connect to the database pool
- Run the SQL seed (create tables)
- Start the Express server
- Run the tests
- Shutdown the Express server & release db pool client
- Repeat steps 1 - 5 for each test case and close the pool at the end
Tests
Each component consists of two test files:
- repo.spec.ts
- http.spec.ts
Both of them make use of so-called TestFactories which prepares the test setup. You'll see their implementation in the next chapter.
Note: If you have a look at the example Git repo, you'll see that there are two more: mock.spec.ts and dto.spec.ts. Former one is discussed later on. The latter is not covered in this article.
repo.spec.ts
A repository is an additional abstract layer that is responsible for interacting with the database like reading and inserting new data. That layer is what we test in here.
Since a database is required in this case, a new pool client is created to connect to the database using the RepoTestFactory before each test case. And it is released, right after the test case is completed. In the end, when all test cases are finished, the pool connection is closed.
Example on GitHub
describe("User component (REPO)", () => {
const factory: RepoTestFactory = new RepoTestFactory()
const dummyUser = new UserDTO("[email protected]", "johndoe")
// Connect to pool
beforeEach(done => {
factory.prepare(done)
})
// Release pool client
afterEach(() => {
factory.closeEach()
})
// End pool
afterAll(done => {
factory.close(done)
})
test("create user", () => {
const repo: UserRepository = new UserRepository()
repo.create(dummyUser).then(user => {
expect(user).to.be.an("object")
expect(user.id).eq(1)
expect(user.email).eq(dummyUser.email)
expect(user.username).eq(dummyUser.username)
})
})
})
http.spec.ts
Here, we test the integration of the user component's routes, controller, and repository. Before each test case, a new pool client is created just as we did above. In addition, a new Express server is started using the HttpTestFactory. In the end, both are closed again.
Example on GitHub
describe("User component (HTTP)", () => {
const factory: HttpTestFactory = new HttpTestFactory()
const dummyUser = new UserDTO("[email protected]", "johndoe")
// Connect to pool && start Express Server
beforeEach(done => {
factory.prepare(done)
})
// Release pool client && stop Express Server
afterEach(done => {
factory.closeEach(done)
})
// End pool
afterAll(done => {
factory.close(done)
})
test("create user", async () => {
const postRes = await factory.app
.post("/users")
.send(dummyUser)
.expect(201)
.expect("Content-Type", /json/)
const postResUser: IUser = postRes.body
expect(postResUser).to.be.an("object")
expect(postResUser.id).eq(1)
expect(postResUser.email).eq(dummyUser.email)
expect(postResUser.username).eq(dummyUser.username)
})
})
Factories
The test factories are actually the heart of our tests. They are responsible for setting up and preparing the environment for each test case. That includes:
- Dropping & creating all db tables
- Initializing the db connection
- Initializing the HTTP server
- Closing both of them again
There are four factories in total: AbsTestFactory, RepoTestFactory, HttpTestFactory, and MockTestFactory. Each of them has its own Typescript class. The last one is discussed in the chapter "Testing without database".
AbsTestFactory
The first one AbsTestFactory is an abstract base class that is implemented by the other three. It includes, among others, a method for connecting to the database pool and one for disconnecting from it.
export abstract class AbsTestFactory implements ITestFactory {
private poolClient: PoolClient
private seed = readFileSync(
join(__dirname, "../../db/scripts/create-tables.sql"),
{
encoding: "utf-8",
}
)
abstract prepareEach(cb: (err?: Error) => void): void
abstract closeEach(cb: (err?: Error) => void): void
protected connectPool(cb: (err?: Error) => void) {
pool
.connect()
.then(poolClient => {
this.poolClient = poolClient
this.poolClient.query(this.seed, cb)
})
.catch(cb)
}
protected releasePoolClient() {
this.poolClient.release(true)
}
protected endPool(cb: (err?: Error) => void) {
pool.end(cb)
}
}
Using the create-tables.sql script, the factory drops and recreates all the tables after the connection is established:
DROP TABLE IF EXISTS users;
CREATE TABLE users (
id SERIAL PRIMARY KEY,
email VARCHAR(50) UNIQUE NOT NULL,
username VARCHAR(30) UNIQUE NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
RepoTestFactory
The RepoTestFactory is used by each component's repository test (repo.spec.ts) that you just saw above. All it does is use the parent class AbsTestFactory to connect to the database.
export class RepoTestFactory extends AbsTestFactory {
prepareEach(cb: (err?: Error) => void) {
this.connectPool(cb)
}
closeEach() {
this.releasePoolClient()
}
closeAll(cb: (err?: Error) => void) {
this.endPool(cb)
}
}
The methods prepareEach, closeEach, and closeAll, are called for each test case in the Jest beforeEach, afterEach, and afterAll lifecycle.
HttpTestFactory
The last one, HttpTestFactory, is used by each component's HTTP test (http.spec.ts). Just like RepoTestFactory, it uses the parent class for the database connection. Furthermore, it initializes the Express server.
export class HttpTestFactory extends AbsTestFactory {
private readonly server: Server = new Server()
private readonly http: HttpServer = createServer(this.server.app)
get app() {
return supertest(this.server.app)
}
prepareEach(cb: (err?: Error) => void) {
this.connectPool(err => {
if (err) return cb(err)
this.http.listen(process.env.NODE_PORT, cb)
})
}
closeEach(cb: (err?: Error) => void) {
this.http.close(err => {
this.releasePoolClient()
cb(err)
})
}
closeAll(cb: (err?: Error) => void) {
this.endPool(cb)
}
}
Rewind
Let's jump back to the repo.spec.ts and http.spec.ts test files from above. In both of them, we used the factories' prepareEach method before each and its afterEach method after right each test case.
The closeAll method is called at the very end of the test file. As you have just seen, depending on the type of factory, we establish the database connection and start the HTTP server if needed.
describe("Component Test", () => {
beforeEach(done => {
factory.prepareEach(done)
})
afterEach(() => {
factory.closeEach()
})
afterAll(done => {
factory.closeAll(done)
})
})
One important thing you should keep in mind is that for each test case that uses the database, the factory drops all the tables and recreates them using the provided SQL script afterward. This way we have a clean database with empty tables in each test case.
Testing Without Database
So far, we have run our tests against a test database, but what if we have no access to a database? In this case, we need to mock our database layer implementation (repository.ts), which is quite easy if you have separated it from the business logic, as I recommended in rule #1.
With mocks, the layer does not depend on an external data source anymore. Instead, we provide a custom implementation for the class and each of its methods. Be aware that this does not affect the behavior of our controller since it does not care about where the data comes from.
Example on GitHub
const dummyUser: IUser = {
id: 1,
email: "[email protected]",
username: "john",
created_at: new Date(),
}
// Mock methods
const mockReadAll = jest.fn().mockResolvedValue([dummyUser])
// Mock repository
jest.mock("../repository", () => ({
UserRepository: jest.fn().mockImplementation(() => ({
readAll: mockReadAll,
})),
}))
After mocking the database layer, we can write our tests as usual. Using toHaveBeenCalledTimes(), we make sure that our custom method implementation has been called.
describe("User component (MOCK)", () => {
const factory: MockTestFactory = new MockTestFactory()
// Start Express Server
beforeEach(done => {
factory.prepareEach(done)
})
// Stop Express Server
afterEach(done => {
factory.closeEach(done)
})
test("get users", async () => {
const getRes = await factory.app
.get("/users")
.expect(200)
.expect("Content-Type", /json/)
const getResUsers: IUser[] = getRes.body
cExpect(getResUsers).to.be.an("array")
cExpect(getResUsers.length).eq(1)
const getResUser = getResUsers[0]
cExpect(getResUser).to.be.an("object")
cExpect(getResUser.id).eq(dummyUser.id)
cExpect(getResUser.email).eq(dummyUser.email)
cExpect(getResUser.username).eq(dummyUser.username)
expect(mockReadAll).toHaveBeenCalledTimes(1)
})
})
Note: cExpect is a named import from the "chai" package.
MockTestFactory
Just as we did in the other tests files, we use a test factory here as well. All the MockTestFactory does is run a new Express HTTP instance. It does not establish a database connection since we mock the database layer.
export class MockTestFactory extends AbsTestFactory {
private readonly server: Server = new Server()
private readonly http: HttpServer = createServer(this.server.app)
get app() {
return supertest(this.server.app)
}
prepareEach(cb: (err?: Error) => void) {
this.http.listen(process.env.NODE_PORT, cb)
}
closeEach(cb: (err?: Error) => void) {
this.http.close(cb)
}
}
One drawback we have using this approach is that the layer (repository.ts) is not tested at all because we overwrite it. Nevertheless, we can still test the rest of our application, like the business logic for example. Great!
Running
Using the commands below, we can run the tests with or without a database. Depending on the scenario, the files we do not want to test are excluded from execution.
{
"test:db": "jest --testPathIgnorePatterns mock.spec.ts",
"test:mock": "jest --testPathIgnorePatterns \"(repo|http).spec.ts\""
}
GitHub Actions
The final step is to create a CI/CD pipeline using GitHub actions that runs our tests. The according yaml file is available here. There's also a very good tutorial published on GitHub.
You can decide whether to run the tests against a test database or use the mocked data layer. I decided to go with the former.
When running the pipeline with a test database, we need to make sure that we set the correct environment variables for it. Here you can find a test run.
Last Words
My last tip is to have a look at the example repository on GitHub and to read it carefully There are some more tests and code snippets that I did not cover in this article. Moreover, check out the links below. Happy coding!