Congratulations! You know how to build a Docker image, and are able to compose multiple containers into a meaningful application. Hopefully, you’ve already created a Continuous Delivery pipeline and know how to push your newly created image into production or testing environment.
Now, the question is
How do we test our Docker containers?
There are multiple testing strategies we can apply. In this post, I’ll highlight them presenting benefits and drawbacks for each.
This is the default approach for most people. It relies on a CI server to do the job. When taking this approach, the developer is using Docker as a package manager, a better option than the jar/rpm/deb approach. The CI server compiles the application code and executes tests (unit, service, functional, and others). The build artifacts are reused in Docker build to produce a new image. This becomes a core deployment artifact. The produced image contains not only application “binaries”, but also a required runtime including all dependencies and application configuration.
We are getting application portability, however, we are loosing the development and testing portability. We’re not able to reproduce exactly the same development and testing environment outside the CI. To create a new test environment, we’ll need to setup the testing tools (correct versions and plugins), configure runtime and OS settings, and get the same versions of test scripts as well as perhaps, the test data.
To resolve these problems leads us to the next one.
Here, we try to create a single bundle with the application “binaries” including required packages, testing tools (specific versions), test tools plugins, test scripts, test environment with all required packages.
The benefits of this approach:
This approach has significant drawbacks:
Here’s a simplified Dockerfile. It illustrates this approach.
There has to be a better way for in-container testing and there is.
Today, Docker’s promise is
Build -> Ship -> Run
build the image, ship it to some registry, and run it anywhere. IMHO there’s a critical missing step — Test.
The right and complete sequence should be**:**
Build -> Test -> Ship -> Run
Let’s look at a “test-friendly” Dockerfile syntax and extensions to Docker commands. This important step could be supported natively. It’s not a real syntax, but bear with me. I’ll define the “ideal” version and show how to implement something that’s very close.
ONTEST [INSTRUCTION]
Let’s define a special ONTEST instruction, similar to existing ONBUILD instruction. The ONTEST instruction adds a trigger instruction to the image to be executed at a later time when the image is tested. Any build instruction can be registered as a trigger.
The ONTEST instruction should be recognized by a new docker test command.
docker test [OPTIONS] IMAGE [COMMAND] [ARG...]
The docker test command syntax will be similar to docker run command, with one significant difference: a new “testable” image will be automatically generated and even tagged with <image name>:<image tag>-test tag (“test” postfix added to the original image tag). This “testable” image will generated FROM the application image, executing all build instructions, defined after ONTEST command and executing ONTEST CMD (or ONTEST ENTRYPOINT). The docker test command should return a non-zero code if any tests fail. The test results should be written into an automatically generated VOLUME that points to /var/tests/results folder.
Let’s look at a modified Dockerfile below — it includes the new proposed ONTEST instruction.
We believe Docker should make docker-test part of the container management lifecycle. There is a need to have a simple working solution today and I’ll describe one that’s very close to the ideal state.
As mentioned before, Docker has a very useful ONBUILD instruction. This instruction allows us to trigger another build instruction on succeeding builds. The basic idea is to use ONBUILD instruction when running docker-test command.
The flow executed by docker-test command:
Why not create a new Dockerfile.test without requiring the ONBUILD instruction?
Because in order to test right image (and tag) we’ll need to keep FROM always updated to image:tag that we want to test. This is not trivial.
There is a limitation in the described approach — it’s not suitable for “onbuild” images (images used to automatically build your app), like Maven:onbuild
Let’s look at a simple implementation of docker-test command. It highlights the concept: the docker-test command should be able to handle build and run command options and be able to handle errors properly.
Let’s focus on the most interesting and relevant part.
Let’s say we have an application built from tens or hundreds of microservices. Let’s say we have an automated CI/CD pipeline, where each microservice is built and tested by our CI and deployed into some environment (testing, staging or production) after the build and tests pass. Pretty cool, eh? Our CI tests are capable of testing each microservice in isolation — running unit and service tests (or API contract tests). Maybe even micro-integration tests — tests run on subsystem are created in ad-hoc manner (for example with docker compose help).
This leads to some issues that we need to address:
There should be a better way than just dropping a new microservice version into production and tightly monitoring it for a while.
There should be a special Integration Test Container. These containers will contain only testing tools and test artifacts: test scripts, test data, test environment configuration, etc. To simplify orchestration and automation of such containers, we should define and follow some conventions and use metadata labels (Dockerfile LABEL instruction).
The Integration Test Container is just a regular Docker container. It does not contain any application logic and code. Its sole purpose is to create repeatable and portable testing. Recommended content of the Integration Test Container:
Integration Test Containers should run in an operational environment where all microservices are deployed: testing, staging or production. These containers can be deployed exactly as all other services. They use same network layer and thus can access multiple services; using the selected service discovery method (usually DNS). Accessing multiple services is required for real integration testing — we need to simulate and validate how our system is working in multiple places. Keeping integration tests inside some application service container not only increases the container footprint, but also creates an unneeded dependency between multiple services. We keep all these dependencies at the level of the Integration Test Container. Once our tests (and testing tools) are packaged inside the container, we can always rerun the same tests on any environment including the developer machine. You can always go back in time and rerun a specific version of Integration Test Container.
WDYT? Your feedback, particularly on standardizing the docker-test command, is greatly appreciated.
Video and slides are available at Codefresh site