Congratulations! You know how to build a image, and are able to compose multiple containers into a meaningful . Hopefully, you’ve already created a Continuous Delivery pipeline and know how to push your newly created image into production or testing environment. Docker application Now, the question is How do we test our Docker containers? There are multiple testing strategies we can apply. In this post, I’ll highlight them presenting benefits and drawbacks for each. The “Naive” Approach This is the default approach for most people. It relies on a CI server to do the job. When taking this approach, the developer is using Docker as a package manager, a better option than the approach. The CI server compiles the application code and executes tests (unit, service, functional, and others). The build artifacts are reused in Docker to produce a new image. This becomes a core deployment artifact. The produced image contains not only application “binaries”, but also a required runtime including all dependencies and application configuration. jar/rpm/deb build We are getting application portability, however, we are loosing the development and testing portability. We’re not able to reproduce exactly the same development and testing environment outside the CI. To create a new test environment, we’ll need to setup the testing tools (correct versions and plugins), configure runtime and OS settings, and get the same versions of test scripts as well as perhaps, the test data. To resolve these problems leads us to the next one. App & Test Container Approach Here, we try to create a single bundle with the application “binaries” including required packages, testing tools (specific versions), test tools plugins, test scripts, test environment with all required packages. The of this approach: benefits We have a repeatable test environment — we can run exactly the same tests using the same testing tools — in our CI, development, staging, or production environment We capture test scripts at a specific point in time so we can always reproduce them in any environment We do not need to setup and configure our testing tools — they are part of our image This approach has significant : drawbacks Increases the image size — because it contains testing tools, required packages, test scripts, and perhaps even test data Pollutes image runtime environment with test specific configuration and may even introduce an unneeded dependency (required by integration testing) We also need to decide what to do with the test results and logs; how and where to export them Here’s a simplified . It illustrates this approach. Dockerfile There has to be a better way for in-container testing and there is. Test Aware Container Approach Today, Docker’s promise is Build -> Ship -> Run build the image, ship it to some registry, and run it anywhere. IMHO there’s a critical missing step — . Test The right and complete sequence should be**:** Build -> Test -> Ship -> Run Let’s look at a “test-friendly” Dockerfile syntax and extensions to Docker commands. This important step could be supported natively. It’s not a real syntax, but bear with me. I’ll define the “ideal” version and show how to implement something that’s very close. [INSTRUCTION] ONTEST Let’s define a special instruction, similar to existing instruction. The instruction adds a trigger instruction to the image to be executed at a later time when the image is tested. Any build instruction can be registered as a trigger. ONTEST ONBUILD ONTEST The instruction should be recognized by a new command. ONTEST docker test [OPTIONS] IMAGE [COMMAND] [ARG...] docker test The command syntax will be similar to command, with one significant difference: a new “testable” image will be automatically generated and even tagged with tag (“test” postfix added to the original image tag). This “testable” image will generated the application image, executing all build instructions, defined after command and executing (or ). The docker test command should return a non-zero code if any tests fail. The test results should be written into an automatically generated that points to folder. docker test docker run <image name>:<image tag>-test FROM ONTEST ONTEST CMD ONTEST ENTRYPOINT VOLUME /var/tests/results Let’s look at a modified Dockerfile below — it includes the new proposed instruction. ONTEST Making “Test Aware Container” Real We believe Docker should make part of the container management lifecycle. There is a need to have a simple working solution today and I’ll describe one that’s very close to the ideal state. docker-test As mentioned before, Docker has a very useful instruction. This instruction allows us to trigger another build instruction on succeeding builds. The basic idea is to use instruction when running d command. ONBUILD ONBUILD ocker-test The flow executed by command: docker-test will search for instructions in application Dockerfile and will … docker-test ONBUILD generate a temporary from original Dockerfile.test Dockerfile execute with additional options supported by command: that will be automatically appended to option docker build -f Dockerfile.test [OPTIONS] PATH docker build -test tag If build is successful, execute docker run -v ./tests/results:/var/tests/results [OPTIONS] IMAGE:TAG-test [COMMAND] [ARG…] remove file Dockerfile.test Why not create a new without requiring the instruction? Dockerfile.test ONBUILD Because in order to test right image (and tag) we’ll need to keep always updated to that we want to test. This is not trivial. FROM image:tag There is a limitation in the described approach — it’s not suitable for “onbuild” images (images used to automatically build your app), like Maven:onbuild Let’s look at a simple implementation of command. It highlights the concept: the command should be able to handle build and run command options and be able to handle errors properly. docker-test docker-test Let’s focus on the most interesting and relevant part. Integration Test Container — Testing Strategy Let’s say we have an application built from tens or hundreds of microservices. Let’s say we have an automated CI/CD pipeline, where each microservice is built and tested by our CI and deployed into some environment (testing, staging or production) after the build and tests pass. Pretty cool, eh? Our CI tests are capable of testing each microservice in isolation — running unit and service tests (or API contract tests). Maybe even micro-integration tests — tests run on subsystem are created in ad-hoc manner (for example with help). docker compose This leads to some issues that we need to address: What about real integration tests or long running tests (like performance and stress)? What about resilience tests (“chaos monkey” like tests)? Security scans? What about test and scan activities that take time and should be run on a fully operational system? There should be a better way than just dropping a new microservice version into production and tightly monitoring it for a while. There should be a special . These containers will contain only testing tools and test artifacts: test scripts, test data, test environment configuration, etc. To simplify orchestration and automation of such containers, we should define and follow some conventions and use metadata labels (Dockerfile instruction). Integration Test Container LABEL Integration Test Labels — test type; default integration; can be one of: , , , or any text; the presence of this label states that this is the test.type integration performance security chaos Integration Test Container — VOLUME for test results; default test.results /var/tests/results — any other test related metadata; just use prefix for label name test.XXX test. Integration Test Container The is just a regular Docker container. It does not contain any application logic and code. Its sole purpose is to create repeatable and portable testing. Recommended content of the : Integration Test Container Integration Test Container — Phantom.js, Selenium, Chakram, Gatling, … The Testing Tool — Node.js, JVM, Python, Ruby, … Testing Tool Runtime — environment variables, config files, bootstrap scripts, … Test Environment Configuration — as compiled packages or script files Tests — any kind of data files, used by tests: json, csv, txt, xml, … Test Data — some “main” startup script to run tests; just create test.sh and launch your testing tool from it. Test Startup Script should run in an operational environment where all microservices are deployed: testing, staging or production. These containers can be deployed exactly as all other services. They use same network layer and thus can access multiple services; using the selected service discovery method (usually DNS). Accessing multiple services is required for real integration testing — we need to simulate and validate how our system is working in multiple places. Keeping integration tests inside some application service container not only increases the container footprint, but also creates an unneeded dependency between multiple services. We keep all these dependencies at the level of the . Once our tests (and testing tools) are packaged inside the container, we can always rerun the same tests on any environment including the developer machine. You can always go back in time and rerun a specific version of . Integration Test Containers Integration Test Container Integration Test Container WDYT? Your feedback, particularly on standardizing the command, is greatly appreciated. docker-test Video and slides are available at Codefresh site
