How Automatically Scan Docker Images For Vulnerabilities Using Amazon Elastic Container Registry

When using Docker to run applications security is a major concern, but it can sometimes be easy to forget as we focus first on functionality. This doesn't need to be the case, as AWS Elastic Container Registry (ECR) can now be setup to automatically scan images on push, and provide feedback on any vulnerabilities that need to be addressed.

1) How does ECR image scanning work?

ECR uses the Common Vulnerabilities and Exposures (CVEs) database from the open-source Clair project. Clair performs static scanning of Docker images, meaning that it happens prior to deployment, as opposed to dynamic scanning which happens at runtime.

Clair analyses each layer of the Docker image, then returns vulnerabilities that might affect the image. These vulnerabilities are then shown to us in the AWS Console or we can fetch them via the AWS Command Line Interface (CLI).

Additionally, ECR offers these features:

• automatically scan on pushing an image to ECR
• an event is sent to AWS EventBridge when an image scan completes, allowing for further process automation

2) Scanning images with ECR: hands on

Let's start off by scanning everyone's favourite base image, Alpine Linux. It's a lightweight Linux distribution that is used as a base image for many popular applications in Docker. It's also one of the official approved Docker images. ✅

At the time of writing version 3.11 of Alpine, it was not compatible with ECR image scanning, so we'll use version 3.10.

Pulling the image locally

We'll first pull the Alpine 3.10 image locally:

$ docker pull alpine:3.10
3.10: Pulling from library/alpine
89d9c30c1d48: Pulling fs layer
89d9c30c1d48: Verifying Checksum
89d9c30c1d48: Download complete
89d9c30c1d48: Pull complete
Digest: sha256:c19173c5ada610a5989151111163d28a67368362762534d8a8121ce95cf2bd5a
Status: Downloaded newer image for alpine:3.10
docker.io/library/alpine:3.10

And now it's in our list of images:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             
alpine              3.10                965ea09ff2eb        2 months ago 

Pushing to ECR

Let us use the AWS CLI to create a repository in ECR:

$ aws ecr create-repository --repository-name alpine --image-scanning-configuration scanOnPush=true

Info: we're setting the image-scanning-configuration to enable automatic scanning when we push an image to this repository

As you probably know, to push an image to ECR you need to.

1. Login to ECR
2. Tag the image with your ECR repository which follows the format 

   <aws-account-id>.dkr.ecr.

   region>

   .amazonaws.com/<repository-name>

3. Push the image

So firstly to login, let us run 

$ aws ecr get-login --region <region> --no-include-email

Now let us tag our local Alpine image.

$ docker tag alpine:3.10 <aws-account-id>.dkr.ecr.<region>.amazonaws.com/alpine:3.10

For the source image you can either use the image ID or repository:tag

Now we can push the image:

$ docker push <aws-account-id>.dkr.ecr.<region>.amazonaws.com/alpine:3.10
The push refers to repository [299404798587.dkr.ecr.eu-west-1.amazonaws.com/alpine]
77cae8ab23bf: Preparing
77cae8ab23bf: Pushed
3.10: digest: sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a size: 528

Viewing scan results

At this point we could view the scan results from the API using:

aws ecr describe-image-scan-findings --repository-name <repository-name> --image-id imageTag=<image-tag> --region <region>

Or we can use the AWS Console, which will be a bit easier to read. In the console, we'll go to Services > ECR then select the Alpine repository:

You can see here that Alpine 3.10 doesn't have any known vulnerabilities. They've obviously been keeping everything up-to-date. How about an older version of Alpine?

Scanning Alpine 3.9

If we repeat the above process with Alpine 3.9. i.e.

1. Pull the alpine:3.9 image locally
2. Tag the image with our ECR repository
3. Push the image

We'll see the following result in the AWS Console:

Uh-oh, looks like we've got some vulnerabilities here. ⚠️ Let's drill down and see what the issues are:

Thankfully there is only 1 vulnerability, but it's high priority. Clicking on it we get the full details from cve.mitre.org:

CVE-2019-14697 musl libc through 1.1.23 has an x87 floating-point stack adjustment imbalance, related to the math/i386/ directory. In some cases, use of this library could introduce out-of-bounds writes that are not present in an application's source code.

Interpreting scan results

The docker-library FAQ offers some words of advice, making these main points about vulnerabilities found in official Docker images, of which Alpine is one:

• the CVE may not have been addressed due to it not being deemed a high risk
• or time constraints i.e. not worth back-porting a fix to an old image version
• the CVE could be a false positive. The vulnerability scanner isn't perfect and determines results based on heuristics. It might not actually be possible to trigger the vulnerability in an image.

If we search GitHub for mentions of CVE-2019-14697 with relation to Alpine Linux we can see several comments saying that it's fixed in 3.10 so won't be fixed in 3.9 (i.e. the 2nd point above).

'Images based on Alpine 3.9 don't pass vulnerability scanner' GitHub issue

Let's try scanning OpenJDK Alpine 3.10

Just for fun then, let's try scanning the OpenJDK Docker images that use Alpine 3.9 and 3.10, which we've already scanned. Here are the results:

Interesting to note that Alpine 3.9 and JDK 13 Alpine 3.9 both have the same vulnerability. Open JDK 8 Alpine 3.9 introduces an additional vulnerability.

3) Some recommendations

So even though image vulnerability scanning isn't quite as deterministic as we might like, there are definitely some key takeaways here:

Ensure your application uses the most recent base image

If you're relying on a base image such as Alpine, try to make sure you're using the latest version, as older versions may have vulnerabilities. E.g., if you're relying on OpenJDK, try to use openjdk:14-jdk-alpine3.10.

Setup scanning on push

As you've seen in this post it's easy to setup scanning with ECR. Ensure you scan every image that's pushed to learn about potential vulnerabilities.

Setup continuous scanning

Because the CVE database is being continually updated, a scan may produce different results tomorrow than it did today, for the same image. You could consider automating this process daily, using the 

aws ecr start-image-scan

4) Limits and costing

AWS imposes a limit of one scan per day per image, otherwise, a  hrottlingException gets returned.

Currently, AWS offers ECR scanning for free, so it's a no-brainer to switch on automatic scanning on push.

About the author - Sudip is a Solution Architect with more than 15 years of working experience, and is the founder of Javelynn. He likes sharing his knowledge by regularly writing for Hackernoon, DZone, Appfleet and many more. And while he is not doing that, he must be fishing or playing chess.

Previously posted at https://appfleet.com/.