How to secure container image?

2020-07-16|By Kamil Zabielski|Security

How to build a secure Docker image? The biggest goal of this article is to be a comprehensive guide on building and delivering secure and safe container images. Keeping that in mind, we'll try to focus on build-time, therefore we won't cover registry, orchestrator and runtime protection here. Those are so broad, that they deserve completely separate write-ups of their own.

Compliance, standards and order are the key, to organized and relatively secure environment. Every organization's environment, its threats and malicious actors are different. Our intention is to present the mindset for container images security; some terms might be very global, others may strictly apply to specific projects and processes. Having that said, in the following points we will cover some terms that might not be related to your specific environment, dear Reader, however, we still hope you will find this guidance useful.


  • Use as much stateless and ephemeral characteristics as possible.
  • Build with --no-cache.
  • Build with --pull.
  • Do not inject secrets in the image.
  • Squash images after build time.
  • Use multi-stage builds to minimize size and installed packages.
  • Use non-root user.
  • Use .dockerignore.
  • Do not use latest tag.
  • Do not use public registries.
  • Sign and verify signatures of the image.
  • Analyze the Dockerfile code statically.
  • Analyze compliance and security of images.

Safety of characteristics

In general, we divide containers into two subsets: stateless and stateful. Mostly, we define, that container is stateless if and only if, it does not require to save any kind of state to local storage or any other block-device. This requirement is still valid for cache, sessions and assets. The more stateless the container is, the better, and we can cut out more capabilities, resources and syscalls out of its runtime.

Try to convince the operators and developers that ephemerality is one the biggest containers advantages. More stateless is better, for all of you.

Build with --no-cache

Cache is a nightmare, when speaking from the security and privacy standpoint; at the same time, it happens to be worth its weight in gold when it comes to performance.

Focusing on security, we highly recommend building every image in the infrastructure without cache. It is strictly related to the container platform layers approach. Take a look at the following example:

A simple nginx image

FROM alpine
LABEL maintainer="Kamil Zabielski <[email protected]>"
RUN apk add nginx
[...]

The responsibility of this image is very simple. Run nginx

. If we take a look at the logs of the build, we will see the log at the end of this section. If we run the following build again, on the same host, few hours later, even if remote repositories have changed — it will use cached layers, because none of the commands has changed for the time being. Hence, security updates will not be installed, which leads to image insecurity.

A buildtime log for nginx

≫ docker build -t limakzi/example:1.0.0 .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM alpine:3.9
3.9: Pulling from library/alpine
9123ac7c32f7: Pull complete
Digest: sha256:115731bab0862031b44766733890091c17924f9b7781b79997f5f163be262178
Status: Downloaded newer image for alpine:3.9
 ---> 82f67be598eb
Step 2/3 : LABEL maintainer="Kamil Zabielski <[email protected]>"
 ---> Running in 2ce04bd78dbb
Removing intermediate container 2ce04bd78dbb
 ---> 71c106f92525
Step 3/3 : RUN apk add nginx
 ---> Running in 9a4e2f47ee78
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/2) Installing pcre (8.42-r1)
(2/2) Installing nginx (1.14.2-r5)
Executing nginx-1.14.2-r5.pre-install
Executing busybox-1.29.3-r10.trigger
OK: 7 MiB in 16 packages
Removing intermediate container 9a4e2f47ee78
 ---> 3f7d600acd18
Successfully built 3f7d600acd18
Successfully tagged limakzi/example:1.0.0
≫

Use --pull at build time

Similarly, we highly recommend using --pull parameter. Orchestrators happen to make this functionality easy to use.1 It is still worth to remember about it during build time, because for some of the delivery systems, docker is invoked directly, without orchestrator.

Use multi stage builds

Some applications require compilers and external libraries or SDKs, but only during build time. (Like numpy for Python or dotnet) For the others, the final product of the build is a binary file, statically linked application. For each of these use cases, properly used multi-stage builds minimize attack surfaces, sizes, and deployment times. Let's take a look at the example:

An example of multi stage dotnet

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish -c Release -o out


FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "aspnetapp.dll"]

Non root application user

Container runtime has an isolation functionality, called user namespace remapping

. Security is redundancy. If application does not have to run as user with id==0, it should not run as this user. Assuming a malicious actor would access the container and escape from it , the attacker gains root privileges on the host immediately.

Sign the image

For the enterprise environment, having an internal public-key infrastructure is nothing special. We can apply public-key security to container images as well. We can sign a built container image with the key. The following approach prevents image poisoning or image overwriting by the attacker. Take a look at this example:

An example of pulling signed and non signed image

≫ docker pull nginx
Using default tag: latest
Pull (1 of 1): nginx:[email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b: Pulling from library/nginx
Digest: sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Status: Image is up to date for [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Tagging [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b as nginx:latest
docker.io/library/nginx:latest
≫

≫ docker pull mysqlboy/mydumper
Using default tag: latest
Error: remote trust data does not exist for docker.io/mysqlboy/mydumper: notary.docker.io does not have trust data for docker.io/mysqlboy/mydumper
≫

In the example above, we tried to pull nginx:latest and mysqlboy/mydumper:latest images. As you can see, only the nginx image was successfully pulled. mysqldumper was rejected, because there was no signature. The most common tool that provides image signing is Notary

.

Use .dockerignore

One of the most common mistakes, we see among developers and some operators, is a recursive COPY strategy.

A recursive copy

COPY * /srv/app

Such recursive copy instruction can cause a lot of problems, starting from performance and build instabilities, through internal information leak, ending with complete application and infrastructure takeover. We highly recommend properly using .dockerignore.

.dockerignore is very similar to the .gitignore , but made for Docker. The operator or developer instructs Docker not to copy the specific paths accordingly to the context of runtime. The most common list of .dockerignore content is the following.

A minimum list for `.dockerignore`

Dockerfile
docker-compose.yml
docker-compose.yaml
LINCENSE.txt
Makefile
.git

Of course, this is just a basic example, not very comprehensive. Each project might has its own entries, like the following example with Python.

A minimum list for `.dockerignore`

__pycache__
*.pyc
*.pyo
*.pyd
.Python
env

Another approach is to make a policy and prevent recursive-copies in your applications while auditing, and to always require a direct, absolute path. 2

Squash images after build-time

Image layers are one of the biggest threats. Very often developers and operators leave very sensitive information there. Let's modify our nginx Dockerfile.

An example of multi layered image

FROM alpine:3.9
LABEL maintainer="Kamil Zabielski <[email protected]>"
RUN apk add nginx
RUN echo "fizz"
RUN echo "buzz"

We've added two more steps, echoing fizz and buzz. If the attacker gains the access to the image, per se, he is able to reverse the whole build time with exactly one command.

A quick reverse engineering of image build commands

≫ docker history --format "{{.ID}}: {{.CreatedBy}}" limakzi/example:1.0.0 | tail -r
<missing>: /bin/sh -c #(nop) ADD file:f4f85ec73d7cc9496…
82f67be598eb: /bin/sh -c #(nop)  CMD ["/bin/sh"]
71c106f92525: /bin/sh -c #(nop)  LABEL maintainer=Kamil Za…
3f7d600acd18: /bin/sh -c apk add nginx
9486a1a6d8ed: /bin/sh -c echo "fizz"
edb670dd828e: /bin/sh -c echo "buzz"
≫

Let's use docker-squash

to mitigate this information leak.3

Squashing a multi-layer image

≫ docker-squash  limakzi/example:1.0.0
2020-03-26 02:40:27,850 root         INFO     docker-squash version 1.0.8, Docker afacb8b, API 1.40...
2020-03-26 02:40:27,850 root         INFO     Using v2 image format
2020-03-26 02:40:27,860 root         INFO     Old image has 6 layers
2020-03-26 02:40:27,860 root         INFO     Checking if squashing is necessary...
2020-03-26 02:40:27,860 root         INFO     Attempting to squash last 6 layers...
2020-03-26 02:40:27,860 root         INFO     Saving image sha256:edb670dd828e1ed5722182c56ec73069c25f72897781e2b75243e701da7ac9ef to /var/folders/qh/sylx84mx21z5q3trnt6hd9tw0000gp/T/docker-squash-7yzrcz54/old directory...
2020-03-26 02:40:28,042 root         INFO     Image saved!
2020-03-26 02:40:28,042 root         INFO     Squashing image 'limakzi/example:1.0.0'...
2020-03-26 02:40:28,043 root         INFO     Starting squashing...
2020-03-26 02:40:28,043 root         INFO     Squashing file '/var/folders/qh/sylx84mx21z5q3trnt6hd9tw0000gp/T/docker-squash-7yzrcz54/old/074c599ca3ae41207a2376e1af24f9b96ec58528aff9da690b1726f6df564cef/layer.tar'...
2020-03-26 02:40:28,060 root         INFO     Squashing file '/var/folders/qh/sylx84mx21z5q3trnt6hd9tw0000gp/T/docker-squash-7yzrcz54/old/76fd8bf97c43091059e090842f65663e6ab8b6008d2ba2b66aab87539c6e9248/layer.tar'...
2020-03-26 02:40:28,141 root         INFO     Squashing finished!
2020-03-26 02:40:28,171 root         INFO     Original image size: 8.20 MB
2020-03-26 02:40:28,171 root         INFO     Squashed image size: 8.17 MB
2020-03-26 02:40:28,171 root         INFO     Image size decreased by 0.39 %
2020-03-26 02:40:28,171 root         INFO     New squashed image ID is 1fb82410d6da35a23981f6d813ef2ff5b80804ca8d928f1c2e2c63451559afe0
2020-03-26 02:40:28,623 root         INFO     Done
≫

≫ docker history 1fb82410d6da35a23981f6d813ef2ff5b80804ca8d928f1c2e2c63451559afe0
IMAGE               CREATED             CREATED BY          SIZE                COMMENT
1fb82410d6da        17 seconds ago                          8.26MB
≫

Non-public images

It is very tempting to use public image, like php-fpm:7.3

. Many developers and operators would say that it is an official container image of the php-fpm, maintainers will update it as soon as the vulnerability appears - and they would be right… in a perfect utopian world. In reality, they are terribly wrong. Even at the first glance, the contents of some of the official images immediately expose a set of mistakes and inaccuracies. More over, using the public image renders proper compliance and implementation of the appropriate standards impossible. Public images and repositories tend to be a perfect target and tend to have outages . Our recommendation may sound cumbersome, but it really neither hard nor expensive to implement. Build internal images and keep them up to date.

No latest tag

Using latest tag is a very common problem, linked to outages, downtimes and problems with operationability of the infrastructure. It is a direct cause of many issues related to the unintended upgrades. Having that said, it can be linked directly to instability of the processes. Using lastest" tag at production is an immediate, clear way for the attacker to poison the image. We highly recommend not to use latest tag at any point of the build, even and especially for internal and compliant images. Proper Continuous Integration / Continuous Deployment processes provide most of the benefits coming from using the latest tag.

Static code analysis - Dockerfile

One of of the most common tools to statically analyze Dockerfile is hadolint

. There is no need to install hadolint on the host machine. You can just grab the hadolint image, run the container, and inject your Dockerfile.

An example of hadolint scan

≫ docker run --rm -i hadolint/hadolint < Dockerfile
Unable to find image 'hadolint/hadolint:latest' locally
latest: Pulling from hadolint/hadolint
c9b1b535fdd9: Pull complete
e7349a44e6b8: Pull complete
Digest: sha256:b309229a6e14f9a92d407c11718d0bb082dc642b9b1fe59937ec23626d48feb4
Status: Downloaded newer image for hadolint/hadolint:latest
/dev/stdin:5 DL3018 Pin versions in apk add. Instead of `apk add <package>` use
`apk add <package>=<version>`
/dev/stdin:5 DL3019 Use the `--no-cache` switch to avoid the need to use
`--update` and remove `/var/cache/apk/*` when done installing packages
≫

Vulnerabilities analysis

To show you, why all of this is so important, we will try to prove that the newest, fresh and official nginx image taken directly from Dockerhub is very vulnerable. Let's download nginx:latest.

Pulling new, fresh nginx image

≫ export DOCKER_CONTENT_TRUST=1
≫
≫ docker pull nginx
Using default tag: latest
Pull (1 of 1): nginx:[email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b: Pulling from library/nginx
Digest: sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Status: Image is up to date for [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Tagging [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b as nginx:latest
docker.io/library/nginx:latest
≫

Now, let's scan the same, may I remind you, "newest and fresh" image, with aquasec/trivy

.

A security test of nginx image

≫ trivy --light --severity CRITICAL,HIGH  nginx:latest
2020-03-26T01:25:44.415+0100  WARN  You should avoid using the :latest tag as it is cached. You need to specify '--clear-cache' option when :latest image is changed
2020-03-26T01:25:44.417+0100  INFO  Need to update DB
2020-03-26T01:25:44.417+0100  INFO  Downloading DB...
3.96 MiB / 3.96 MiB [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2.95 MiB p/s 2s
2020-03-26T01:25:47.463+0100  INFO  Reopening DB...
2020-03-26T01:25:47.560+0100  INFO  Detecting Debian vulnerabilities...

nginx:latest (debian 10.3)
==========================
Total: 15 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 13, CRITICAL: 2)

+-----------------+------------------+----------+---------------------------+---------------+
|     LIBRARY     | VULNERABILITY ID | SEVERITY |     INSTALLED VERSION     | FIXED VERSION |
+-----------------+------------------+----------+---------------------------+---------------+
| bash            | CVE-2019-18276   | HIGH     | 5.0-4                     |               |
+-----------------+------------------+          +---------------------------+---------------+
| libc-bin        | CVE-2019-1010022 |          | 2.28-10                   |               |
+                 +------------------+          +                           +---------------+
|                 | CVE-2020-1752    |          |                           |               |
+-----------------+------------------+          +                           +---------------+
| libc6           | CVE-2019-1010022 |          |                           |               |
+                 +------------------+          +                           +---------------+
|                 | CVE-2020-1752    |          |                           |               |
+-----------------+------------------+----------+---------------------------+---------------+
| libjpeg62-turbo | CVE-2019-2201    | CRITICAL | 1:1.5.2-2                 |               |
+-----------------+------------------+----------+---------------------------+---------------+
| libpcre3        | CVE-2017-11164   | HIGH     | 2:8.39-12                 |               |
+-----------------+------------------+          +---------------------------+---------------+
| libseccomp2     | CVE-2019-9893    |          | 2.3.3-4                   |               |
+-----------------+------------------+          +---------------------------+---------------+
| libsystemd0     | CVE-2020-1712    |          | 241-7~deb10u3             |               |
+-----------------+------------------+          +---------------------------+---------------+
| libtasn1-6      | CVE-2018-1000654 |          | 4.13-3                    |               |
+-----------------+------------------+          +---------------------------+---------------+
| libtiff5        | CVE-2017-9117    |          | 4.1.0+git191117-2~deb10u1 |               |
+-----------------+------------------+          +---------------------------+---------------+
| libudev1        | CVE-2020-1712    |          | 241-7~deb10u3             |               |
+-----------------+------------------+          +---------------------------+---------------+
| libwebp6        | CVE-2016-9085    |          | 0.6.1-2                   |               |
+-----------------+------------------+          +---------------------------+---------------+
| nginx           | CVE-2013-0337    |          | 1.17.8-1~buster           |               |
+-----------------+------------------+----------+---------------------------+---------------+
| tar             | CVE-2005-2541    | CRITICAL | 1.30+dfsg-6               |               |
+-----------------+------------------+----------+---------------------------+---------------+
≫

A security test of nginx image

≫ trivy --light --severity CRITICAL,HIGH  php:7.3-fpm
2020-03-26T10:51:43.787+0100  INFO  Detecting Debian vulnerabilities...

php:7.3-fpm (debian 10.3)
=========================
Total: 52 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 48, CRITICAL: 4)

+---------------------------+------------------+----------+-------------------+---------------+
|          LIBRARY          | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |
+---------------------------+------------------+----------+-------------------+---------------+
| bash                      | CVE-2019-18276   | HIGH     | 5.0-4             |               |
+---------------------------+------------------+          +-------------------+---------------+
| binutils                  | CVE-2017-13716   |          | 2.31.1-16         |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2018-12699   |          |                   |               |
+---------------------------+------------------+          +                   +---------------+
| binutils-common           | CVE-2017-13716   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2018-12699   |          |                   |               |
+---------------------------+------------------+          +                   +---------------+
| binutils-x86-64-linux-gnu | CVE-2017-13716   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2018-12699   |          |                   |               |
+---------------------------+------------------+          +                   +---------------+
| libbinutils               | CVE-2017-13716   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2018-12699   |          |                   |               |
+---------------------------+------------------+          +-------------------+---------------+
| libc-bin                  | CVE-2019-1010022 |          | 2.28-10           |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2020-1752    |          |                   |               |
+---------------------------+------------------+          +                   +---------------+
| libc-dev-bin              | CVE-2019-1010022 |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2020-1752    |          |                   |               |
+---------------------------+------------------+          +                   +---------------+
| libc6                     | CVE-2019-1010022 |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2020-1752    |          |                   |               |
+---------------------------+------------------+          +                   +---------------+
| libc6-dev                 | CVE-2019-1010022 |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2020-1752    |          |                   |               |
+---------------------------+------------------+          +-------------------+---------------+
| libpcre3                  | CVE-2017-11164   |          | 2:8.39-12         |               |
+---------------------------+------------------+          +-------------------+---------------+
| libseccomp2               | CVE-2019-9893    |          | 2.3.3-4           |               |
+---------------------------+------------------+          +-------------------+---------------+
| libsystemd0               | CVE-2020-1712    |          | 241-7~deb10u3     |               |
+---------------------------+------------------+          +-------------------+---------------+
| libtasn1-6                | CVE-2018-1000654 |          | 4.13-3            |               |
+---------------------------+------------------+          +-------------------+---------------+
| libudev1                  | CVE-2020-1712    |          | 241-7~deb10u3     |               |
+---------------------------+------------------+----------+-------------------+---------------+
| linux-libc-dev            | CVE-2019-19813   | CRITICAL | 4.19.98-1         |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19814   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19816   |          |                   |               |
+                           +------------------+----------+                   +---------------+
|                           | CVE-2008-4609    | HIGH     |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2013-7445    |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2018-20669   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-12456   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-12615   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-16229   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-16230   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-16231   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-16232   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-16233   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-16234   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-18814   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19054   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19061   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19064   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19067   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19070   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19072   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19074   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19082   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19083   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2019-19815   |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2020-0041    |          |                   |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2020-1749    |          |                   |               |
+---------------------------+------------------+          +-------------------+---------------+
| m4                        | CVE-2008-1687    |          | 1.4.18-2          |               |
+                           +------------------+          +                   +---------------+
|                           | CVE-2008-1688    |          |                   |               |
+---------------------------+------------------+----------+-------------------+---------------+
| tar                       | CVE-2005-2541    | CRITICAL | 1.30+dfsg-6       |               |
+---------------------------+------------------+----------+-------------------+---------------+
≫

As any attentive Reader will notice, we were looking for CRITICAL and MEDIUM severities only. Of course, CVEs might have a fixed version field empty. It may be unfilled for various reasons; some CVEs are an expected behaviour of the software, like CVE-2005-2541; some CVEs were not fixed yet, like CVE-2019-18276; others are just candidates, like CVE-2020-1752.


  1. Particularly, we think about policies and admission controllers in Kubernetes platform.
  2. In later parts of the guide, we're presenting automation tools to achieve it.
  3. Unfortunately, squashing as an internal process by docker without external tools, at the moment of writing this article, seems to be still experimental.

References

LinkedInLinkedInLinkedIn
Kamil Zabielski photo

About the author

Kamil Zabielski