How to build a secure Docker image? The biggest goal of this article is to be a comprehensive guide on building and delivering secure and safe container images. Keeping that in mind, we'll try to focus on build-time, therefore we won't cover registry, orchestrator and runtime protection here. Those are so broad, that they deserve completely separate write-ups of their own.
Compliance, standards and order are the key, to organized and relatively secure environment. Every organization's environment, its threats and malicious actors are different. Our intention is to present the mindset for container images security; some terms might be very global, others may strictly apply to specific projects and processes. Having that said, in the following points we will cover some terms that might not be related to your specific environment, dear Reader, however, we still hope you will find this guidance useful.
- Use as much stateless and ephemeral characteristics as possible.
- Build with
--no-cache. - Build with
--pull. - Do not inject secrets in the image.
- Squash images after build time.
- Use multi-stage builds to minimize size and installed packages.
- Use non-root user.
- Use
.dockerignore. - Do not use
latesttag. - Do not use public registries.
- Sign and verify signatures of the image.
- Analyze the
Dockerfilecode statically. - Analyze compliance and security of images.
Safety of characteristics
In general, we divide containers into two subsets: stateless and stateful. Mostly, we define, that container is stateless if and only if, it does not require to save any kind of state to local storage or any other block-device. This requirement is still valid for cache, sessions and assets. The more stateless the container is, the better, and we can cut out more capabilities, resources and syscalls out of its runtime.
Try to convince the operators and developers that ephemerality is one the biggest containers advantages. More stateless is better, for all of you.
Build with --no-cache
Cache is a nightmare, when speaking from the security and privacy standpoint; at the same time, it happens to be worth its weight in gold when it comes to performance.
Focusing on security, we highly recommend building every image in the infrastructure without cache. It is strictly related to the container platform layers approach. Take a look at the following example:
A simple nginx image
FROM alpine
LABEL maintainer="Kamil Zabielski <[email protected]>"
RUN apk add nginx
[...]
The responsibility of this image is very simple. Run nginx
A buildtime log for nginx
≫ docker build -t limakzi/example:1.0.0 .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM alpine:3.9
3.9: Pulling from library/alpine
9123ac7c32f7: Pull complete
Digest: sha256:115731bab0862031b44766733890091c17924f9b7781b79997f5f163be262178
Status: Downloaded newer image for alpine:3.9
---> 82f67be598eb
Step 2/3 : LABEL maintainer="Kamil Zabielski <[email protected]>"
---> Running in 2ce04bd78dbb
Removing intermediate container 2ce04bd78dbb
---> 71c106f92525
Step 3/3 : RUN apk add nginx
---> Running in 9a4e2f47ee78
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/2) Installing pcre (8.42-r1)
(2/2) Installing nginx (1.14.2-r5)
Executing nginx-1.14.2-r5.pre-install
Executing busybox-1.29.3-r10.trigger
OK: 7 MiB in 16 packages
Removing intermediate container 9a4e2f47ee78
---> 3f7d600acd18
Successfully built 3f7d600acd18
Successfully tagged limakzi/example:1.0.0
≫
Use --pull at build time
Similarly, we highly recommend using --pull parameter. Orchestrators happen to make this functionality easy to use.1 It is still worth to remember about it during build time, because for some of the delivery systems, docker is invoked directly, without orchestrator.
Use multi stage builds
Some applications require compilers and external libraries or SDKs, but only during build time. (Like numpy for Python or dotnet) For the others, the final product of the build is a binary file, statically linked application. For each of these use cases, properly used multi-stage builds minimize attack surfaces, sizes, and deployment times. Let's take a look at the example:
An example of multi stage dotnet
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish -c Release -o out
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "aspnetapp.dll"]
Non root application user
Container runtime has an isolation functionality, called user namespace remapping
. Security is redundancy. If application does not have to run as user withid==0, it should not run as this user. Assuming a malicious actor would access the container and escape from it , the attacker gains root privileges on the host immediately.Sign the image
For the enterprise environment, having an internal public-key infrastructure is nothing special. We can apply public-key security to container images as well. We can sign a built container image with the key. The following approach prevents image poisoning or image overwriting by the attacker. Take a look at this example:
An example of pulling signed and non signed image
≫ docker pull nginx
Using default tag: latest
Pull (1 of 1): nginx:[email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b: Pulling from library/nginx
Digest: sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Status: Image is up to date for [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Tagging [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b as nginx:latest
docker.io/library/nginx:latest
≫
≫ docker pull mysqlboy/mydumper
Using default tag: latest
Error: remote trust data does not exist for docker.io/mysqlboy/mydumper: notary.docker.io does not have trust data for docker.io/mysqlboy/mydumper
≫
In the example above, we tried to pull nginx:latest and mysqlboy/mydumper:latest images. As you can see, only the nginx image was successfully pulled. mysqldumper was rejected, because there was no signature. The most common tool that provides image signing is Notary
Use .dockerignore
One of the most common mistakes, we see among developers and some operators, is a recursive COPY strategy.
A recursive copy
COPY * /srv/app
Such recursive copy instruction can cause a lot of problems, starting from performance and build instabilities, through internal information leak, ending with complete application and infrastructure takeover. We highly recommend properly using .dockerignore.
.dockerignore is very similar to the .gitignore , but made for Docker. The operator or developer instructs Docker not to copy the specific paths accordingly to the context of runtime. The most common list of .dockerignore content is the following.A minimum list for `.dockerignore`
Dockerfile
docker-compose.yml
docker-compose.yaml
LINCENSE.txt
Makefile
.git
Of course, this is just a basic example, not very comprehensive. Each project might has its own entries, like the following example with Python.
A minimum list for `.dockerignore`
__pycache__
*.pyc
*.pyo
*.pyd
.Python
env
Another approach is to make a policy and prevent recursive-copies in your applications while auditing, and to always require a direct, absolute path. 2
Squash images after build-time
Image layers are one of the biggest threats. Very often developers and operators leave very sensitive information there. Let's modify our nginx Dockerfile.
An example of multi layered image
FROM alpine:3.9
LABEL maintainer="Kamil Zabielski <kamil.zabi[email protected]>"
RUN apk add nginx
RUN echo "fizz"
RUN echo "buzz"
We've added two more steps, echoing fizz and buzz. If the attacker gains the access to the image, per se, he is able to reverse the whole build time with exactly one command.
A quick reverse engineering of image build commands
≫ docker history --format "{{.ID}}: {{.CreatedBy}}" limakzi/example:1.0.0 | tail -r
<missing>: /bin/sh -c #(nop) ADD file:f4f85ec73d7cc9496…
82f67be598eb: /bin/sh -c #(nop) CMD ["/bin/sh"]
71c106f92525: /bin/sh -c #(nop) LABEL maintainer=Kamil Za…
3f7d600acd18: /bin/sh -c apk add nginx
9486a1a6d8ed: /bin/sh -c echo "fizz"
edb670dd828e: /bin/sh -c echo "buzz"
≫
Let's use docker-squash
to mitigate this information leak.3Squashing a multi-layer image
≫ docker-squash limakzi/example:1.0.0
2020-03-26 02:40:27,850 root INFO docker-squash version 1.0.8, Docker afacb8b, API 1.40...
2020-03-26 02:40:27,850 root INFO Using v2 image format
2020-03-26 02:40:27,860 root INFO Old image has 6 layers
2020-03-26 02:40:27,860 root INFO Checking if squashing is necessary...
2020-03-26 02:40:27,860 root INFO Attempting to squash last 6 layers...
2020-03-26 02:40:27,860 root INFO Saving image sha256:edb670dd828e1ed5722182c56ec73069c25f72897781e2b75243e701da7ac9ef to /var/folders/qh/sylx84mx21z5q3trnt6hd9tw0000gp/T/docker-squash-7yzrcz54/old directory...
2020-03-26 02:40:28,042 root INFO Image saved!
2020-03-26 02:40:28,042 root INFO Squashing image 'limakzi/example:1.0.0'...
2020-03-26 02:40:28,043 root INFO Starting squashing...
2020-03-26 02:40:28,043 root INFO Squashing file '/var/folders/qh/sylx84mx21z5q3trnt6hd9tw0000gp/T/docker-squash-7yzrcz54/old/074c599ca3ae41207a2376e1af24f9b96ec58528aff9da690b1726f6df564cef/layer.tar'...
2020-03-26 02:40:28,060 root INFO Squashing file '/var/folders/qh/sylx84mx21z5q3trnt6hd9tw0000gp/T/docker-squash-7yzrcz54/old/76fd8bf97c43091059e090842f65663e6ab8b6008d2ba2b66aab87539c6e9248/layer.tar'...
2020-03-26 02:40:28,141 root INFO Squashing finished!
2020-03-26 02:40:28,171 root INFO Original image size: 8.20 MB
2020-03-26 02:40:28,171 root INFO Squashed image size: 8.17 MB
2020-03-26 02:40:28,171 root INFO Image size decreased by 0.39 %
2020-03-26 02:40:28,171 root INFO New squashed image ID is 1fb82410d6da35a23981f6d813ef2ff5b80804ca8d928f1c2e2c63451559afe0
2020-03-26 02:40:28,623 root INFO Done
≫
≫ docker history 1fb82410d6da35a23981f6d813ef2ff5b80804ca8d928f1c2e2c63451559afe0
IMAGE CREATED CREATED BY SIZE COMMENT
1fb82410d6da 17 seconds ago 8.26MB
≫
Non-public images
It is very tempting to use public image, like php-fpm:7.3
php-fpm, maintainers will update it as soon as the vulnerability appears - and they would be right… in a perfect utopian world.
In reality, they are terribly wrong. Even at the first glance, the contents of some of the official images immediately expose a set of mistakes and inaccuracies. More over, using the public image renders proper compliance and implementation of the appropriate standards impossible. Public images and repositories tend to be a perfect target and tend to have outages .
Our recommendation may sound cumbersome, but it really neither hard nor expensive to implement. Build internal images and keep them up to date.No latest tag
Using latest tag is a very common problem, linked to outages, downtimes and problems with operationability of the infrastructure. It is a direct cause of many issues related to the unintended upgrades. Having that said, it can be linked directly to instability of the processes. Using lastest" tag at production is an immediate, clear way for the attacker to poison the image. We highly recommend not to use latest tag at any point of the build, even and especially for internal and compliant images. Proper Continuous Integration / Continuous Deployment processes provide most of the benefits coming from using the latest tag.
Static code analysis - Dockerfile
One of of the most common tools to statically analyze Dockerfile is hadolint
hadolint on the host machine. You can just grab the hadolint image, run the container, and inject your Dockerfile.An example of hadolint scan
≫ docker run --rm -i hadolint/hadolint < Dockerfile
Unable to find image 'hadolint/hadolint:latest' locally
latest: Pulling from hadolint/hadolint
c9b1b535fdd9: Pull complete
e7349a44e6b8: Pull complete
Digest: sha256:b309229a6e14f9a92d407c11718d0bb082dc642b9b1fe59937ec23626d48feb4
Status: Downloaded newer image for hadolint/hadolint:latest
/dev/stdin:5 DL3018 Pin versions in apk add. Instead of `apk add <package>` use
`apk add <package>=<version>`
/dev/stdin:5 DL3019 Use the `--no-cache` switch to avoid the need to use
`--update` and remove `/var/cache/apk/*` when done installing packages
≫
Vulnerabilities analysis
To show you, why all of this is so important, we will try to prove that the newest, fresh and official nginx image taken directly from Dockerhub is very vulnerable. Let's download nginx:latest.
Pulling new, fresh nginx image
≫ export DOCKER_CONTENT_TRUST=1
≫
≫ docker pull nginx
Using default tag: latest
Pull (1 of 1): nginx:[email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b: Pulling from library/nginx
Digest: sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Status: Image is up to date for [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Tagging [email protected]:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b as nginx:latest
docker.io/library/nginx:latest
≫
Now, let's scan the same, may I remind you, "newest and fresh" image, with aquasec/trivy
A security test of nginx image
≫ trivy --light --severity CRITICAL,HIGH nginx:latest
2020-03-26T01:25:44.415+0100 WARN You should avoid using the :latest tag as it is cached. You need to specify '--clear-cache' option when :latest image is changed
2020-03-26T01:25:44.417+0100 INFO Need to update DB
2020-03-26T01:25:44.417+0100 INFO Downloading DB...
3.96 MiB / 3.96 MiB [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2.95 MiB p/s 2s
2020-03-26T01:25:47.463+0100 INFO Reopening DB...
2020-03-26T01:25:47.560+0100 INFO Detecting Debian vulnerabilities...
nginx:latest (debian 10.3)
==========================
Total: 15 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 13, CRITICAL: 2)
+-----------------+------------------+----------+---------------------------+---------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |
+-----------------+------------------+----------+---------------------------+---------------+
| bash | CVE-2019-18276 | HIGH | 5.0-4 | |
+-----------------+------------------+ +---------------------------+---------------+
| libc-bin | CVE-2019-1010022 | | 2.28-10 | |
+ +------------------+ + +---------------+
| | CVE-2020-1752 | | | |
+-----------------+------------------+ + +---------------+
| libc6 | CVE-2019-1010022 | | | |
+ +------------------+ + +---------------+
| | CVE-2020-1752 | | | |
+-----------------+------------------+----------+---------------------------+---------------+
| libjpeg62-turbo | CVE-2019-2201 | CRITICAL | 1:1.5.2-2 | |
+-----------------+------------------+----------+---------------------------+---------------+
| libpcre3 | CVE-2017-11164 | HIGH | 2:8.39-12 | |
+-----------------+------------------+ +---------------------------+---------------+
| libseccomp2 | CVE-2019-9893 | | 2.3.3-4 | |
+-----------------+------------------+ +---------------------------+---------------+
| libsystemd0 | CVE-2020-1712 | | 241-7~deb10u3 | |
+-----------------+------------------+ +---------------------------+---------------+
| libtasn1-6 | CVE-2018-1000654 | | 4.13-3 | |
+-----------------+------------------+ +---------------------------+---------------+
| libtiff5 | CVE-2017-9117 | | 4.1.0+git191117-2~deb10u1 | |
+-----------------+------------------+ +---------------------------+---------------+
| libudev1 | CVE-2020-1712 | | 241-7~deb10u3 | |
+-----------------+------------------+ +---------------------------+---------------+
| libwebp6 | CVE-2016-9085 | | 0.6.1-2 | |
+-----------------+------------------+ +---------------------------+---------------+
| nginx | CVE-2013-0337 | | 1.17.8-1~buster | |
+-----------------+------------------+----------+---------------------------+---------------+
| tar | CVE-2005-2541 | CRITICAL | 1.30+dfsg-6 | |
+-----------------+------------------+----------+---------------------------+---------------+
≫
A security test of nginx image
≫ trivy --light --severity CRITICAL,HIGH php:7.3-fpm
2020-03-26T10:51:43.787+0100 INFO Detecting Debian vulnerabilities...
php:7.3-fpm (debian 10.3)
=========================
Total: 52 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 48, CRITICAL: 4)
+---------------------------+------------------+----------+-------------------+---------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |
+---------------------------+------------------+----------+-------------------+---------------+
| bash | CVE-2019-18276 | HIGH | 5.0-4 | |
+---------------------------+------------------+ +-------------------+---------------+
| binutils | CVE-2017-13716 | | 2.31.1-16 | |
+ +------------------+ + +---------------+
| | CVE-2018-12699 | | | |
+---------------------------+------------------+ + +---------------+
| binutils-common | CVE-2017-13716 | | | |
+ +------------------+ + +---------------+
| | CVE-2018-12699 | | | |
+---------------------------+------------------+ + +---------------+
| binutils-x86-64-linux-gnu | CVE-2017-13716 | | | |
+ +------------------+ + +---------------+
| | CVE-2018-12699 | | | |
+---------------------------+------------------+ + +---------------+
| libbinutils | CVE-2017-13716 | | | |
+ +------------------+ + +---------------+
| | CVE-2018-12699 | | | |
+---------------------------+------------------+ +-------------------+---------------+
| libc-bin | CVE-2019-1010022 | | 2.28-10 | |
+ +------------------+ + +---------------+
| | CVE-2020-1752 | | | |
+---------------------------+------------------+ + +---------------+
| libc-dev-bin | CVE-2019-1010022 | | | |
+ +------------------+ + +---------------+
| | CVE-2020-1752 | | | |
+---------------------------+------------------+ + +---------------+
| libc6 | CVE-2019-1010022 | | | |
+ +------------------+ + +---------------+
| | CVE-2020-1752 | | | |
+---------------------------+------------------+ + +---------------+
| libc6-dev | CVE-2019-1010022 | | | |
+ +------------------+ + +---------------+
| | CVE-2020-1752 | | | |
+---------------------------+------------------+ +-------------------+---------------+
| libpcre3 | CVE-2017-11164 | | 2:8.39-12 | |
+---------------------------+------------------+ +-------------------+---------------+
| libseccomp2 | CVE-2019-9893 | | 2.3.3-4 | |
+---------------------------+------------------+ +-------------------+---------------+
| libsystemd0 | CVE-2020-1712 | | 241-7~deb10u3 | |
+---------------------------+------------------+ +-------------------+---------------+
| libtasn1-6 | CVE-2018-1000654 | | 4.13-3 | |
+---------------------------+------------------+ +-------------------+---------------+
| libudev1 | CVE-2020-1712 | | 241-7~deb10u3 | |
+---------------------------+------------------+----------+-------------------+---------------+
| linux-libc-dev | CVE-2019-19813 | CRITICAL | 4.19.98-1 | |
+ +------------------+ + +---------------+
| | CVE-2019-19814 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19816 | | | |
+ +------------------+----------+ +---------------+
| | CVE-2008-4609 | HIGH | | |
+ +------------------+ + +---------------+
| | CVE-2013-7445 | | | |
+ +------------------+ + +---------------+
| | CVE-2018-20669 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-12456 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-12615 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-16229 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-16230 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-16231 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-16232 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-16233 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-16234 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-18814 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19054 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19061 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19064 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19067 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19070 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19072 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19074 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19082 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19083 | | | |
+ +------------------+ + +---------------+
| | CVE-2019-19815 | | | |
+ +------------------+ + +---------------+
| | CVE-2020-0041 | | | |
+ +------------------+ + +---------------+
| | CVE-2020-1749 | | | |
+---------------------------+------------------+ +-------------------+---------------+
| m4 | CVE-2008-1687 | | 1.4.18-2 | |
+ +------------------+ + +---------------+
| | CVE-2008-1688 | | | |
+---------------------------+------------------+----------+-------------------+---------------+
| tar | CVE-2005-2541 | CRITICAL | 1.30+dfsg-6 | |
+---------------------------+------------------+----------+-------------------+---------------+
≫
As any attentive Reader will notice, we were looking for CRITICAL and MEDIUM severities only. Of course, CVEs might have a fixed version field empty. It may be unfilled for various reasons; some CVEs are an expected behaviour of the software, like CVE-2005-2541; some CVEs were not fixed yet, like CVE-2019-18276; others are just candidates, like CVE-2020-1752.
1 Particularly, we think about policies and admission controllers in Kubernetes platform.
2 In later parts of the guide, we're presenting automation tools to achieve it.
3 Unfortunately, squashing as an internal process by docker without external tools, at the moment of writing this article, seems to be still experimental.
References
- NGINX - Alpine Wiki. (n. d.). https://wiki.alpinelinux.org/wiki/Nginx (accessed July 16, 2020).
- User Namespace Remapping. (n. d.). https://docs.docker.com/engine/security/userns-remap/ (accessed July 16, 2020).
- Docker/runc container vulnerability. (n. d.). https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5736 (accessed July 16, 2020).
- Escape from Docker - dragonsector.pl. (n. d.). https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-and.html (accessed July 16, 2020).
- Notary - Getting Started. (n. d.). https://docs.docker.com/notary/getting_started/ (accessed July 16, 2020).
- Official docs on .dockerignore. (n. d.). https://docs.docker.com/engine/reference/builder/#dockerignore-file (accessed July 16, 2020).
- Official docs on .gitignore. (n. d.). https://git-scm.com/docs/gitignore (accessed July 16, 2020).
- Docker-squash source page. (n. d.). https://github.com/jwilder/docker-squash (accessed July 16, 2020).
- Official PHP-FPM 7.3 image on DockerHub. (n. d.). https://hub.docker.com/_/php (accessed July 16, 2020).
- Trend Micro on Dockerhub incident. (n. d.). https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/docker-hub-repository-suffers-data-breach-190-000-users-potentially-affected (accessed July 16, 2020).
- Quay.io outage report. (n. d.). https://status.quay.io/history?page=3 (accessed July 16, 2020).
- GitHub - hadolint/hadolint. (n. d.). https://github.com/hadolint/hadolint (accessed July 16, 2020).
- GitHub - aquasecurity/trivy. (n. d.). https://github.com/aquasecurity/trivy (accessed July 16, 2020).