A 3 GB image with 2.8 GB in one layer: what docker history says and how to fix it

Show & TellJimmy5 replies

JimmyMember#1

A colleague sent me a Dockerfile for a service. The image was 3.1 GB. The application in it was about 40 MB of JavaScript.

This is the most common packaging mistake there is, and it is invisible until you look, because the build succeeds, the container runs, and nobody checks the size.

Reading the image

Two commands, and the second one is the one that helps:

docker images myapp
docker history --no-trunc myapp:latest

docker history prints one row per layer, oldest first, with the command that created it. The size column is the interesting one. A row that is 2.8 GB with a command like RUN apt-get install -y ... is your answer: a package manager wrote a database, a cache and its own logs into a layer that will now be in the image forever, because a new layer cannot remove files from an old one.

The other way to see it is a COPY . /app step that is far too big, which is usually a missing .dockerignore.

The fix is a multi-stage build

The rule is simple: the stage that needs the toolchain is not the stage that ships. Build with everything, copy the result into a clean runtime stage, and let the toolchain stay behind.

# build
FROM node:22-bookworm AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

# runtime
FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
COPY package.json ./
USER node
CMD ["node", "dist/main.js"]

The runtime stage has no compiler, no build cache and no sources. The npm ci step stays in the build stage, which is also where it belongs: it is the slow step, and this way its layer cache survives a change to a source file.

Do not forget the ignore file

A missing .dockerignore is the other half of the problem, and it is the half that copies your secrets:

node_modules
dist
.git
.env
.env.*
*.log

COPY . . copies everything that is not ignored, including your .git directory, your local .env and your node_modules from a different platform. On a developer laptop that is a 400 MB layer and a leaked key; in CI it is usually a different story but the same waste.

How much is left

The size you get is the size of the runtime stage plus the runtime's own base image, and the base image is worth choosing deliberately. node:22-bookworm-slim is a smaller Debian than node:22-bookworm, and node:22-alpine is smaller again, at the cost of a different libc and the occasional surprise with native modules.

If you want to see where the bytes really went, dive walks the layers and tells you which files are in which one, which is the same information as docker history but per file. We use it on images we did not build.

5 replies

EdwardMember#2

Two things I would add.

docker history only tells you the size of the layer, not what is in it, so for a 2.8 GB apt-get layer the next step is to build the image and look inside that layer's directory.

And docker system df shows what the whole machine is holding, including the build cache, which on a CI box is often several times the size of every image you actually run.

DaisyRegular#3

Does the multi-stage approach work if the build step needs something from the runtime stage, like a system library?

I inherited a Go service that links against a version of a library that is only in the runtime image, and I have not worked out how to split that.

JimmyAuthorMember#4

In reply to Edward (#2)

It works if the thing you need is in the base image of the build stage, and it does not if the dependency only exists in the runtime stage — that is the case where the stages have to be arranged the other way round, with the runtime as the builder's base.

For Go specifically the usual pattern is different anyway: compile to a static binary in the build stage and copy that one file into a scratch or distroless image, which ends up in the tens of megabytes.

HermanMember#5

One more thing to watch on a small self-hosted box: the layers are not the only thing that eats the disk. The build cache and the old images do.

docker image prune -a and docker builder prune on a schedule is the routine answer, and docker system df is how you find out which of the two is the problem. The multi-stage build helps the image you ship and the image you pull, but it does nothing about the intermediate layers sitting in the builder cache.

IvanMember#6

Counterpoint from experience: the multi-stage build that saves 2 GB is also the one that saves you the ability to debug inside the container.

If your runbook says "docker exec into the box and look around", a slim runtime with no shell and no compiler is a different job. We keep the full image for staging and the slim one for production, and the difference is written in the runbook so nobody is surprised at 2 a.m.