diffoci: diffing container images to check they are reproducible

A comparison tool for images, not a builder

diffoci is described as diff for Docker and OCI container images, and it compares two of them to help with reproducible builds. The README is careful about one abbreviation, noting that OCI here refers to the Open Container Initiative and not to Oracle Cloud Infrastructure, which is a sensible warning given how often the name is assumed otherwise.

What the tool does is narrow and well defined. It takes two images and reports, for each path inside them, whether the content differs, and it prints the digests of both sides. The output in the README is a table with a type, a name and the two input columns.

$ diffoci diff --semantic alpine:3.18.2 alpine:3.18.3
TYPE    NAME                                 INPUT-0                                                             INPUT-1

The rows beneath it are files, and the two long hex strings are the digests of the same path in each image. In the example, lib/libcrypto.so.3, usr/bin/iconv, usr/lib/engines-3/capi.so and bin/busybox all differ, which is what you would expect between two Alpine point releases and is exactly the kind of change a security team wants to see spelled out.

The project sits in the reproducible builds community, and its existence is a comment on that field. Making a build reproducible requires pinning timestamps, sorting file order, fixing the build history and controlling the environment. diffoci does none of that. It is the instrument you use to check whether somebody else has.

Strict mode catches everything, semantic mode ignores the noise

There are two modes, and the README's comment on the strict one is blunt: the strict mode is often too strict, and you should consider the non-strict mode.

Strict mode is the plain invocation.

diffoci diff IMAGE0 IMAGE1

It reports every difference it can find, which for two independently built images includes differences that have nothing to do with your source code: the time the layer was created, the order of files inside a tar layer, the history metadata attached to the image, and the name the image was tagged with.

Semantic mode, also called non-strict, ignores exactly those four things: timestamps, build histories, file ordering in tar layers, and image name annotations.

diffoci diff --semantic IMAGE0 IMAGE1

What is left is the content of the files, which is the question reproducible builds actually cares about. If the same source produces two images whose file contents differ, something in the build is not under control. If the only differences are metadata, the build is reproducible in the sense that matters and the noise can be ignored.

The choice between the modes is the real judgement call the tool asks of you. Strict mode is the audit, semantic mode is the regression check, and the README's own advice is that strict is the one you will often want to turn off.

Dumping the conflicting files for a real diff

The digest table tells you which paths differ, and the report directory option tells you what the differences are.

Setting --report-dir=DIR dumps the conflicting files into that directory.

$ diffoci diff --semantic --report-dir=/tmp/diff alpine:3.18.2 alpine:3.18.3

The dumped layout mirrors the two inputs, with a manifests directory and layers directory per side, and the README then shows the result of running an ordinary diff over them.

$ diff -ur /tmp/diff/input-0/ /tmp/diff/input-1/
Binary files /tmp/diff/input-0/manifests-0/layers-0/bin/busybox and /tmp/diff/input-1/manifests-0/layers-0/bin/busybox differ
diff -ur /tmp/diff/input-0/manifests-0/layers-0/etc/alpine-release /tmp/diff/input-1/manifests-0/layers-0/etc/alpine-release
--- /tmp/diff/input-0/manifests-0/layers-0/etc/alpine-release   2023-06-15 00:03:14.000000000 +0900
+++ /tmp/diff/input-1/manifests-0/layers-0/etc/alpine-release   2023-08-07 22:09:12.000000000 +0900
@@ -1 +1 @@
-3.18.2
+3.18.3

This is the part of the tool that makes it more than a yes or no check. A digest list tells you that a binary changed; the dump tells you that the version file says 3.18.2 against 3.18.3, which is the difference between a deliberate base image bump and something stranger. Because the output is a directory of files, you can hand it to any diff tool you like, including one that renders binary differences.

Installing from a release, from mise or from source

There are three routes, and the simplest is a binary.

Binaries are available for Linux and macOS from the project's releases page. If you use mise, the polyglot tool versions manager, the README says you can install it with a single command.

mise use -g diffoci@latest

Building from source needs Go, and the README states Go 1.21 or later.

go install github.com/reproducible-containers/diffoci/cmd/diffoci@latest

There is a detail worth checking if you plan to build from source. The README says Go 1.21 or later, while the module file in the repository declares a much newer language version, go 1.26.6, and the Dockerfile builds on golang:1.26.6-bookworm. The toolchain line in go.mod is what the compiler enforces, so a Go 1.21 install will fail on the module requirement even though the README's floor suggests it should work. The README text and the module file disagree, and the module file wins.

The rest of the module dependencies are the container stack. containerd with its continuity, errdefs, log and platforms modules, the distribution reference parser, the Docker CLI, the OCI image spec and go-digest, cobra and pflag for the command line, go-cmp for comparisons, bbolt for the local store, and the golang.org/x/sys module.

Backends: containerd by default, docker:// and podman:// on demand

diffoci has to know where the images live, and the README covers three sources.

Containerd is the default when containerd v1.7 or later is running, and the default namespace is default. You can force it either way.

diffoci --backend=containerd --containerd-address=ADDRESS --containerd-namespace=NAMESPACE

diffoci --backend=local

For images that exist only in a local Docker or Podman daemon and were never pushed to a registry, you prefix the image name with the daemon's scheme.

docker build -t foo ~/foo
docker build -t bar ~/bar
diffoci diff docker://foo docker://bar

The README states clearly that you do NOT need to specify a custom --backend to access Docker images, and that the same applies to Podman: prepend podman:// and read docker as podman everywhere in the example.

Private images work the way you would expect from the Docker world. To access them, create a credential file as ~/.docker/config.json using docker login, and diffoci picks it up.

The dependency on containerd explains a lot about the shape of the tool. Rather than shelling out to docker save and untarring the result, diffoci reads the image store directly, which is why it can produce a file level report of layer contents rather than a manifest comparison.

Platform selection is where most people get stuck

The README carries a tip that addresses the most common failure. Non-Linux users, and those comparing multi-platform images from Docker, typically have to specify --platform explicitly.

The example given is diffoci diff with --platform=linux/amd64 followed by the two images.

The reason is in the reference format. A modern image reference can point at a manifest list that covers several architectures, so a plain image name is ambiguous. Without an explicit platform, the tool has to pick one, and on a machine that is not Linux the natural choice may not be the one you meant. Comparing a Linux image on a Mac without --platform is a reliable way to get a confusing result.

There is a second constraint attached to the docker:// scheme, and it is a version floor. Platform filtering for docker:// images requires Docker v28 or later, so on an older Docker you are limited to images whose platform is unambiguous.

The practical sequence is therefore: pin the platform when the image is multi-platform or you are not on Linux, and check your Docker version before relying on the docker:// form with a specific platform.

Where diffoci is the wrong tool

The first case is a team that wants reproducible images rather than verified ones. diffoci reports differences; it does not remove them. If your two builds differ because your Dockerfile does not pin a timestamp, diffoci will show you that difference and nothing more. The fix belongs in the build.

The repository's own Makefile is a good illustration of that side of the problem. It exports CGO_ENABLED as 0, DOCKER_BUILDKIT as 1, and SOURCE_DATE_EPOCH from the last commit timestamp with git log -1 --pretty=%ct, which is the standard way to make a build date deterministic. The artifacts target cross compiles the binary for linux on amd64, arm with GOARM 7, arm64, ppc64le, riscv64, s390x and darwin on amd64, and the version string is stamped into the binary with -X so that a built binary records the git describe output.

The second case is a comparison where the metadata differences are the point. If you are investigating why a base image bump changed your image, the strict mode is the right setting, and the semantic mode will hide the very differences you are chasing.

The third case is a large scale image audit. The tool is a diff for two images, and the report option writes the conflicting files to disk. Pointing that at a catalogue of hundreds of images means pulling layers out of the store and writing them to a filesystem, and the README does not describe a batch mode.

The alternative approach to the same problem is to compare the image digests, which is what most pipelines do and which costs nothing. The difference in approach is that a digest comparison tells you that something changed and nothing about what, while diffoci gives you the file level answer. If you only need the signal, digests are enough. If you need to explain the signal to somebody, diffoci is the tool that produces the explanation.

Editorial conclusion

diffoci suits a team that publishes container images and wants to know whether two builds of the same source really are identical, or wants to see exactly which files a base image bump changed. It does not suit a team that wants its build to be reproducible in the first place, since it reports differences rather than removing them, and it does not replace pinning a build timestamp. Verify first whether the images you are comparing are multi-platform, because the README notes that non-Linux users and Docker users comparing multi-platform images usually have to pass --platform explicitly, and that platform filtering for docker:// images needs Docker v28 or later.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. reproducible-containers/diffoci on GitHub