izumi: a bag of independent Scala libraries for everyday productivity

Nine components, one name

The project describes itself as an ecosystem of independent libraries and frameworks whose purpose is to significantly increase the productivity of your Scala development. The name is a Japanese word for spring water, which is a quieter claim than it sounds.

The list of components is the substance of the project, and it is worth reading as a stack rather than as a catalogue. There are nine entries, and they cover dependency injection, testing, logging, effects, reflection, API definition, build tooling, and a domain modelling pattern.

The first is distage, a compile-time safe, transparent and debuggable multi-modal dependency injection framework for pure functional Scala. The phrase multi-modal matters: the framework can produce a wired object graph at compile time, at run time, or both, and the transparency and debuggability claims are about being able to see what it did.

The second is distage-testkit, described as a hyper-pragmatic pure functional test framework. The three properties it claims are specific: it shares heavy resources globally across all test suites, so that a database container or a cache is not started per test class; it lets you easily swap implementations of a component; and it uses your effect type for parallelism, so the concurrency model of the tests is the same one as the application.

The third is a distage extension for running Docker containers in tests or for local application runs, and it ships with example containers for Postgres, Cassandra, Kafka, and DynamoDB. So the testing story is not just a framework, it is a way to have real dependencies in tests without hand-written fixture code.

The fourth is LogStage, which produces structural logs automatically from Scala string interpolations. The fifth is BIO, a typeclass hierarchy for tagless final style with bifunctor effect types, focused on ergonomics and ease of use with zero boilerplate. Tagless final is the Scala way of writing an effect type once and having it work for any monad, and bifunctor means the effect can carry two values, which is what you need for an error and a result.

Three of the last four are the ones that have moved. The reflection library, which is a portable, lightweight and kind-polymorphic alternative to the standard reflection type tag, has moved to the ZIO organisation, which is the clearest sign in the list that a component outgrew its origin. An API definition, data modelling and RPC language, with generated servers and clients for Go, TypeScript, C#, and Scala, has moved to its own repository. A set of opinionated build plugins that reduce the verbosity of build definitions has moved to a repository of its own. And the last is a pattern rather than a library, a way of modelling very complex domains and orchestrating complicated processes.

The pattern is named with an acronym that expands to percept, plan, execute, repeat, and it is described as enabling the modelling of very complex domains and making the orchestration of deadly complex processes easier than you are used to. The project says the future plans are more tools based on that approach, including a build system, a cluster orchestration tool, and a load testing and macro benchmarking tool.

Four goals, and one of them is a joke

The key goals section is four items long, and the fourth is the one people remember.

The first is to boost productivity and reduce code bloat. The second is to be as non-invasive as possible. The third is to be introspectable. And the fourth is to be better than anything else out there, with a emoticon that makes clear the author does not entirely expect to be taken at his word.

Each of the first three maps onto a specific design decision, and it is worth taking them one at a time because they are the actual specification of the project.

Non-invasive is the constraint that shapes everything else. It means the libraries should not require you to restructure your application around them, and for a dependency injection framework in particular that is a strong promise. The usual cost of typed dependency injection is that your components become classes with annotations and you inherit from a base class or implement a trait that the framework owns. A framework that is non-invasive in the strong sense has to find another way to get the types, and the answer in a pure functional setting is usually to make the wiring a value rather than a structure.

Introspectable is the second constraint, and it is a response to a real problem. A dependency injection framework that hides what it did is a framework you cannot debug, and the traditional answer to a wiring problem is to read a stack trace and guess. A framework that can print its own plan is one you can read instead. The same argument explains why the project is built around a design-time, staged wiring approach rather than a reflective one: a plan that exists before the program runs can be printed, and a plan that is discovered at run time cannot.

Reducing code bloat is the third, and it is where the typeclass hierarchy and the logging library earn their place. A tagless final hierarchy means you write your effectful code against a typeclass and get it to work with any monad, so the same business logic runs in the test, in the application, and in a different effect type, without duplication. Automatic structural logging from string interpolations means a log line is one interpolation rather than a builder call with a key and a value.

And the fourth goal, with the emoticon, is a statement of intent rather than a promise. The libraries are opinionated, they overlap with existing projects, and the author is aware of that. The projects that are listed on the awesome Scala list, of which the badge in the header is a mark, are the alternatives his libraries are measured against.

The parts that left, and what it means

The list of components has three entries that have been moved to other repositories, and that is the most interesting structural fact about the project.

The reflection library now lives under the ZIO organisation, and the readme says so explicitly, with a link to the new location. The API definition and RPC language moved to a repository named for its first version. The build plugins moved to a separate repository of their own. And the readme marks each of these with the word moved, in bold, so a reader is not left with the impression that these are still developed here.

The reason is not hard to guess, and it is a sign of health rather than of failure. A component that becomes useful outside the collection has to be usable on its own, and a component that lives in a monorepo with a build and a publishing pipeline is awkward to depend on for somebody who wants one library. Moving the reflection library to the ZIO organisation is the strongest version of that, because it places the library next to a well known effect library rather than next to a dependency injection framework, which is where it will be used.

The build plugins are the clearest case. A build plugin has to be available to a project that does not otherwise use anything from the collection, and the description of them as reducing the verbosity of build definitions, and introducing shared test scopes between projects and bill of materials plugins, is a description of a tool for anyone. Keeping that inside a monorepo whose build is itself one of the plugins would be awkward.

So what is left in the repository is the part that has to be together. The distage framework, its test framework, the Docker extension, LogStage, BIO, and the pattern, plus the shared build infrastructure. And the layout matches: a directory for distage, one for logstage, one for the fundamentals, one for the build generator, a documentation directory, and the build definition at the root.

For a reader deciding whether to adopt any of this, the practical consequence is that the collection is a compatibility set rather than a bundle. The components are published under one organisation on Maven Central, with the badge pointing at the core module for the Scala 2.13 line, and the Sonatype links for releases and snapshots. But you can take the logging library without the dependency injection framework, and the framework without the test kit, and the test kit is the one that most obviously assumes the others.

The default branch is develop, and the releases are on a milestone line rather than a stable one, which is the honest signal about the project's maturity.

Structured logging from an interpolation

LogStage is the component with the clearest single idea, and it is worth understanding on its own.

The claim is automatic structural logs from Scala string interpolations. The traditional way to log structured data in a typed language is to build a message with a key and a value, and the usual libraries give you a DSL for it. The cost is that the log call and the data it carries are two different things, and a reader of the code has to match them up.

If the log statement is an interpolation, the key and the value are in the same place, and the structure can be derived from the interpolation itself. That is the whole idea, and it explains the name of the component in the project's own list of slides, where it is called zero-cost structured logging. Zero cost because there is no builder object to allocate and no key to look up at run time, and the structure comes from a macro that reads the interpolation at compile time.

The library is listed among the projects that use the standard logging facade interface, along with a logger implementation library, and the topics the repository carries name structured logging explicitly. That combination is the conventional way to publish a logging library in the JVM ecosystem: implement the facade so that any of the existing backends can be used underneath, and the structured data is the part that is new.

The compile-time reading is also why the project is functional in approach rather than incidental. A macro that inspects an interpolation at compile time is a natural fit in a language with macros and a type system, and the alternative, parsing the formatted string at run time, would give you a string with the structure lost.

For a reader who has never used it, the interesting question is what it costs to adopt. The answer is in the second key goal. If logging is a call with an interpolation rather than a builder, then adopting it means changing how you write log statements, and changing a convention is the least invasive thing a library can ask for. It does not mean changing the structure of your application, it does not require a base class, and it works with the logging facade your project already uses.

There is a dedicated talk on the zero-cost structured logging in the project's video list, and a set of slides with the same name, which is a reasonable way to judge the idea before reading the source.

Testing with real dependencies and a shared effect type

The test framework is the component that says the most about what the author thinks about software development, and it is worth pulling apart because the three claims in the readme are unusual.

The first is that the framework shares heavy resources globally across all test suites. The heavy resources are the things that are expensive to start: a database, a message broker, a cache. In most test setups each suite starts its own, which means a test run spends most of its time waiting for containers to become healthy. Sharing them globally means one set is started for the whole run and reused, and the cost is paid once.

The second is that it lets you easily swap implementations of a component. In a functional setting, where components are values rather than objects with injectable state, swapping an implementation is a matter of providing a different value, and the framework makes that easy. The motivation is testing a real application rather than a set of mocks: you can run the actual component graph with one part replaced by a fake, and the point of a test framework is to make that cheap.

The third is that it uses your effect type for parallelism. That is the one that will decide whether you can use it. If your application is written against the tagless final hierarchy, then the test framework's parallelism is expressed in the same abstraction, and a test can run effects concurrently without the framework knowing what your effect is. If your application is written against a different effect library, the test framework cannot be used as is.

The Docker extension is what makes the first claim practical. Containers for Postgres, Cassandra, Kafka, and DynamoDB come as examples, and the extension is for using Docker containers in tests or for local application runs. The local application part is worth noting: the same container that a test needs can be started for a local run of the application, so a developer and the test suite see the same environment.

All of this hangs together into a single philosophy, which the project states in the title of one of its talks: hyper-pragmatic pure functional testing. Pragmatic means the framework is designed around the problems people actually have, and in this case the problem is that functional Scala codebases have a hard time testing, because everything is a value and there are no objects to mock. The framework's answer is not to make mocking easier but to make it unnecessary, by making the real components replaceable and the expensive ones shareable.

The stack of the three of them is the point. A dependency injection framework that can wire at compile time, a test framework that shares resources and uses your effect type, and a container extension for the dependencies, are three parts of one answer to the same question.

The build, the tooling, and how a contribution works

The repository layout and the tooling configuration describe how this is developed, and a few of them are worth noting.

The build is a single project definition at the root, and the source directories are the components: the dependency injection framework, the logging library, the fundamentals, the build generator, the documentation directory, and a project directory holding the build's own metadata. The version lives in its own small file, and the documentation site is built from a microsite source inside the documentation directory, with the logo asset in the same place.

The formatting configuration and a script for formatting everything in one go are both present, along with an editor configuration, a markdown lint configuration, a dependency update configuration, and a JVM options file. The dependency update configuration is the tool that opens pull requests for outdated libraries, which is the standard way to keep a large multi-project build current without a human doing it.

Two entries in the listing are less usual and more interesting. There is an environment file for a directory environment tool, and there is a Nix flake with a lock file. So the project has a reproducible shell environment, defined declaratively, which is what lets a contributor and the continuous integration run with the same tool versions.

And there is a file with an encrypted extension, containing a tar archive. That is a sealed secrets file: the credentials needed to publish to Maven Central and to the Sonatype repositories are encrypted in the repository and decrypted at release time. It is the standard approach for a multi-module project that publishes many artefacts from one build, and it is worth noting because it means the publishing credentials are never in the repository in plain form.

The contribution side is described in a separate document, and the readme points at it. There is a code of conduct, an agent instruction file, and the usual badges: a build status, a coverage report, a code factor rating, a chat room, a sponsorship link, and a listing in the awesome Scala collection. The coverage badge is the one to look at, because a project with this many components and a coverage report is one that takes its own tests seriously.

The last two things in the readme are the state of the project and the call for help. It is looking for early adopters, contributors, and sponsors, and it says plainly that it is a work in progress. Then it lists the future plans: a build system, a cluster orchestration tool, and a load testing and macro benchmarking tool, all of them to be built on the same modelling approach.

The scale of that list is worth registering. The author is proposing to build, in a functional style, the three tools that the current ecosystem mostly does not have in that style. Whether they arrive is another matter, but the fact that the plan is stated that concretely is a better signal than a general ambition.

Editorial conclusion

izumi fits a Scala team that has decided to go functional and wants its dependency injection, logging, and test framework to come from the same author and the same ideas, rather than assembled from four unrelated projects. It does not fit a team that wants one small library, since the dependency list here is a stack rather than a utility, and it does not fit a team that needs a stable long term commitment from an individual maintainer. Verify first which components still live in this repository and which have moved, because several are now maintained elsewhere, that the effect type you want to use is supported by the tagless final hierarchy you plan to build on, and that the version you take matches the Scala version your project is on, since the release line is on a milestone and the build spans several cross targets.

Frequently asked questions

Official sources

1. 7mind/izumi on GitHub
2. License: BSD-2-Clause
3. Project website
4. README
5. Releases