datawave: an ingest and query framework on top of Accumulo

A query layer over Accumulo

DataWave is a Java based ingest and query framework that uses Apache Accumulo to provide fast, secure access to your data, and the project site is code.nsa.gov/datawave.

The relationship to Accumulo is the first thing to understand. Accumulo is a key value store built on HDFS, and it is good at storing a great deal of key value data and doing range scans over it. What it does not do is understand documents, and a document that is spread across several cells in a row is awkward to reason about.

DataWave is the layer that turns that into something you can query like data, and it is where the security model lives, because the authorisation story is the part that a general purpose key value store does not have.

The use cases are listed in the README, and they are the reason to look at this project.

Data fusion across structured and unstructured datasets. Construction and analysis of distributed graphs. Multi-tenant data architectures, where the tenants have distinct security requirements and data access patterns. And fine-grained control over data access, integrated easily with existing user-authorization services and PKI.

That last one is the distinctive feature. A multi-tenant system where each tenant has its own access rules needs the authorisation decision to happen at query time, on the rows, and it needs to come from a system the organisation already runs rather than from a second identity store inside the data platform. The mention of PKI in the same sentence says the same thing: the identity is external, and the framework is built to consume it.

The project is Apache 2.0 licensed, written in Java, and not archived, with the last push on 2026-10-20. The default branch is integration rather than master, which is a hint that the integration branch is where the pieces are assembled.

A repository that is really a set of repositories

The section on how to use the repository is the most unusual thing about the project, and it is worth reading before you clone anything.

The microservices and the associated utility projects are intended to be developed, versioned, and released independently, and the repository lists the subdirectories that hold those independently versioned modules.

The list is long and it is a map of the system. Under contrib there is the datawave utilities. Under core there is an in-memory Accumulo implementation and a metrics reporter. Under microservices there is a parent pom and a service parent pom, a starters group with the audit, cache, cached results, datawave, metadata, query, and query metric starters, and a services group with the Accumulo, audit, authorisation, config, dictionary, file provider, Hazelcast, map, MapReduce query, modification, query, query executor, and query metric services.

Each of those subdirectories contains a file that keeps track of where the code came from, which is the mechanism, and the mechanism is the git subrepo tool. At one point the project used submodules to link in all of the sub repositories, and it has now switched to including the submodules' code directly into the main repository. The subrepo mechanism was used to facilitate that transition, and it can still be used to pull in changes from the other repositories as needed until they can be removed altogether.

The commands are the standard ones, with the repository and the directory as the arguments for the initial clone, and the directory alone for pulling in later changes.

git subrepo clone <repo> <dir>

git subrepo pull <dir>

This arrangement has a consequence that the README states explicitly and that will bite you if you miss it. The utility modules, the starters, and the services are actually tagged and deployed separately, and hence the snapshot versions within those sub repositories are not connected together.

So a checkout of the core repository gives you the source of everything, but it does not give you a coherent set of released service versions. If you are deploying the microservices, you are deploying a set of independently versioned artifacts, and the versions have to be chosen deliberately rather than assumed to line up.

Building on Java 11, with a Java 17 profile

The build section is detailed, and the detail is mostly about the fact that the root build still targets Java 11 while the toolchain moves on.

The root build remains Java 11 by default. Builds on Java 17 are supported through the java17 profile, and they remain backward compatible by compiling with the release flag set to 11, so the produced bytecode runs on an older runtime.

mvn -Ddocker-release -Ddist clean install -T 1C

mvn -Pjava17 -Ddocker-release -Ddist clean install -T 1C

The thread count flag is not optional in practice. It is recommended to build the project using multiple threads, and the one thread per core value is the setting the README uses. The default build does not build the starters, the utilities, and the services, and the additional flags add them.

mvn -Ddocker-release -Dmicroservice-docker -Ddist -Dutils -Dservices -Dstarters clean install -T 1C

mvn -Pjava17 -Ddocker-release -Dmicroservice-docker -Ddist -Dutils -Dservices -Dstarters clean install -T 1C

And there is a flag for building the service interfaces without the service implementations themselves, which is the onlyServiceApis option.

The Java 17 profile has one trap, and the README explains it carefully. When you use the profile, make sure that Maven itself is running on a full JDK 17 and not a JRE only runtime, and the version command should report a Java 17 runtime that includes the compiler. If Maven is launched with a trimmed runtime, compilation can fail with an error that the release version 11 is not supported, which is a confusing message that has nothing to do with the thing you actually changed.

There is one more detail for the Java 17 case. When building microservices on Java 17, invoke Maven from the microservices reactor, or from another parent that defines the java17 profile, and not from an individual leaf module, because the profile is defined at the parent level.

The key binaries, and the Accumulo version that is pinned

The README has a short section on the key binaries used throughout the repository, and it is candid about the fact that they do not all match.

Accumulo, Zookeeper, Hadoop, and Wildfly are the key binaries used throughout this repository. Ideally, the versions used should be the same, but many of the microservices use different versions.

Then comes the practical consequence. There are upgrade issues with the MiniAccumuloCluster that is used in several tests, so the Accumulo version is set to 2.1.2 in the root pom.xml, and because of this, projects such as the Quickstart must override the version.accumulo property to use a more current version.

That is the sort of detail that only appears in a project that has been running for years, and it tells you something about the codebase. The pinned version is old, and it is pinned for a reason, namely that the in-memory Accumulo implementation used by the test suite does not work with a newer Accumulo. Anything built on top, including the quickstart, therefore has to override it, and the override is a maintenance burden that has been carried for long enough to be documented.

The rest of the repository is consistent with a large Java platform project. There is a Maven configuration directory, a root pom, a build instructions document, a release document for the microservices, a checkstyle configuration, a spotbugs exclusion file, an import control file for the Accumulo packages, and directories for the common test code, the commons, the contrib modules, the core, the coverage reports, the docker setup, the documentation, the git hooks, the properties, the warehouse, and the web services.

Getting started, and where the documentation lives

The README is a signpost rather than a manual, and it points at three places.

The easiest way to get started is the DataWave quickstart, and the documentation as a whole is on the same site. The basic build instructions are in a separate document in the repository, which is the file to read before you try to compile anything.

The Docker readme in the repository covers the deployment of the quickstart and of the microservices, and the microservice side has its own release document.

That spread of documentation is normal for a platform of this size, and the README's own structure helps you navigate it. The what it is section is four bullets, the how to use this repository section explains the sub-repository arrangement, the building section covers the Maven invocations, and the key binaries section explains the version constraints. Anything beyond that is in the linked documents.

For an organisation evaluating the project, the two things that decide it are the authorisation model and the operational footprint. The authorisation model is fine-grained and integrates with an existing authorisation service and PKI, and it is the reason the project exists separately from Accumulo. The operational footprint is a set of microservices, each of them independently versioned and deployed, on top of Accumulo, Zookeeper, Hadoop, and Wildfly.

That is a serious system, and it is aimed at a serious use case: fusing structured and unstructured data across tenants with different access rules, at a scale where a general purpose query engine would not give you the control over who sees which row.

Editorial conclusion

datawave fits an organisation that already runs Accumulo and Hadoop, that needs a query layer with row level authorisation tied to an existing identity system, and that has the operational capacity to run a dozen services. It does not fit a team starting from nothing, because the key binaries are Accumulo, Zookeeper, Hadoop, and Wildfly, and the build expects a specific toolchain. Verify first that you can run Accumulo and Zookeeper yourself, that the Java version you have matches the profile you build with, since a JRE only runtime fails with a confusing release version error, and that you have read the note about the service and starter versions being released separately, because a snapshot built from the core repository is not a coherent set of microservices.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. NationalSecurityAgency/datawave on GitHub
4. Project website
5. README