Data Transfer Project: a Java framework for service-to-service data portability

The problem DTP addresses: moving data between two services without a download in the middle

Export tools usually give you a file. You wait for the archive, download it, then upload it somewhere else. The Data Transfer Project targets a different path: the README states that DTP "eliminates the need to download data at all. Instead, data is transferred directly between service providers." The project was formed in 2017 as an open-source, service-to-service portability platform, and the repository is a collaboration of organizations contributing code to a common framework rather than a single vendor's product.

The audience is not end users. There is no consumer binary here. The people who benefit are engineers at online service providers who need to expose an import or export endpoint that another provider can call, and engineers who want to add an adapter for a service that is not yet covered. The README is explicit that DTP is looking for partner organizations and individuals to contribute, and that the architecture and implementation are still being defined.

That framing matters when you evaluate it. DTP is a framework and an ecosystem, in the README's own words, not a finished transfer service. If you arrive expecting a drop-in product, the repository layout will tell you otherwise: the top level is a set of Gradle modules (portability-api, portability-transfer, portability-spi-api, portability-spi-transfer, portability-types-common and others) rather than a single deployable.

How the Java modules fit together, from the SPI to the transfer worker

The module names encode the layering. portability-spi-api, portability-spi-cloud, portability-spi-service and portability-spi-transfer hold the service provider interfaces that an integrator implements. portability-types-common and portability-types-transfer hold shared data types, and portability-types-client holds client-facing types. portability-api is the HTTP surface, portability-transfer is the worker that actually moves data, and portability-api-launcher and portability-bootstrap-vm provide entry points.

The docker-compose.yml in the repository describes the runtime shape directly. Its comment calls the dtp service "the system under test: API + transfer worker in one JVM, as SingleVMMain requires." The reason is stated in the same comment: "LocalJobStore shares state through private static maps, so the two only ever see the same job when co-located." That is a real architectural constraint, not a deployment preference. Splitting the API and the worker across two JVMs would break job visibility with the default job store.

The compose file also notes that JettyTransport hardcodes port 8080, and that ApiMain "builds a JWTTokenManager unconditionally, so it will not boot without" JWT_KEY and JWT_SECRET. The comment adds that the values are never checked against anything, because the offline demo bypasses OAuth entirely. So the JWT variables are a boot requirement, not an authentication mechanism in the demo configuration.

Installing DTP and running a first transfer against the demo server

The repository does not ship a consumer installer. The supported path described in the repository is the Dockerfile, which exists so you can run the Java/Gradle build and test suite with no local JDK install. The Dockerfile comment gives two commands. The first builds an image tagged datatransferproject/dev, and the second runs the default check task with the source bind-mounted at run time. Note that the image is not rebuilt per code change; the source is mounted instead.

docker build -t datatransferproject/dev .
docker run --rm -v "$PWD":/workspace -v gradle-cache:/home/gradle/.gradle datatransferproject/dev

The Dockerfile pins eclipse-temurin:11-jdk-jammy. That pin is deliberate: the comment explains that build.gradle uses the legacy maven plugin, removed in Gradle 7+, so the build only works under the wrapper-pinned Gradle 6.9.2 recorded in gradle/wrapper/gradle-wrapper.properties, and that the JDK is pinned to 11 because the 6.9.2 and Groovy-DSL combination cannot parse newer bytecode when compiling build.gradle and settings.gradle. Expect the build to fail if you point it at a newer Gradle or JDK.

The docker-compose.yml file gives the three-service workflow in its header comment:

docker compose run --rm gradle      # the build
docker compose up dtp               # the server, on https://localhost:8080
./e2e/run.sh                        # the end-to-end transfer

The gradle service runs the Gradle build, and its ENTRYPOINT is ./gradlew, so arguments you pass to it are Gradle arguments. The dtp service runs the packaged demo-server jar that the build produces and publishes port 8080 so you can poke at a running DTP by hand. Its environment block sets JWT_KEY to e2e-key and JWT_SECRET to e2e-secret, which is the minimum needed for the API to boot in this configuration. The e2e service drives a transfer against the running server. The compose comment is candid that dtp reuses build: . rather than defining an image of its own, because the repository publishes no server image today, and that the harness pins a contract (an API on 8080 plus a log stream) rather than an artifact.

Where DTP is the wrong tool: no server image, an early-stage warning, and a single-JVM job store

Three limitations are visible without running anything. First, there is no published server image. The compose file says so outright and explains that the dtp service is a stand-in until one exists. If your deployment pipeline expects to pull a versioned container from a registry, you will be building that image yourself from this repository.

Second, the README carries its own caution: "Since the code is in active development, please do thorough testing and verification before implementing." It also states that DTP is "early-stage open source code that is built and maintained entirely by DTP community members." The most recent release listed is v1.0.4 from 2024-02-14, preceded by v1.0.3 in 2023-12-08 and v1.0.2 in 2023-11-13. The repository is not archived and its last push was on 2026-09-15, but the release cadence on the tagged versions is slow, and the README still describes the project as being in its early stages.

Third, the default job store couples the API and the worker. LocalJobStore keeps state in private static maps, so co-location in one JVM is a requirement rather than a tuning choice. If your architecture assumes a stateless API tier that can scale independently of background workers, DTP's default configuration does not match it. The compose comment implies a real job store would be needed to break that coupling, but the repository does not name one.

The toolchain pin is a fourth constraint. A build that requires Gradle 6.9.2 and JDK 11 because of the legacy maven plugin is a build you cannot casually modernize without touching build.gradle and settings.gradle.

DTP compared with a plain export archive or a custom API integration

The obvious alternative is the export-archive model that most services already offer: the user requests a download, receives a file, and uploads it to the destination. That approach needs no shared framework at all, and it works between any two services regardless of whether either side has heard of DTP. Its cost is exactly what the README points at: the user pays for bandwidth and time, and the README expects DTP to matter most "in global markets where downloading or uploading data is expensive and/or slow." So the difference is not features, it is where the bytes travel. Archive export moves data through the user's machine; DTP moves it between the two providers.

The second alternative is a bespoke point-to-point integration: you and one partner agree on an endpoint and a payload format. That is faster to ship for a single pair of services. DTP's bet is the opposite one: a common framework with shared data models and protocols, so that adding the tenth provider costs less than adding the second. That bet only pays off if other providers implement the same SPI. With a small set of participating services, a custom integration is less machinery for the same outcome.

A third comparison is a hosted portability service operated by a platform. DTP is not that. It is source code you build and run, with a demo server and an end-to-end test harness, and the README asks you to do your own testing and verification before implementing.

Maintenance, build cost, and what Apache-2.0 means for integrators

The maintenance picture is split. The repository is not archived and its last push was on 2026-09-15, so the codebase sees activity. The tagged releases, however, stop at v1.0.4 on 2024-02-14. Anyone tracking versions rather than commits should plan around that gap and decide whether they are consuming releases or the master branch.

Upgrade cost is dominated by the toolchain pin. The Dockerfile comment states that build.gradle uses the legacy maven plugin, removed in Gradle 7+, and that the build only works under the wrapper-pinned Gradle 6.9.2. The JDK is pinned to 11 for the same reason. Moving either forward is a source change, not a version bump, and the wrapper distribution is resolved and cached into an image layer at build time so the image works offline.

On licensing, the repository carries Apache-2.0, which is a permissive licence that generally allows commercial use and modification with the usual notice and attribution conditions. That is a general description of the licence family, not legal advice; if you are embedding DTP in a product, have your own counsel review the LICENSE file and any third-party dependencies the build pulls in. The README does not document a support contract, an SLA, or a deprecation policy, so there is no stated compatibility guarantee across releases.

What the demo configuration does not tell you about production

The compose setup is explicitly an offline demo. Its comment says the JWT values are never checked against anything because "offline-demo bypasses OAuth entirely." That means the credential path exercised by the end-to-end test is not the credential path a real deployment would use. LocalAppCredentialStore reads secrets from the environment, and ApiMain constructs a JWTTokenManager unconditionally, so the variables are needed for boot, but their values in the demo carry no security meaning. Do not read the demo's configuration as a template for authentication.

Similarly, the e2e harness pins a contract rather than an artifact. A passing end-to-end run tells you the API answers on 8080 and the transfer completes against the demo server; it does not tell you how the system behaves under concurrent transfers, because the default job store is a set of static maps in one JVM. The repository does not include a documented production job store, so that is a question to raise with the project rather than assume an answer to. The README points questions at [email protected], and the Developer documentation lives at Documentation/Developer.md in the repository.

Editorial conclusion

Adopt DTP if you are a service provider willing to write adapters against its SPI and run the transfer worker next to the API in one JVM, as SingleVMMain requires. Do not adopt it if you need a supported, packaged server: the repository publishes no server image, the README calls the code early-stage, and the newest release is v1.0.4 from 2024-02-14. Before committing, verify that the wrapper-pinned Gradle 6.9.2 build passes on your JDK 11 toolchain and that your storage layer can tolerate LocalJobStore's private static maps, which only work when the API and worker share a process.

Frequently asked questions

Official sources

1. dtinit/data-transfer-project on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases