CDAP: an application platform layer for the Hadoop ecosystem

A platform layer over Hadoop

CDAP is described as an integrated, open source application development platform for the Hadoop ecosystem, and the README is precise about what that means. It is a layer of software running on top of Apache Hadoop platforms such as Cloudera Enterprise Data Hub or the Hortonworks Data Platform, not a replacement for them.

The problem it addresses is the fragmentation of the Hadoop world. Different teams write MapReduce jobs, then Spark jobs, then Hive queries, each against different APIs, and none of them share a way to describe a pipeline, track where data came from, or deploy the same code to a different cluster. CDAP's answer is a set of abstractions over that variety: logical representations of underlying data, decoupling of applications from the infrastructure they run on, and services that make application creation faster.

Two capabilities stand out in the README's list. Metadata management that automatically captures metadata and lineage, which is the part that saves an organisation from maintaining data lineage by hand. And CDAP pipelines with an integrated UI for click-and-drag development, which is how a non-Java analyst gets a pipeline onto the cluster without writing a job class.

The framework lands applications and data on existing infrastructure such as HBase, HDFS, YARN, MapReduce, Hive and Spark, and the README says you can run anything from simple MapReduce jobs and complete ETL pipelines up to enterprise-scale data-intensive applications. The primary language is Java, the project is by Cask Data, Inc., and the default branch is develop.

For an engineer arriving today, the honest first question is which Hadoop generation still runs this. The answer shapes everything else.

Where CDAP sits between the developer and the cluster

CDAP exposes developer APIs for creating applications and accessing core CDAP services, and the vocabulary it introduces is worth understanding before you read any code.

The two central abstractions are the application and the data. An application is a pipeline composed of stages, and a dataset is the logical representation of something stored on the cluster, with the physical location decoupled from how your code refers to it. That decoupling is what the README means by portability of applications through decoupling underlying infrastructures: a pipeline written against a logical dataset can be deployed against a different physical store without rewriting the pipeline.

On the execution side, CDAP defines and implements a collection of services that place applications and data onto the existing infrastructure. A Spark application is not a Spark job you launch; it is a CDAP application whose stages run on Spark, and the same application definition can run a different stage on MapReduce. This is also why the module list includes separate API artifacts per Spark version, including cdap-api-spark3_2.12 for Spark 3 on Scala 2.12.

Around that core sit the extensions, and the naming scheme in the repository shows the pattern clearly. There is cdap-authenticator-ext-gcp, cdap-credential-ext-gcp-wi, cdap-encryption-ext-tink, cdap-messaging-ext-spanner, cdap-metadata-ext-spanner, cdap-runtime-ext-dataproc, cdap-messaging-spi, cdap-metadata-spi, cdap-master-spi, cdap-log-publisher-spi and cdap-event-reader-spi. The spi suffix marks a service provider interface, which is the extension point, and the ext suffix marks a concrete implementation. Each SPI therefore has a matching extension, which is the mechanism by which a deployment swaps in a cloud provider's identity, key management or messaging without touching pipeline code.

That is the architecture in one sentence: pipeline definitions above, services below, and a pluggable seam between them for the parts that differ per vendor.

Building the whole thing with Maven

The build is the straightforward part, and the README is explicit about what it produces.

The prerequisites are three. JDK 8 or newer is required to run CDAP, with JAVA_HOME set. Node.js is required to run the CDAP UI, with any version greater than v4.5.0 recommended. Apache Maven 3.0 or newer is required to build the example applications, and 3.1 or newer to build CDAP itself.

The build sequence from source is short:

  git clone https://github.com/caskdata/cdap.git
  cd cdap
  mvn clean package

After that completes you have built all the modules for CDAP. For other build options the README points at BUILD.rst in the repository root.

One detail is worth flagging before you run it. The clone URL in the README points at caskdata/cdap, while this repository is cdapio/cdap, which suggests the project was moved to a new organisation and the README was not fully updated. Anyone building today should clone the cdapio/cdap repository, which is the one the releases and the default branch belong to.

The scale of the build is the other thing to plan for. The module list runs to dozens of artifacts, from cdap-api and cdap-client through cdap-cli, cdap-common, cdap-gateway, cdap-master, cdap-app-fabric, cdap-data-fabric, cdap-distributions and cdap-docs, plus separate test modules such as cdap-app-fabric-tests, cdap-data-fabric-tests, cdap-client-tests, cdap-cli-tests, cdap-common-unit-test, cdap-integration-test and cdap-e2e-tests. A full mvn clean package on a developer machine is a long build, which is why the developer quickstart guide in DEVELOPERS.rst and the Gitpod configuration in .gitpod.yml exist.

The module layout and what it reveals

The top level directory listing is long enough to be worth reading as documentation, because it shows how a platform of this kind decomposes.

The API layer is cdap-api with cdap-api-common and the version-specific Spark APIs. The client side is cdap-client, and cdap-cli is the command line interface, with cdap-cli-tests alongside it. The fabric modules, cdap-app-fabric and cdap-data-fabric, are the runtime that turns application definitions into jobs on the cluster, and each has a matching tests module. cdap-formats covers the data format handling, cdap-proto holds the protocol definitions, cdap-gateway and cdap-master are the service layer, and cdap-features and cdap-coverage are build-time machinery for feature flags and test coverage respectively.

Then there is the extension layer, described above, and the integrations: cdap-kafka, cdap-elastic, cdap-kubernetes, cdap-distributions for packaging, cdap-docs and cdap-docs-gen for the documentation build, cdap-app-templates for application scaffolding, and cdap-operational-stats-core for operational telemetry.

Two things in the listing stand out as signals about the project's shape. The first is the presence of .rubocop.yml and .rubocop_todo.yml, which means parts of the tooling surface are Ruby rather than Java, most likely the build and release scripts, and the todo file is a record of unfixed offences. The second is SECURITY.md alongside CODE_OF_CONDUCT.md, which shows the project treats governance as part of the repository rather than as a document on a website.

The docs build deserves a note too. The developers manual lives in the source distribution at cdap-docs/developers-manual/source and is also published online, and the README links a collection of it under the Developers' Manual. For a project of this age and size, that manual is the real specification; the README is an introduction.

Where CDAP fits and where it does not

CDAP is a well-built answer to a problem that the Hadoop ecosystem has since moved past in several directions, and being clear about that is the most useful thing an evaluation can produce.

The first constraint is the platform. The README names Cloudera Enterprise Data Hub and the Hortonworks Data Platform as the platforms CDAP runs on, and both vendors have changed shape considerably since. A CDAP deployment assumes a specific generation of Hadoop, and the Spark API artifacts, one of them pinned to Spark 3.2 on Scala 2.12, tell you which one.

The second is the Java and Node.js baseline. JDK 8 is the stated runtime requirement and Node.js is needed for the UI. That is fine for an enterprise estate with a managed runtime, and awkward for a team that has standardised on a current container base image.

The third is the size of the commitment. cdap-distributions, cdap-docs-gen and the e2e test modules all point to a platform that is deployed as a product rather than used as a library. You are not adding a dependency, you are standing up a control plane with a gateway, a master service, an application fabric and a data fabric, and then deploying applications into it. The README is clear that the same application runs in a Sandbox, on-premises Enterprise, or in the Cloud, which means the deployment target is a choice you make deliberately rather than an accident of configuration.

The fourth is the pace. The releases tell the story: v6.11.0 in March 2025, v6.11.1 in August 2025, and v6.10.1 before that in May 2024. The last push was on 2026-10-19, so commits continue to land on develop, but the release tags have not moved in a way that suggests a monthly or quarterly cadence.

None of that makes the project bad. It makes it a fit for a specific estate. If you still run Cloudera or Hortonworks Hadoop and want a governed pipeline platform with lineage, CDAP remains a coherent choice.

An Apache-style contribution process

The contribution model is described in unusual detail for a commercial-origin project, and it says something about how the code is governed.

The README describes a simple pull-based development model with a consensus-building phase, similar to Apache's voting process. The mechanics are laid out step by step: discuss large changes on the cdap-dev mailing list first, fork the repository, create a topic branch, work on the code, open a pull request with all description fields filled in, and after review the code is committed to the caskdata/cdap repository.

Two mailing lists carry the discussion. cdap-user is for people using the product to build applications, where you can expect questions from users, release announcements and general discussion. cdap-dev is for developers actively working on the product, and it is the forum for design, architecture and technical decisions going forward, and it also receives all the JIRA and GitHub notifications.

Issues are filed on a separate tracker at issues.cask.co rather than on GitHub Issues, and the README shows an example of browsing a project filter with a JQL query, which tells you the tracker is a Jira instance with its own query language.

For a contributor, the practical guidance is in DEVELOPERS.rst, which the README calls the Contributor Quickstart Guide, and in BUILD.rst for the build options. There is also a CODE_OF_CONDUCT.md and a SECURITY.md in the repository root.

The consensus phase is the part that will slow you down, and it is deliberate: it is the mechanism that keeps design decisions from being made unilaterally. Whether that suits your organisation depends on how much you value that consistency against how quickly you need to land a change.

Licence, trademark and the Cask lineage

The licence position needs a sentence of care because the repository metadata and the README do not say quite the same thing.

The README states that CDAP is licensed under the Apache License, Version 2.0, with the copyright line reading 2014-2017 Cask Data, Inc., and points at the Apache 2.0 text. The repository itself is classified by GitHub as carrying a custom licence that cannot be classified, with LICENSE.txt at the root, and the file listing also includes a KEYS file, which is the usual companion for verifying signed release artefacts. The practical reading is that the Apache 2.0 grant applies, and that a project with an approved-licence process should confirm the terms in LICENSE.txt rather than rely on the badge. This is not legal advice.

There is also a trademark notice. Cask is a trademark of Cask Data, Inc., and the README notes that Apache, Apache HBase and HBase are trademarks of the Apache Software Foundation, used with permission and with no endorsement implied. For a product that ships a layer named CDAP in its own interface, that distinction between the licence and the marks is worth keeping straight.

The other thing the README documents is where the project came from and where it lives now. The code is by Cask Data, the documentation is on docs.cask.co, the issue tracker is on issues.cask.co, and the clone URL still points at the caskdata organisation while the repository itself sits under cdapio. The default branch is develop rather than master, which is consistent with a project that runs a stable release line and a development line in parallel.

For anyone planning a long-term commitment, the useful question is not the licence but the stewardship: who merges the pull requests, who runs the release tags, and where the issues are actually filed.

Editorial conclusion

Adopt CDAP if you are running Cloudera or Hortonworks style Hadoop estates and want one programming model with a pipeline UI, automatic lineage and a single-node sandbox for development before deployment. Do not adopt it for a modern cloud data stack, since the platform assumptions, the JDK 8 baseline and the Cask Data lineage all point to the Hadoop era rather than the current lakehouse tooling. Before you plan around it, clone the repository and run mvn clean package with Maven 3.1 or newer, confirm your cluster version is one CDAP supports, and read the developers manual in cdap-docs/developers-manual/source, because the README is an introduction rather than a specification.

Frequently asked questions

Official sources

1. cdapio/cdap on GitHub
2. Issues
3. Project website
4. README
5. Releases