# Apache Bigtop: packaging and smoke tests for the Hadoop ecosystem

> Apache Bigtop builds community around the packaging and interoperability testing of the Hadoop ecosystem, producing RPM and DEB packages and a suite of smoke tests written in Groovy on the iTest framework. It is an Apache Foundation project under Apache 2.0, and the quickest way into it is a Docker provisioner that spins up a whole distribution with one click.

**apache/bigtop** — Bigtop is an Apache Foundation project for Infrastructure Engineers and Data Scientists looking for comprehensive packaging, testing, and configuration of the leading open source big data components.

- Repository: https://github.com/apache/bigtop
- Website: https://bigtop.apache.org
- Stars: 682 · Forks: 531
- Language: Groovy
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/apache-bigtop

## A test suite for the distribution, not the application

Apache Bigtop is a project for the development of packaging and tests for the Apache Hadoop ecosystem, and the description is more specific than the usual big data framing suggests. The primary goal is to build a community around the packaging and interoperability testing of Hadoop related projects, with testing at various levels including packaging, platform, runtime and upgrade, developed with a focus on the system as a whole rather than on individual projects.

That focus on the whole is the differentiator. Every Hadoop component has its own unit tests, and those tests pass on a developer's machine. What Bigtop does is test the assembled distribution, which is where the failures actually live: a package that installs but puts a config file in the wrong place, a service that starts but the client cannot reach it, a version mismatch between the client and the server.

The primary language of the repository is Groovy, which is the language of the smoke tests, and the source layout is organised by function. bigtop-deploy holds the deployment scripts and the Puppet configuration. bigtop-packages holds the RPM and DEB specifications for the subcomponents. bigtop-test-framework is the iTest utilities that the smoke tests are written against. bigtop-tests splits into test-artifacts, the source of the tests, and test-execution, the Maven pom drivers that run them. bigtop-toolchain holds the Puppet scripts for setting up a machine that can build Bigtop, and it sets up the utilities such as jdk, maven and protobufs.

The two directories that are the on-ramp for newcomers are the provisioner, described as a Docker provisioner that automatically spins up a Hadoop environment with one click, and the docker directory with the Dockerfiles and Docker Sandbox build scripts.

## The Docker provisioner is the fastest way in

The readme is direct about how to get a feel for the project, and the advice is to skip the theory and run something.

You cd into the provisioner directory and try out the docker recipes. It rapidly spins up and runs the Bigtop smoke tests on a local Bigtop based big data distribution. Once you have the gist, you can hack around with the recipes to learn how the Puppet, rpm and smoke test pieces fit together, going deeper into whichever component interests you.

That sequence, spin up then dissect, is a good one, and it mirrors how the project is built. The provisioner is a Docker environment assembled from the same Puppet code and the same packages that the rest of the project produces, so a recipe is a working example of a configuration rather than a simplified teaching version.

The wider repository supports the same idea. There is a bigtop-ci directory, a docker directory with the Dockerfiles, a build.gradle with a gradlew wrapper alongside a pom.xml, and a settings.gradle and a release.gradle, which tells you the build is in the middle of a transition. A separate Apache Bigtop blueprints project is also underway, aiming to create templates and examples that demonstrate and compare various components of the Hadoop ecosystem with one another.

For a project that has existed for a long time, the number of build files at the root is a fair signal of the scale of the work: Bigtop is not a library you consume, it is a build system for a family of distributions.

## Smoke tests are organised as one Maven submodule per component

The integration test structure is the part you will touch first if you are adding a component, and it is explained precisely.

The warning comes first. Since testing packages requires installing them on a live system, it is highly recommended to use VMs. Testing is done with the iTest framework, and the tests are organised in Maven submodules with one submodule per Bigtop component. The pom file at bigtop-tests/test-execution/smokes/pom.xml defines all the submodules to be tested, and each submodule sits in its own directory under smokes, for example smokes/hadoop/pom.xml and smokes/hive/pom.xml.

The new way, with the Gradle build in place, is a two step process. Step one installs the smoke tests for one or more components, either with gradle installTestArtifacts for the general case or gradle install-hadoop for just the Hadoop-specific ones. Step two runs the smoke tests on your cluster.

The project is on the route of migrating subprojects under a top-level Gradle build, and it says so. To see which projects have been converted you run gradle projects, and to see the tasks in a subproject such as itest-common you run gradle itest-common:tasks.

The old way is still documented for anyone with an existing Maven-based setup. It builds the smokes with snapshots, so that all transitive dependencies are in your repository, using mvn clean install with the skip flags for tests and integration tests and the perform release flag, against the bigtop-test-framework module.

Having both paths documented is practical rather than untidy, because a team part way through the migration has half its components on one build and half on the other, and the readme tells you exactly which command belongs to which.

## Who uses it, and how to contribute

The readme answers the use case question in three personas, which is a good way to work out whether the project is relevant to you.

Application developers working on Hadoop can download a Bigtop built Hadoop 2.0 virtual machine from the website, giving them a running pseudo-distributed cluster to test their code against. Cluster administrators and deployment people can run the smoke tests to check that their cluster works. Vendors can build their own Hadoop distribution, customised from the Bigtop pieces.

That third one is the sharpest use of the project, and it is the reason the packaging work exists. Building a redistributable Hadoop with a particular set of components and a particular configuration is a lot of packaging work, and Bigtop is the shared base for it.

The contribution guidance is organised the same way, by the kind of expertise you have. Puppet contributors work on the deploy and packaging tooling, with recipes for other tools such as Chef and Ansible also welcome. Groovy contributors write the smokes and the iTest framework. Maven contributors build the smokes and define the high level project. RPM and DEB contributors package Hadoop ecosystem projects into GNU/Linux installable packages for the popular distributions, which is how a new project gets added or an existing package improved. Hadoop users can contribute by using the smokes, improving them and evaluating their breadth.

Two contribution routes are singled out as unusually valuable. Contributing your own workloads enables testing projects against real use cases, and it is the mechanism by which the people who care about a specific use case verify it keeps working. Giving feedback, telling the maintainers how you use Bigtop and what was and was not great about it, is the other.

The community channels are the Apache ones, with JIRA issues under the BIGTOP project and the dev mailing list at dev-subscribe@bigtop.apache.org.

## An Apache Foundation project still shipping

The maintenance picture is positive, with the usual Apache governance caveats.

The repository is not archived, the default branch is master, and the last push was on 2026-09-28, about three weeks before the current date. The licence is Apache 2.0, with the standard ASF header in the readme, the LICENSE and NOTICE files at the root, and a.asf.yaml file that carries the foundation's own repository metadata.

There are no GitHub releases, which is normal for an Apache project. The version history lives in CHANGES.txt at the root, and BUILDING.txt is the file for building from source, with a Jenkinsfile.site for the website and a MAINTAINERS.txt listing the committers.

The readme itself is a good example of how the documentation has accumulated. The recipe list says that since Bigtop has different subprojects, the recipes will continue to evolve, and it points specific questions to the mailing list or JIRA. The source directory overview, the contribution list and the use case list read like a project that has been through several generations of its own tooling, which is exactly what a long established Apache project looks like from the inside.

The judgement to make is about fit rather than health. Bigtop is a community and a build system for distributions, not a tool you install to solve a problem. If your problem is proving that an assembled Hadoop stack works, or building your own branded distribution, it is exactly the right size. If your problem is running Hadoop, it is the wrong project entirely, and the ecosystem has plenty of other answers.

## Conclusion

Bigtop suits an infrastructure engineer who needs to build a vendor distribution of the Hadoop stack or to prove a freshly built cluster actually works, since that combination of packaging specs and smoke tests is what the project is for. It does not suit someone looking for a quick way to run Hadoop, since the purpose is testing rather than deployment, and the build tooling is mid-migration between Maven and Gradle. Before starting, cd into the provisioner directory and run the Docker recipes, then read the smoke test readme in bigtop-tests, and plan your integration runs on VMs because installing the packages touches a live system.

## FAQ

### What is Apache Bigtop for?

It is a project for the development of packaging and tests for the Apache Hadoop ecosystem, with a primary goal of building a community around packaging and interoperability testing at levels including packaging, platform, runtime and upgrade, focused on the system as a whole rather than on individual projects.

### How do I try Apache Bigtop without setting anything up?

Change into the provisioner directory and try the docker recipes. It spins up rapidly and runs the Bigtop smoke tests on a local Bigtop based big data distribution, and you can then modify the recipes to learn how the Puppet, rpm and smoke test pieces fit together.

### How are the Bigtop integration tests organised?

Testing uses the iTest framework, with the tests in Maven submodules and one submodule per Bigtop component, all listed in bigtop-tests/test-execution/smokes/pom.xml. With the Gradle build you install them with gradle installTestArtifacts or gradle install-hadoop, and the project is migrating subprojects to a top-level Gradle build.

### What should I know before running the Bigtop integration tests?

Testing packages requires installing them on a live system, so the documentation highly recommends using VMs. The project also documents both the new Gradle route and the older Maven route, where the smokes are built with snapshots so all transitive dependencies end up in your repository.

## Sources

- [apache/bigtop on GitHub](https://github.com/apache/bigtop)
- [Issues](https://github.com/apache/bigtop/issues)
- [License: Apache-2.0](https://github.com/apache/bigtop/blob/master/LICENSE)
- [Project website](https://bigtop.apache.org)
- [README](https://github.com/apache/bigtop/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/apache-bigtop