CommCare HQ: the server behind an offline-first field data platform

The server half of an offline-first platform

CommCareHQ is the server backend for CommCare, described as the world's largest platform for designing, managing and deploying offline-first mobile applications to frontline workers worldwide, and the repository is the web application for building complex, customizable frontline worker solutions.

That is a specific and demanding brief. The users are often health workers, survey enumerators or aid workers collecting data somewhere with poor connectivity, and the application has to keep working when the network does not. The offline-first part is in the description of CommCare itself, and the headquarters project is what the mobile client talks to.

The mobile side is a separate repository, commcare-android, and the readme also points at the bundled web application platform, formplayer, which is the thing that renders the forms inside the mobile app. So there are three pieces, and the server is one of them.

The features the server side provides are listed as eight components:

+ CommCare application builder
+ OpenRosa compliant XForms designer
+ SMS integration
+ Domain/user/mobile worker management
+ XForms data collection
+ Case management
+ Over-the-air (ota) restore of user and cases
+ Integrated web and email reporting

Two of those deserve a note because they explain a lot about the architecture.

Over-the-air restore of user and cases means a field worker who has lost their device can be resynchronised without a supervisor visiting them. That is a hard problem in an offline-first system, because the device holds state the server also holds and they have diverged, and it is the reason the case management component exists alongside data collection.

SMS integration is the other one. If a worker has no data connection at all, an SMS path is a genuine fallback, and it explains why the dependency list includes a phone number field library and a two-factor authentication package, because the server is authenticating people over channels that are not a browser.

The multi-tenant application building is what makes it a platform rather than a single application, and the domain, user and mobile worker management is the piece that carries the tenant boundary.

The project is BSD-3-Clause licensed, written in Python, and the last push was on 2026-10-20. There are no GitHub releases in the repository, and the homepage points at Dimagi's open source page rather than at a deployment of the project itself.

Three ways to get a CommCare HQ running

The readme points at three distinct arrangements, and they are for different users rather than different preferences.

To try CommCare there is a production instance of hosted CommCare. That is the fastest route for somebody who wants to build an application and see how it behaves, and it costs nothing but an account.

For setting up a local CommCare HQ developer environment, there is a separate document, DEV_SETUP.md in the repository, and the readme points at it as the thing to follow.

For a production environment, the answer is CommCare Cloud, described as the toolkit for deploying and maintaining CommCare servers, with its own documentation on Read the Docs. That is the important distinction: the open source repository is the application, and the tooling to run it in production is a separate product.

The additional documentation is on Read the Docs, and the community has a forum for questions.

The local environment has more than one route too. There is a DEV_SETUP.md, a DEV_SETUP_MAC.md and a DEV_FAQ.md, so the setup is documented for two platforms and there is a document for the questions that come up repeatedly while setting it up. There is also a local_dev_guide.rst at the repository root, which is a separate thing from the setup document and appears to be a guide to working on the code once it is running.

That layering of setup documents is a sign of a project where getting the environment running is a genuine obstacle rather than a formality, which is what you would expect from a stack this size. The.python-version file, the requirements/ directory, the pyproject and the uv.lock at the root, and the Dockerfiles at the top level all point the same way.

The repository also carries an AGENTS.md and a CLAUDE.md at the root, the same pattern of giving automated coding assistants their own instruction files.

What the pieces of the stack say about where the effort goes

The repository layout is large enough to be worth reading as a map of the project's concerns, because the directory names line up with the eight components the readme lists.

There is a corehq/ directory, which is the Django project itself, and a custom/ directory alongside it for the parts that are specific to CommCare rather than to Django. There is a deployment/ directory and a docker/ directory, which are the two things a Django application accumulates once it is running somewhere other than a laptop.

Then there are the directories that tell you what this application actually does. A locale/ directory with a .tx/ subdirectory and a .transifexrc.example file means the interface is translated through the Transifex platform, and given that the user base is frontline workers worldwide, that is not an optional feature. There is a testapps/ directory for the example applications the readme mentions, which matters because a data collection platform is only as good as its examples. There is a requirements/ directory of its own, with a Makefile inside it, which is how a project with a large and frequently adjusted dependency set keeps the resolution reproducible. And there is a submodules/ directory, so some parts are pulled in from other repositories rather than vendored.

The configuration files are as revealing as the directories. There is a settings.py at the root, a dev_settings.py, a settingshelper.py and a testsettings.py, with a localsettings.example.py for the machine-specific overrides. There is a .pytype.cfg and an .isort.cfg, both of which are older tools that are still in use here, and an .editorconfig. There is a .coveragerc and a codecov.yml, so coverage is measured and reported. There is a .readthedocs.yml, so the documentation build is configured for the Read the Docs hosting that the readme points to. And there is a .python-version file, which is the one the pyproject comments refer to when they explain how the version floor is kept in step.

The two lock files deserve one more look, because they are the least common thing in the tree and the most telling. There is a migrations.lock and a serializer-pickle-files.lock, both at the root, and the Makefile targets that generate them are named after the scripts that produce them. A lock file for migrations means the set of applied schema changes is treated as a fixed artefact rather than as a branch of history that moves with the code, and for a platform that has been running migrations against real data for years, that is the difference between an upgrade being a deployment and an upgrade being a research project.

The serializer pickle files are a more obscure artefact with a similar purpose. The setup script in the repository makes them, and the file name suggests it covers the serialised forms the project ships, which in a form-heavy application are effectively part of the schema. Locking them means a form definition that changes shape does not silently change the meaning of data already collected against the old one.

So the two lock files point at the same underlying concern from different directions, which is that in a data collection system the definition of the data is not the same thing as the code that processes it, and both have to be versioned deliberately.

The remaining root entries are the usual ones for a large Django application with a long history. There is a fabfile.py for deployment tasks, a Gruntfile.js for the front end build, a package.json for the JavaScript libraries, a git-hooks/ directory, an hqscripts/ directory for project-specific management commands, a scripts/ directory, and eslint/ and eslint.config.js for the front end linting. There is a get_resource_versions.py, a schemathesis.toml.example for API schema testing, and a SECURITY.md alongside a CONTRIBUTING.rst and a STANDARDS.rst.

A file called STANDARDS.rst next to a coding standards configuration is worth a note, because it implies the project treats its conventions as something to be argued for in prose as well as enforced by tooling. For a codebase that has been open to contributions for a long time, that is how the conventions survive the people who wrote them leaving.

And there is a .coderabbit.yaml, which configures an automated code review service, alongside the AGENTS.md and CLAUDE.md files. Three separate mechanisms for the same purpose, automated assistance to contributors, which tells you the project takes outside contributions seriously enough to invest in making them easier.

A Django project pinned to a single Python version

The Python configuration is where the project's conventions are visible, because the comments explain the decisions rather than just stating them.

name = "commcare-hq"
license = { file = "LICENSE" }
version = "1.0.0"
requires-python = ">=3.13"
dependencies = [
    "alembic",
    "architect",
    "CommcareTranslationChecker>=0.9.7",
    "django~=5.2.17",
    "djangorestframework",

The comment above the requires-python line is the useful part. The version of Python that will be installed when creating a new virtual environment is stored in the .python-version file, and may need to be updated if requires-python is changed here, with the tool to do it named. So the version floor is declared in two places and there is a documented command for keeping them in step.

That is a deliberate convention, and it explains something about the Docker image as well, which is built as the dimagi/commcarehq-pyX.Y image where X.Y is the version in the .python-version file, and which is used for running the tests. One Python version per test image, so a test failure can be attributed to a version rather than to a variable.

The dependency list is long and its comments are unusually specific. Some entries carry an initial describing who uses them and for what, so beautifulsoup4, bleach and the css extra of bleach are marked as used for rich text messaging emails and some tests. django-crispy-forms is pinned to a specific commit of a Dimagi fork with a git URL and a commit hash, and django-two-factor-auth is pinned to a Jazzband commit with the comment that they are waiting for the next release after 1.16.0.

So the pins are doing real work. Two packages are pinned to unreleased commits because the released versions do not have what the project needs, and one of those is the two-factor authentication package, which is the same component the SMS fallback depends on.

The framework is Django 5.2 with djangorestframework alongside it, and the rest of the list is the shape of a large Django application: celery with redis for background work, django-celery-results for storing it, django-redis, django-cte for common table expressions, django-tables2 and django-tastypie for tabular and API surfaces, django-recaptcha, django-otp for one-time passwords, django-oauth-toolkit, django-transfer for large file movement, django-field-audit for tracking model changes, and django-statici18n.

There is a datadog dependency and a ddtrace one, so the project is instrumented for tracing and metrics. There is an elasticsearch6 dependency with an upper bound below 7.0.0, which is the kind of pin that tells you the project is on a major version it has not migrated from. And there is a dropbox dependency, which is presumably a data import path.

The Makefile at the root has only five real targets and they describe the project's maintenance rhythm. The default target runs requirements, the serializer pickle files lock and translations. There is an upgrade-requirements target, a docs target that delegates to a Makefile in the docs directory, and the two lock file targets, which run a script to make the serializer pickle files and a shell script to make the translations.

So there are two lock files at the root, one for migrations and one for the serializer pickle files, and both are generated by a script rather than written by hand. The migrations one in particular is the interesting one, because a lock file for migrations means the migration set is treated as something to be pinned, which is how you keep a large deployment from applying changes it should not.

The Docker image, and what the apt list says about the dependencies

The Dockerfile is built as the dimagi/commcarehq-pyX.Y image and used for running the tests, and it is based on the astral-sh uv image for Python 3.13 on bookworm slim.

FROM ghcr.io/astral-sh/uv:0.7.17-python3.13-bookworm-slim
ENV PYTHONUNBUFFERED=1 \
    PYTHONUSERBASE=/vendor \
    PATH=/vendor/bin:$PATH \
    NODE_VERSION=24.11.1 \
    UV_COMPILE_BYTECODE=1 \
    UV_LINK_MODE=copy \
    UV_PROJECT_ENVIRONMENT=/vendor

The environment variables are the interesting part, because each one is a decision with a reason attached.

PYTHONUSERBASE points at a vendor directory and the PATH is prefixed with the vendor bin directory, so the dependencies are installed into a directory that is not the system site-packages, which is what makes the image work with the mounted volume that the comment refers to.

UV_COMPILE_BYTECODE is set to one, and the comment gives two reasons. It compiles bytecode during installation to improve startup time, and it suppresses a couchdbkit syntax error that happens during bytecode compilation. So one flag is doing an optimisation and a bug workaround at once, and the bug is in a CouchDB client rather than in the project's own code.

UV_LINK_MODE is set to copy, with the comment that it copies from the cache instead of linking since it is a mounted volume. Linking is faster and uses less space, but it breaks across a mount boundary, so the image trades speed for correctness there.

The apt install list reads like a list of everything the test suite needs rather than of what the application needs. There is build-essential, and the comment says it allows uv to build dependencies and increases the image size by 240 MB. There is a default JRE, which suggests a Java component is exercised in the tests. There is gettext for the translations, libmagic1 for file type detection, libpq5 for PostgreSQL, and the libxml2, libxmlsec1 and libxmlsec1-openssl set, with the comment that it is for xmlsec on Python 3.13. And there is google-chrome-stable, installed from the Google apt repository with its signing key added, which means the tests drive a real browser.

That last point is the one that explains the size of the image and the presence of the other packages. A project with a forms designer and a web application builder has a front end that has to be tested in a browser, and the browser tests need Chrome.

Node is installed separately at the end rather than through apt, by downloading the official tarball for the pinned version and extracting it into /usr/local with the leading path stripped. The version is set as an environment variable at the top, so the Node version is declared in one place and used in the download.

The top level of the repository also shows a JavaScript half that is much older than the Python half. The package manifest is named commcare-hq with the description CommCare HQ Javascript Libraries, and it is marked private, so it is never published. The dependency list is a long list of pinned versions, with a mix of carets and exact pins, and among the pinned ones are backbone, backbone.marionette, bootstrap 3.4.1 and bootstrap 5, d3, the mapbox drawing and geocoder libraries, ace-builds, the eonasdan datetimepicker, and turf for geospatial work.

The presence of both bootstrap 3 and bootstrap 5 in the same manifest, and of a Gruntfile.js alongside an eslint.config.js, suggests the front end tooling spans a migration from an older build system. There is a .yarnrc and a .tx/ directory with a .transifexrc.example for the translation platform.

The Gruntfile.js and the eslint.config.js side by side is a small snapshot of the project's history, and it is the kind of thing you only see in an application that has been in production for a long time and has been modernised incrementally rather than rewritten.

Editorial conclusion

CommCareHQ suits a development team building a data collection application for workers in low-connectivity environments, since the offline-first constraints are baked into the server design rather than bolted on, and the SMS and restore paths are the parts that are hard to retrofit. It does not suit you if you want a quick local install without a browser and a database, because the test image pulls in Chrome, a JRE and PostgreSQL libraries that reflect the real test surface. Before you set it up, read DEV_SETUP.md and the FAQ rather than guessing, and if you are deploying it yourself, budget for the fact that CommCare Cloud is a separate product from this repository and the tooling for running it in production is not in the code you can see here.

Frequently asked questions

Official sources

1. dimagi/commcare-hq on GitHub
2. Issues
3. License: BSD-3-Clause
4. Project website
5. README