# Datashare: a self-hosted document search engine for investigative teams

> Datashare is the search and analysis platform built by the International Consortium of Investigative Journalists: it ingests PDFs, emails, spreadsheets, images and archives, extracts text with OCR, enriches it with named entities, and exposes the result through a web UI and REST API on your own machines. It suits newsroom and research teams that cannot send sensitive documents to a third party, at the cost of running a stack of services.

**ICIJ/datashare** — A self‑hosted search engine for documents

- Repository: https://github.com/ICIJ/datashare
- Website: https://datashare.icij.org
- Stars: 768 · Forks: 73
- Language: Java
- License: AGPL-3.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/icij-datashare

## Search built around documents, not web pages

Datashare is an open source, self-hosted document search and analysis platform built by the International Consortium of Investigative Journalists. The short version in the repository header is a self-hosted search engine to find stories in any files, and the README is more specific: it ingests heterogeneous data such as PDFs, emails, spreadsheets, images and archives, extracts text including via OCR, enriches it with metadata and named entities, and exposes everything through a search UI and a REST API.

The reason this exists in the form it does is stated in the README itself. Because Datashare runs on your own machines, you keep full control over sensitive files and no external cloud services are required. For an investigative newsroom, a leaked document archive is exactly the kind of evidence that cannot be uploaded to a search service, and building a search index over it is a problem that general purpose tools do not solve.

The feature list translates that into concrete capabilities. Full-text search over PDFs, emails, office documents, images and archives. OCR on scans and images so that visual text becomes searchable text. Named entity extraction that detects people, organisations, locations and emails. Stars and tags for marking key documents. Advanced filters and operators combining facets with boolean, wildcard and fuzzy queries. Team or server mode for multi-user deployment with shared tags and recommendations. And a plugin architecture for custom modules.

The code is Java, the licence is AGPL-3.0, and the project publishes a User Guide, a Storybook for the frontend, a Cookbook file and a download link for the packaged distribution.

## The module split mirrors the processing pipeline

The repository layout makes the architecture legible, because each directory is a stage of the pipeline or a surface that talks to it.

There is datashare-extract for pulling text out of files, datashare-index for the indexing side, datashare-nlp-corenlp for the natural language processing work, and datashare-tasks for the asynchronous job handling. Around those sit datashare-api for the REST surface, datashare-app for the application service, datashare-cli for the command line entry point, and datashare-db for the database layer. There is also a commons-test directory for shared test fixtures, a datashare-dist module for the distribution package, and tools and docs directories.

The separation between extract, index and nlp is the part that matters for extension. A document arrives in datashare-extract, where its text is recovered and, if it is a scan, OCR supplies what is missing. The enriched text goes to datashare-nlp-corenlp for entity detection, and then to datashare-index where it is written into the search index. The plugin architecture, described in the feature list as a way to extend Datashare with custom modules, is what lets a team add a new extractor or a new enrichment step without forking the whole pipeline.

The search side of the system is Elasticsearch 8.x, with 7.x still supported, and the frontend is a separate client project published as a Storybook, with a REST API documented in the User Guide.

## The service stack you are committing to

Datashare is not a single jar you drop next to your application. The requirements section lists JDK 21, Apache Maven 3.8 or higher as the primary build tool for the backend, GNU Make for shortcuts, PostgreSQL 13 or higher with two databases expected, one for development named datashare and one named test, Elasticsearch 8.x, and Redis 5 or later for session storage and async task orchestration.

The docker-compose.yml in the repository root starts the whole set:

```bash
docker compose up -d
```

The compose file pulls its service definitions from a shared file in the .devcontainer directory, and the services it starts are amqp, s3mock, redis, postgres, elasticsearch, temporal and temporal-ui. Postgres also mounts an init.sql script into the container's entrypoint directory so the databases and their schemas exist on first start.

The Temporal service is worth flagging, because it is not mentioned in the requirements list but appears in the compose file, and it sits alongside Redis as the machinery for long running work. The S3 mock stands in for object storage. The AMQP broker is RabbitMQ-style messaging, and the README's default configuration table lists an amqpUri property pointing at amqp://guest:guest@localhost:5673.

So the honest count is five or six backing services plus the application, and a developer who has only run a single container before has a learning curve in front of them. That is the price of the self-hosted, no-cloud constraint, and it is the trade the project is making on purpose.

## Building the project: devenv, install and the Elasticsearch download

The build sequence is short once the services are up, and the Makefile is the entry point.

The first step is the development environment configuration. Datashare uses a properties file to configure service URIs for local development and testing, and running the devenv target creates a gitignored datashare-devenv.properties from the template file in the repository:

```bash
make devenv
```

The default URIs it expects are the ones in the configuration table: Elasticsearch at http://localhost:9200, Redis at redis://localhost:6379, an S3 mock at http://localhost:9090, the AMQP broker, and Postgres as a JDBC URL against a dstest database with a test password. The properties file is loaded automatically when running tests through the -Ddevenv.file system property, which is what lets the same suite run against services in containers or on localhost.

Then the build. The project is modular, and there are two relevant targets:

```bash
# Build and install all modules (runs migrations first, then jOOQ codegen)
make install

# Or build distribution JARs only
make build
```

The install and build targets automatically run database migrations before building, ensuring that the jOOQ sources are generated from the current schema rather than from a checked-in copy that has drifted.

There is also a target that fetches Elasticsearch for you. Datashare can automatically download and install Elasticsearch locally with make elasticsearch, and the distribution package made by make dist downloads Elasticsearch 8.x into ~/.local/share/datashare/elasticsearch/ if it is not already there, with an installation script that detects the platform, Linux or macOS, and the architecture, x86_64 or aarch64, and picks the appropriate build.

## Tests, Liquibase migrations and jOOQ generated sources

Datashare has both unit and integration tests, and the integration tests expect Postgres, Elasticsearch and Redis to be reachable, which is why the service stack comes first.

```bash
# Run the whole test suite
make test

# Or run a single module
mvn -pl datashare-api test

# Or a single test class
mvn -pl datashare-api -Dtest=org.icij.datashare.PropertiesProviderTest test
```

The three levels of granularity are useful in practice. A full make test run is the gate before a change goes out, the module level run is what you use while working in one part of the code, and the single class run is the inner loop.

Schema management is done with Liquibase, and the Makefile separates the two database operations that are easy to confuse. The migrate target applies database migrations, while generate produces the jOOQ sources and is documented as something to run after schema changes. The reset-db target resets the database and reapplies migrations, and it is marked destructive in the Makefile help text, which is the sort of label worth reading before typing it.

There is a Makefile lint target that applies linter corrections and a check target that checks codestyle, so the style gate is part of the build rather than a separate opinion. The release target creates a new release and requires NEW_VERSION=x.y.z, and the docker target builds the Docker image.

## The devcontainer path and the cost of the stack

For contributors, the repository carries a .devcontainer directory and a .gitsecret directory, and the README has a Devcontainer section covering prerequisites, starting the devcontainer, and build and project initialisation. The compose file at the root is itself partly an extension of the devcontainer compose file, so the two setups are meant to stay in step.

This is the part of the project where a prospective adopter should do its own arithmetic. The stack is PostgreSQL, Elasticsearch, Redis, an AMQP broker, Temporal, an S3 mock, and the Java application, and Elasticsearch is the one component that is genuinely expensive to operate, since it is designed for distributed search workloads and is awkward to run well on a single modest machine.

For a newsroom that already runs Elasticsearch and Redis, or for a research group that has a virtualised environment, this is manageable. For a small team that just wants to search a folder of PDFs, the operational cost of the platform is larger than the problem.

The offsetting argument is the one the README makes. Self-hosting means the sensitive files, which for investigative work can be sources' identities and unpublished reporting, never leaves the organisation, and there is no external cloud service in the path. The download link in the status table is there so that a team can evaluate the packaged distribution without building from source first.

## Conclusion

Adopt Datashare if your team works with documents you cannot place in a hosted service and needs full text search, OCR and named entity extraction over them, especially if you already run Elasticsearch, Postgres and Redis. Do not adopt it for a small internal file search, because the service count, the JVM and the Elasticsearch download make it heavy for that job, and do not treat the AGPL-3.0 licence as a detail to check later if you plan to modify and host it. Before your first build, start the services with docker compose up -d, create the development configuration with make devenv, and run the migrations and jOOQ generation through make install, noting that the latest release is 21.23.1 from 2026-10-16 and the last push was on 2026-10-19.

## FAQ

### What is Datashare and who built it?

Datashare is an open source, self-hosted document search and analysis platform built by the International Consortium of Investigative Journalists. It ingests PDFs, emails, spreadsheets, images and archives, extracts text including via OCR, enriches it with named entities, and exposes it through a web UI and a REST API.

### What services does Datashare need to run?

The requirements are JDK 21, Maven 3.8 or higher, PostgreSQL 13 or higher with a datashare and a test database, Elasticsearch 8.x with 7.x still supported, and Redis 5 or later. The provided docker-compose.yml starts amqp, s3mock, redis, postgres, elasticsearch, temporal and temporal-ui.

### How do I set up a Datashare development environment?

Run docker compose up -d to start the services, then make devenv to create a gitignored datashare-devenv.properties from the template. Build with make install, which runs database migrations and jOOQ code generation first, and run the suite with make test.

### Which licence does Datashare use?

The repository is licensed under AGPL-3.0, with the licence text in LICENSE.txt at the root of the repository.

### Does Datashare support OCR and named entity recognition?

Yes. The feature list includes OCR on scans and images, which turns visual text into searchable text, and named entity extraction that auto-detects people, organisations, locations and emails. There is a datashare-nlp-corenlp module for the language processing work.

## Sources

- [ICIJ/datashare on GitHub](https://github.com/ICIJ/datashare)
- [License: AGPL-3.0](https://github.com/ICIJ/datashare/blob/main/LICENSE)
- [Project website](https://datashare.icij.org)
- [README](https://github.com/ICIJ/datashare/blob/main/README.md)
- [Releases](https://github.com/ICIJ/datashare/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/icij-datashare