# elasticsearch-hadoop: connecting a Hadoop or Spark job to an Elasticsearch cluster

> elasticsearch-hadoop is Elastic's Apache-2.0 connector that lets Map/Reduce, Hive and Spark jobs read from and write to an Elasticsearch cluster through its REST interface, distributed as a small dependency-free jar. The judgement is that its value is the three integration surfaces and the tiny deployment footprint, and that the version-alignment rule is the operational tax you pay for it.

**elastic/elasticsearch-hadoop** —  :elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

- Repository: https://github.com/elastic/elasticsearch-hadoop
- Website: https://www.elastic.co/products/hadoop
- Stars: 1,966 · Forks: 992
- Language: Java
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/elastic-elasticsearch-hadoop

## One jar, three ways into Elasticsearch

elasticsearch-hadoop is described as Elasticsearch real-time search and analytics natively integrated with Hadoop, and it exposes three integration surfaces: Map/Reduce, Apache Hive and Apache Spark.

The deployment model is the first thing to understand. The requirement is stated as an Elasticsearch cluster accessible through REST, and that is all. The documentation says significant effort went into creating a small, dependency-free, self-contained jar that can be downloaded and put to use without any dependencies, and that the user simply makes it available to the job classpath.

The Map/Reduce section puts a number on it: the library is around 300 kB and has no dependencies. It can be bundled with the job, added through the Hadoop DistributedCache, or provisioned onto the cluster manually. A connector for a search engine is usually a tree of transitive dependencies that changes the resolution of the job, and this one deliberately is not.

The repository layout mirrors the three surfaces. There are `mr/`, `hive/` and `spark/` directories, plus `repository-hdfs/` for the Hadoop filesystem repository, `docs/`, `test/`, `qa/`, `dist/`, `buildSrc/` and a Gradle build with `gradlew` and `gradlew.bat` wrappers. There is also a `catalog-info.yaml` and a `renovate.json` for dependency updates, and a `NOTICE.txt` alongside `LICENSE.txt`.

The homepage is `https://www.elastic.co/products/hadoop`, and the project is Apache-2.0 licensed.

## Configuration is a handful of es-prefixed properties

The configuration surface is deliberately narrow. All configuration properties start with the `es` prefix, and the `es.internal` namespace is reserved for the library's own use and must not be used by the user at any point. The properties are read mainly from the Hadoop configuration, though the user can specify some of them directly depending on which library is in use.

There is one required property:

```bash
es.resource=<ES resource location, relative to the host/port specified above>
```

And three that the documentation calls essential:

```bash
es.query=<uri or query dsl query>              # defaults to {"query":{"match_all":{}}}
es.nodes=<ES host address>                     # defaults to localhost
es.port=<ES REST port>                         # defaults to 9200
```

Two details are worth pausing on. The `es.query` default is a full query DSL document, `{"query":{"match_all":{}}}`, not a URI, and the property accepts either form, so a job that reads an entire index needs no query at all. And the default host and port are localhost and 9200, which is right for a job running on the same machine as the cluster and wrong for a distributed job, where the node list has to be set explicitly.

The full property list lives in the configuration chapter of the online documentation rather than in the repository, so the README gives you the shape of the configuration and the docs give you the whole list.

## Map/Reduce reads and writes through InputFormat and OutputFormat

The Map/Reduce integration is the low-level path, and the documentation positions it for basic, low-level or performance-sensitive environments. It provides dedicated `InputFormat` and `OutputFormat` classes that read and write data to Elasticsearch, and the API is exposed through `EsInputFormat` and `EsOutputFormat`.

Reading is a configuration object plus a class assignment:

```java
Configuration conf = new Configuration();
conf.set("es.resource", "radio/artists");
conf.set("es.query", "?q=me*");
Job job = new Job(conf)
job.setInputFormatClass(EsInputFormat.class);
...
job.waitForCompletion(true);
```

Writing is the same shape with a different format class:

```java
Configuration conf = new Configuration();
conf.set("es.resource", "radio/artists");
Job job = new Job(conf)
job.setOutputFormatClass(EsOutputFormat.class);
...
job.waitForCompletion(true);
```

The `es.query` value in the read example is a URI-style query, `?q=me*`, and the comment in the code says to replace it with the relevant query. The `es.resource` value in the write example carries a different comment, index or indices used for storing data, which is the same property with a different role depending on direction.

The one thing the documentation does not spell out for Map/Reduce is write conflict handling. The property list shown covers only the resource, the query, the node and the port, so how a job that writes duplicate documents behaves, whether it overwrites or fails, is not answered here. That is a question worth taking to the reference documentation before you point a production job at an index.

The alternative to this path within the same project is Hive or Spark, which is the subject of the following sections.

## Hive gets an external table over an index

The Hive integration is a storage handler, `org.elasticsearch.hadoop.hive.EsStorageHandler`, which lets you define an external table on top of an Elasticsearch index and then use it like any other Hive table.

The jar goes on the auxiliary path or is registered in the Hive script, and the documentation recommends the script route:

```bash
ADD JAR /path_to_jar/es-hadoop-<version>.jar;
```

Reading defines the table with a query in the table properties:

```SQL
CREATE EXTERNAL TABLE artists (
    id      BIGINT,
    name    STRING,
    links   STRUCT<url:STRING, picture:STRING>)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'radio/artists', 'es.query' = '?q=me*');
```

The table columns are mapped to the JSON when talking to Elasticsearch, which is why the nested `links` field is declared as a `STRUCT` rather than a string. The `TBLPROPERTIES` carry the resource location and, for reading, the query.

Writing uses a similar definition with a different `es.resource`, and any data passed to the table is passed down to Elasticsearch:

```SQL
INSERT OVERWRITE TABLE artists
    SELECT NULL, s.name, named_struct('url', s.url, 'picture', s.picture) FROM source s;
```

Once the table is defined, querying it is ordinary Hive SQL, for example `SELECT * FROM artists;`.

The Hive path is the one most likely to be the right choice for an existing warehouse. If your data is already in Hive tables and you want it searchable in Elasticsearch, an external table with a storage handler avoids writing a custom job. The cost is that the mapping between Hive types and Elasticsearch JSON is expressed as column declarations, so a schema change means changing the table definition.

## Spark artifacts are keyed by Scala and Spark version

The Spark integration is the one with the most matrix-like installation, and the reason is that the connector is published as separate artifacts for each supported Scala and Spark combination.

For Scala 2.12 with Spark 3.0 through 3.5, the artifact is `elasticsearch-spark-30_2.12`. For Scala 2.13 with Spark 3.2 through 3.5, it is `elasticsearch-spark-30_2.13`. For Scala 2.13 with Spark 4.0, it is `elasticsearch-spark-40_2.13`, and for Scala 2.13 with Spark 4.1 or 4.2, it is `elasticsearch-spark-41_2.13`.

```xml
<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch-spark-41_2.13</artifactId>
  <version>9.0.0</version>
</dependency>
```

The supported Spark versions are 3.0 through 4.2. The Scala support is narrower and asymmetric, which is the detail to check before you pick a version. Only Scala 2.12 is supported for Spark 3.0 and 3.1. Both Scala 2.12 and 2.13 are supported for Spark 3.2 through 3.5. Only Scala 2.13 is supported for Spark 4.0 and later. Spark 4.2 users should use the `elasticsearch-spark-41` artifact.

The Maven coordinates for the core jar are the same shape as for Spark, without the Scala suffix:

```xml
<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch-hadoop</artifactId>
  <version>9.0.0</version>
</dependency>
```

The examples in the documentation use version 9.0.0 as the stable release. The tag list shows releases moving in parallel across major lines, with 8.19.23, 9.4.8 and 9.5.5 all published on 2026-10-06, which is what a maintenance policy across several supported Elasticsearch versions looks like in practice.

## Version alignment is the operational rule

The compatibility guidance is short and worth internalising before you deploy anything.

While an effort has been made to keep ES-Hadoop backwards compatible with older versions of Elasticsearch, the documentation says it is best to use the version of ES-Hadoop that is the same as the Elasticsearch version, and points to the product compatibility support matrix for details.

That is the tension in the project. Backwards compatibility is attempted but not guaranteed, so a connector that is a version or two behind your cluster might work, might work partially, or might fail in ways that are hard to diagnose. The safe practice is to match them, which means the connector participates in your cluster upgrade cadence rather than running on its own schedule.

The supported platform ranges are stated too. ES-Hadoop is developed for and tested against Hadoop 2.x and 3.x on YARN. Spark support runs from 3.0 to 4.2 with the Scala restrictions above.

The last push to the repository was on 2026-10-18, so the code is actively being changed, and the recent release dates show several lines being maintained at once. The build is Gradle with a wrapper, and the repository includes `docs/` for the source of the reference documentation, `thirdparty/` and `licenses/` for the dependency and licence accounting, and a `catalog-info.yaml` for internal catalogue metadata.

The licence is Apache-2.0 with `LICENSE.txt` and `NOTICE.txt` at the root, which is a permissive grant with no copyleft obligation for a connector you embed in a data platform. That is a licensing observation rather than legal advice, and `LICENSE.txt` governs.

## Conclusion

elasticsearch-hadoop fits a data platform team that already runs Hadoop or Spark and needs Elasticsearch to be searchable from those jobs, or Hive users who want an external table over an index, and who value a single self-contained jar with no transitive dependency tree. It does not fit a team that wants a shared write path for many concurrent producers, since the connector talks to the cluster over its REST interface and the documentation does not describe a bulk-ingestion or conflict-handling policy beyond the configuration keys listed. Verify first that the ES-Hadoop version you deploy matches the Elasticsearch cluster version, since the documentation is explicit that it is best to use the same version, that your Scala and Spark combination maps to the right artifact, for example `elasticsearch-spark-41_2.13` for Spark 4.1 and 4.2, and that the `es.resource`, `es.query`, `es.nodes` and `es.port` properties resolve to the index and cluster you expect in the job configuration.

## FAQ

### How do I add elasticsearch-hadoop to a Maven project?

Add the `org.elasticsearch:elasticsearch-hadoop` dependency, or use the stand-alone ZIP download. The documentation examples use version 9.0.0, and the project states it is best to use the ES-Hadoop version matching your Elasticsearch version.

### Which configuration properties does elasticsearch-hadoop require?

Only `es.resource` is required, pointing at the index relative to the configured host and port. The essential ones are `es.query`, which defaults to a match-all query DSL document, `es.nodes`, which defaults to localhost, and `es.port`, which defaults to 9200. The `es.internal` namespace is reserved for library use.

### How do I read from Elasticsearch in a Hive query?

Create an external table stored by `org.elasticsearch.hadoop.hive.EsStorageHandler` and put the index and query in `TBLPROPERTIES` as `es.resource` and `es.query`. The table columns are mapped to the JSON, and the table can then be selected from like any other.

### Which elasticsearch-hadoop artifact should I use for Spark 4.1 with Scala 2.13?

Use `elasticsearch-spark-41_2.13`. Spark 4.0 uses `elasticsearch-spark-40_2.13`, Spark 3.2 to 3.5 with Scala 2.13 use `elasticsearch-spark-30_2.13`, and Spark 3.0 to 3.5 with Scala 2.12 use `elasticsearch-spark-30_2.12`.

### Does elasticsearch-hadoop need extra libraries on the job classpath?

No. The documentation describes it as a small, dependency-free, self-contained jar of around 300 kB with no dependencies, which you make available to the job classpath by bundling it, using the Hadoop DistributedCache, or provisioning it onto the cluster.

## Sources

- [elastic/elasticsearch-hadoop on GitHub](https://github.com/elastic/elasticsearch-hadoop)
- [License: Apache-2.0](https://github.com/elastic/elasticsearch-hadoop/blob/main/LICENSE)
- [Project website](https://www.elastic.co/products/hadoop)
- [README](https://github.com/elastic/elasticsearch-hadoop/blob/main/README.md)
- [Releases](https://github.com/elastic/elasticsearch-hadoop/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/elastic-elasticsearch-hadoop