Apache Livy: running Spark jobs over a REST API

Livy puts an HTTP endpoint in front of Spark

Apache Livy solves a narrow operational problem: how to submit and observe Spark code from a process that is not itself a Spark client. The project describes itself as an open source REST interface for interacting with Apache Spark from anywhere, and that claim is concrete rather than promotional. It supports executing snippets of code or whole programs in a Spark context that runs either locally or in Apache Hadoop YARN.

That split carries real weight. One HTTP front end can serve a developer poking at a Spark session from a laptop and a scheduler pushing batch programs from a machine that has never seen a Spark distribution. Where the compute actually lives stays a server side configuration rather than a decision each client has to make.

The project also states that no code change is required in the programs you run. For teams already operating Spark on YARN, the practical appeal is that submission no longer requires the submitting machine to hold a Spark distribution or speak the cluster's internal protocol.

The cost is that Livy becomes stateful infrastructure. Someone has to build it, deploy it, point it at a Spark installation, and keep it running. That is the trade the rest of this review examines.

Batch programs, REPL sessions, and impersonated multi-user access

Livy exposes several ways to get work into a Spark context. The project lists interactive Scala, Python and R shells alongside batch submissions in Scala, Java and Python, so the same server can hold an interactive session for one client while a scheduler sends batch programs to another.

The second capability is shared tenancy. The project states that multiple users can share the same server and points at impersonation support, which is the mechanism that stops one user's Spark context from acting with another user's identity when YARN is the execution backend. That is the feature that turns Livy from a developer convenience into a platform component.

The repository layout reflects the breadth. Directories named core/, server/, repl/, client-http/, python-api/, scala-api/ and thriftserver/ sit alongside integration-test/ and examples/, alongside a conf/ directory for server configuration. The tree tells you the project carries HTTP client code, a Thrift server variant and language specific APIs, but the tree does not tell you how session state is expired or how a session is reclaimed after a client disconnects. No session lifecycle is documented in the README, so the API reference on livy.apache.org is the authority on that point rather than the source tree.

The Spark version you build against and the one you run against are separate

The most useful operational fact about Livy is that the Spark version used at build time and the Spark version used at run time do not have to match. The default build targets Apache Spark 4.1.2 with Scala 2.13, and the documentation is explicit that the runtime version need not equal the build version, because Livy handles the differences between Spark versions internally.

The switch is the SPARK_HOME environment variable, set in the Livy server process. For anyone with an existing cluster this is the payoff: build once, then point the server at whichever Spark installation sits on that host, without recompiling Scala against every Spark line you support. The package itself ships no Spark distribution and is expected to work with any supported Spark version without a rebuild.

The requirement at the bottom of that range is Spark 3.0 or later. Anything older is outside what the project supports, so a cluster still on Spark 2 is not a candidate no matter how clean the HTTP interface looks. The corollary is operational rather than architectural: a Livy deployment is only as sound as the Spark installation behind it, and a version mismatch between the two is the first thing to check when the same code behaves differently under Livy than under a local Spark session.

Maven profiles pick the Spark and Scala line at build time

Livy is built with Apache Maven, and the version combination is chosen with profiles rather than edited by hand. The prerequisites listed for building are Maven from either the maven package or a maven3 tarball, a JDK, Python 3.x or later and R 3.x on Debian, Ubuntu, Redhat, CentOS and MacOS. The Python side of the build pulls in cloudpickle, requests, requests-kerberos, flake8, flaky and pytest.

The default path is three commands:

git clone https://github.com/apache/livy.git
cd livy
mvn package

After it completes you should have a Livy package that talks to Spark at run time, but no Spark itself, which is why a Spark installation is still required to actually run the server.

Four profiles adjust the build: -Phadoop2 selects Hadoop2 based dependencies, -Pthriftserver builds and tests the Livy Thrift Server modules, -Pspark3 selects Spark 3.x based dependencies, and -Pscala-2.12 selects Scala 2.12 based dependencies, which the project pairs with -Pspark3. To target Spark 3 with Scala 2.12:

mvn package -Pspark3 -Pscala-2.12

The pairing matters. -Pspark3 is documented to be used with -Pscala-2.12, so taking only one of the two flags gives you a build whose intent is ambiguous. Note also that -Pthriftserver is not merely a packaging switch, it runs the Thrift server tests as part of the build, which lengthens the cycle.

The Docker build path reuses your host Maven cache

A Dockerfile is provided for developers who do not want to assemble the toolchain on their own machine. It lives at dev/docker/livy-dev-base/Dockerfile and is referenced from the README as an alternative to the plain Maven build.

git clone https://github.com/apache/livy.git
cd livy
docker build -t livy-ci dev/docker/livy-dev-base/
docker run --rm -it -v $(pwd):/workspace -v $HOME/.m2:/root/.m2 livy-ci mvn package

The last line is the interesting part. It mounts the current checkout at /workspace inside the container and mounts the host Maven repository at /root/.m2, so the dependency downloads land in your existing cache and later runs do not fetch them again. The image is named livy-ci, which suggests it is aimed at continuous integration rather than production packaging, but it is the documented way to produce a build without installing Maven, a JDK, Python and R side by side on the host.

Two caveats are worth stating plainly. The cache mount means the container writes to your host's Maven repository, so a poisoned or partial download persists across runs. And the image is a build environment, not a server image: nothing in the README describes running a Livy server from it, so the deployment path still has to come from the site documentation.

The JDK prerequisite list contradicts the default build target

There is a genuine inconsistency in the project instructions, and it will cost you time if you follow them literally.

The prerequisite section lists openjdk-8-jdk on Debian and Ubuntu, java-1.8.0-openjdk on Redhat and CentOS, and Oracle's JDK 1.8 on MacOS. A separate note then states that the default build targets Spark 4.1.2 and requires JDK 17 or JDK 21 with Scala 2.13 and Hadoop 3.4.1, and that JDK 8, JDK 11 and Scala 2.12 are not supported by Spark 4. Supported Python versions for Spark 4.1 are given as 3.10 through 3.14.

So the documented toolchain and the documented default build target disagree. A developer who installs the listed JDK 8, then runs mvn package against Spark 4.1.2, hits a failure that has nothing to do with Livy's own code. This is not a subtle edge case either, since JDK 8 is still widely installed by default on many Linux images.

The resolution is to read the JDK requirement from the note about the default build, not from the prerequisite list, and to treat the prerequisite section as a description of what older Spark lines needed. If you must build the -Pspark3 -Pscala-2.12 variant instead, the older JDK 8 expectation becomes plausible again, though the project does not spell out the mapping and the README does not document a supported JDK matrix per profile.

No GitHub releases means commits are your version pinning

The repository has no GitHub releases, so there is no published tag to download and no changelog to read before upgrading. The documentation points to livy.apache.org for guides, example snippets and API documentation, which means the operational knowledge about versions lives outside the repository tree.

The last push to the default branch was on 2026-10-06, so the code is being worked on. That is a fact about the commit history rather than a release cadence, and it should not be read as a promise of packaged releases, because the project does not produce them through GitHub. Anyone running Livy in production needs a plan of their own for obtaining a known good build, whether that is an internal artifact repository, a container image of their own making, or a pinned commit built through the Maven path described earlier.

The licence is Apache-2.0, and the top level of the repository carries both a LICENSE file and a NOTICE file, which is the pairing used in Apache release packaging. Anyone redistributing a Livy build should look at how the NOTICE file is handled in their own packaging. That is a packaging question rather than a legal opinion, and the licence text itself governs.

Where Livy is the wrong tool

Livy is the wrong choice for a fixed batch workload that a single scheduler already knows how to submit. In that case the alternative approach is to run Spark yourself and have the submitting process talk to the cluster directly. The difference in approach is concrete: embedding the client in your own application hands a program to the cluster and ends there, while Livy keeps a long running server that owns a Spark context, hands out sessions over HTTP and multiplexes several clients through it. Direct submission means there is no service to operate, but also no shared interactive sessions, no impersonation layer and no remote API for clients that are not Spark clients at all.

Livy earns its overhead in the opposite situation: many clients, mixed interactive and batch work, and a shared YARN cluster where identity matters. There, the server amortizes the Spark setup and gives every client one contract.

One documentation gap is worth naming. The README lists no REST endpoints, no request bodies and no response shapes, and it does not document a session lifecycle or a rollback path. The API details live on the project's website, so treat the endpoint list there as something you must read and test before committing to a client integration.

Editorial conclusion

Adopt Apache Livy when you need one stable HTTP surface for Spark submission across several clients, YARN impersonation on a shared server, or interactive Scala, Python and R sessions, and skip it when a fixed batch workload can talk to the cluster straight from the submitting process. Verify three things in order: the Spark installation behind SPARK_HOME on the server host, the JDK you build with, since the default Spark 4.1.2 target needs JDK 17 or JDK 21 while the prerequisite list still names JDK 8, and the endpoint list on livy.apache.org, because the repository itself documents no REST routes. There is no GitHub release to pin, so pin the commit you build from.

Frequently asked questions

Official sources

1. apache/livy on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README