Apache Toree: a Scala kernel that turns a Spark cluster into a notebook
Mirror of Apache Toree (Incubating)
At a glance
- What is it?
- Apache Toree (Incubating) is a Jupyter Notebook kernel that gives a notebook a preconfigured Spark context and interprets Scala snippets against it, implementing Jupyter message protocol 5.0. It suits Scala data engineers who want to explore a cluster interactively, and it does not suit anyone who needs Python, R or SQL kernels or a plugin that loads on its own.
- Who is it for?
- Apache Toree is worth trying for a Scala team that lives in notebooks and wants to drive a real Spark cluster interactively, because it gives you a preconfigured context, understands the current Jupyter message protocol, and ships as a pip installable that registers itself as a kernel. It is the wrong tool for a Python-first data stack, for a production job, and for anyone who assumes the Spark Monitor plugin is bundled.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 4 days ago.
- What is it written in?
- Mainly Scala, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on October 22, 2026, and from our analysis. They are not legal advice.
Editorial analysis
The kernel gives a notebook a Spark context it did not have to create
Apache Toree (Incubating) is a Jupyter Notebook kernel, and its stated goal is to provide the foundation for interactive applications that connect to and use Apache Spark using the Scala language. It supports Scala and implements the latest Jupyter message protocol, version 5.0, so it plugs into current Jupyter and IPython releases for quick interactive data exploration.
The mechanism is described plainly: Toree provides an interface that allows clients to interact with a Spark Cluster, and clients send libraries and snippets of code that are interpreted and executed using a preconfigured Spark context. The word preconfigured is the part that matters. In a notebook, the usual pattern is that every cell has to build or find its own session and context, and the first cell of a session is always boilerplate. With Toree that work has already happened, so the first cell can be a query.
The list of things those snippets can do covers the interactive workflow rather than a single use case: define and run spark jobs of all kinds, collect results from spark and push them to the client, load necessary dependencies for the running code, and start and monitor a stream. That last one is what turns the kernel from a query tool into something you can explore an event stream with.
The repository is a mirror of the Apache project, laid out in Scala with separate modules for the kernel, the protocol, the communication layer, the scala-interpreter, the sql-interpreter, the kernel-api, plugins, client code, macros and the spark-monitor-plugin.
make is the entry point, and sbt is not to be used directly
The build system is unusual enough to be worth reading twice. The project uses make as the entry point for build, test and packaging, and a local build requires sbt, jupyter or ipython, and the other development requirements installed on your machine. The instruction is blunt: do not use sbt directly.
The everyday commands are make dev, which starts a Jupyter notebook server configured for Spark local mode, accessible depending on your mode at http://localhost:8888 or at http://192.168.44.44:8888, and make test for the test suite. From there you can create notebooks that use Toree configured for Spark local mode.
The example notebooks have their own target, make jupyter, which launches a notebook server in a Docker container with Toree and other dependencies installed. The notebook is then at http://<ip>:8888/, with the address depending on your Docker setup.
Individual components can be built with sbt, but only after the make-level setup is in place. The main Toree assembly jar, without the spark-monitor-plugin, comes from sbt assembly and lands at target/scala-2.12/apache-toree-assembly-<VERSION>.jar. The plugin has its own assembly at spark-monitor-plugin/target/scala-2.12/apache-toree-spark-monitor-plugin-assembly-<VERSION>.jar, and sbt compile builds everything. The note attached to that is important: the spark-monitor-plugin is now built as a separate jar and is not included in the main Toree assembly.
The Makefile shows the version scheme, with BASE_VERSION defaulting to 0.6.0.dev0 and VERSION adding the -incubating suffix, along with the APACHE_SPARK_VERSION default of 3.4.4 and SCALA_VERSION default of 2.12.
Installing the kernel needs a local Spark distribution
The install section states its prerequisite plainly: you need a distribution of Apache Spark downloaded to the system where Apache Toree will run. Then the install is two commands:
pip install --upgrade toree
jupyter toree install --spark_home=<YOUR_SPARK_PATH>The second command is what registers Toree as a Jupyter kernel, and the spark_home argument is why the Spark distribution has to be on the machine. This is a different arrangement from the PySpark kernel that ships with Spark itself, which knows where its own Spark home is.
The same two commands work for a development snapshot, with the pip install pointing at a release URL instead of the package name. Snapshots are published under the Apache dist dev path for the incubator project, and the documentation gives the URL shape for the toree package, for example https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshot, followed by the same jupyter toree install command.
The packaging step mirrors that split. make release produces four artifacts: a source release tarball, a binary package that contains the JAR and executable, a pip installable package that adds Toree as a Jupyter kernel published to PyPI as toree, and the same pip package published under the name apache-toree. The version placeholder used in the artifact names is the full release version, for example 0.6.0-incubating, and the base version is that same string without the -incubating suffix. The release target uses docker, so a release build needs a working Docker installation.
The Spark Monitor plugin has to be pointed at by absolute path
The spark-monitor-plugin is the clearest example of a change that the documentation explains because it confuses people. To enable it in a Toree application, you specify the path to the plugin JAR when starting Toree, and there are three ways to do that.
The command line parameter passes the plugin as a magic URL to the assembly jar:
# Start Toree with spark-monitor-plugin enabled
java -jar target/scala-2.12/apache-toree-assembly-<VERSION>.jar --magic-url file:///path/to/spark-monitor-plugin/target/scala-2.12/apache-toree-spark-monitor-plugin-assembly-<VERSION>.jar [other-options]The kernel installation route does the same thing through the install command, with a kernel name of your choosing and a toree_opts string carrying the magic URL:
jupyter toree install --spark_home=<YOUR_SPARK_PATH> --kernel_name=toree_with_monitor --toree_opts="--magic-url file:///path/to/apache-toree-spark-monitor-plugin-assembly-<VERSION>.jar"The third route is a configuration file, passed with the --profile option:
{
"magic_urls": ["file:///path/to/apache-toree-spark-monitor-plugin-assembly-<VERSION>.jar"]
}and then start the kernel with java -jar apache-toree-assembly-<VERSION>.jar --profile config.json.
The explanation of why this is necessary is the most useful sentence in the section. The plugin is not auto-loaded as an internal plugin because run.sh never puts it on the kernel's classpath, only the main assembly jar is passed to spark-submit, and PluginManager only scans the classpath it is given for Plugin subclasses. So the plugin jar's filename is not what matters; specifying it explicitly through --magic-url is what makes it visible to the kernel. The path must be absolute and the jar must be reachable from where Toree runs.
An incubating Apache project with a mirror and no tagged releases
The repository is the mirror of Apache Toree (Incubating), the incubator repository layout, with an.asf.yaml file and the standard Apache DISCLAIMER, NOTICE and LICENSE files. The licence is Apache-2.0, and the source files carry the standard ASF header, which is visible in the Dockerfile and the Makefile at the top of the file. That is the ordinary Apache position: permissive, with the patent grant and the requirement to preserve notices.
The version scheme is unusual in a revealing way. The Makefile derives VERSION from BASE_VERSION by appending -incubating, so a build is 0.6.0-incubating from a base of 0.6.0.dev0, and it decides whether to mark the build as a snapshot by looking for the string dev in the version. The base is a development version, not a release, and the -incubating suffix is stamped onto every artifact name.
There are no GitHub releases for this repository, and the presence of RELEASE_NOTES.md in the tree is where the release history lives instead.
The last push to the repository was on 2026-10-19, so the mirror is current with the project it mirrors.
Where Toree is the wrong tool
The first limit is language. Toree supports Scala, and the documentation repeats this in more than one place. A data team whose notebooks are Python and SQL will get nothing from it, and the Spark SQL interpreter that exists as a module in the tree is not the same thing as a first-class SQL kernel for general use.
The second limit is that this is a development tool, not a deployment target. It is a notebook kernel for interactive exploration against a cluster you already run. Nothing in the documentation describes Toree as a way to ship a Spark application, and building a release is a package, not a service.
The third limit is the operational weight of the plugin mechanism. The spark-monitor-plugin no longer ships inside the main assembly, and the documentation is clear that it will not load itself. Anyone upgrading who expected monitoring to be there will find it gone, and the fix is an absolute path and a magic URL, and the classpath reasoning means the same fix does not automatically work in every deployment shape, since it depends on how Toree was started.
The fourth is the build system. A make entry point that explicitly forbids using sbt directly is a deliberate choice, but it means a contributor or packager has to learn the project's own layer rather than the familiar sbt one, and the release path additionally requires Docker.
The alternative is the PySpark kernel that comes with Apache Spark itself, which is the obvious path for a Python notebook workflow, and a plain Spark shell or job submission for anything that does not need a notebook. The difference in approach is that Toree exists to give Scala code inside a notebook the same treatment PySpark gets in Python, and the price of that specialisation is the language lock-in and the plugin machinery above.
Editorial conclusion
Apache Toree is worth trying for a Scala team that lives in notebooks and wants to drive a real Spark cluster interactively, because it gives you a preconfigured context, understands the current Jupyter message protocol, and ships as a pip installable that registers itself as a kernel. It is the wrong tool for a Python-first data stack, for a production job, and for anyone who assumes the Spark Monitor plugin is bundled. Before you adopt it, check the three things the documentation is specific about: that you have a distribution of Apache Spark on the machine where the kernel will run, since the install command takes a spark_home path; that Scala is the only language the kernel supports, so a mixed-language workflow needs another kernel alongside it; and that the spark-monitor-plugin is no longer included in the main assembly and has to be pointed at explicitly with a --magic-url, because run.sh never puts it on the kernel classpath.
Frequently asked questions
How do I install Apache Toree as a Jupyter kernel?
You need a distribution of Apache Spark on the machine where the kernel runs, then pip install --upgrade toree and jupyter toree install --spark_home=<YOUR_SPARK_PATH>. Development snapshots are installed the same way with pip install pointed at the release URL.
How do I enable the Spark Monitor plugin in Toree?
You pass the plugin jar explicitly with a --magic-url option, either on the command line when starting the assembly jar, through the --toree_opts argument of jupyter toree install, or in a configuration file used with the --profile option. The plugin is not auto-loaded because run.sh never puts it on the kernel classpath.
Which programming language does Apache Toree support?
Scala only. The documentation states that Toree supports the Scala programming language and implements Jupyter message protocol 5.0.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/apache-incubator-toree)Was this useful?