spark-as-service-using-embedded-server

Spark 2.1 wrapped as a REST service provider on an embedded, reactive-streams based, fully asynchronous HTTP server, with a handful of demo routes proving the shared SparkSession idea.

The application comes as a Spark 2.1 REST service provider running on an embedded HTTP server that is reactive-streams based and fully asynchronous. The README positions the core of the app as a REST service doing big-data cluster computation on Apache Spark, not a web application or a browser-facing tool. Scala is the language of the whole thing, and the server is embedded inside the application rather than bolted on.

The author explains the framework choice in one sentence: you normally do not build your application on top of Akka HTTP, you build it on whatever makes sense and let Akka HTTP handle only the HTTP integration. That made it the right fit for this use case, keeping the service logic independent of the web layer.

Four routes demonstrate the idea. A homepage at localhost:8001 returns hello world. The version route queries a shared SparkSession and reports the Spark version. activeStreams reports how many Spark streams are currently active. The count route runs a random Spark job that counts the elements in a sequence. Together they show that a single shared session can back several unrelated queries.

Optional command-line parameters cover things like spark-master and the akka port. There are two ways to change defaults: edit src/main/resources/application.conf and rebuild, or run mvn exec:java with exec.args to pass them directly. The README keeps the config story to those two paths, and it lists the configurable params if you want to inspect them. Either route leads to the same set of runtime options, so the choice is mostly about whether you prefer the checked-in file or the command line.

• Source 1 · Project repository
• Source 2 · README