Spark NLP: Distributed NLP with Pretrained Pipelines, but Check the Spark Version Fit
State of the Art Natural Language Processing
At a glance
- What is it?
- Spark NLP brings BERT, Llama, and hundreds of other models to Apache Spark pipelines. This review covers how it works, how to run it, and where the version matrix and experimental support may trip you up.
- Who is it for?
- Adopt Spark NLP if you already run Apache Spark and need NLP at scale with pretrained models in Python, Scala, or Java. Skip it if you have a single-machine workload with no distributed need, or if you cannot match your Spark version to the correct spark-nlp package.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Scala, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- JohnSnowLabs/spark-nlp4,163 ★ · +5
| Repository | 2026-09-10 | 2026-11-15 |
|---|---|---|
| JohnSnowLabs/spark-nlp | 4.2k | 4.2k |
5 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
What Spark NLP Actually Solves
Spark NLP provides annotators and pretrained pipelines that integrate with Apache Spark. Teams can use them with DataFrames for tasks such as tokenization, embeddings and named entity recognition. Spark can run locally or on a cluster; whether it is worthwhile depends on data size, model, infrastructure and latency requirements. Review the model catalog and the terms of each model separately.
How the Annotator Pipeline Mechanism Works
The core architectural unit is the annotator. Each annotator performs one NLP task, such as tokenization or part-of-speech tagging, and outputs a column of annotations. You chain annotators into a pipeline, and Spark NLP runs that pipeline as a series of DataFrame transformations. The README's quick start shows a PretrainedPipeline called 'explain_document_dl' that, when run, returns a dictionary with keys like 'entities', 'stem', 'pos', and 'ner'. Under the hood, that pipeline is a pre-assembled sequence of annotators: document assembly, sentence detection, tokenization, embeddings, and a named entity recognizer. The pipeline downloads the model artifacts on first use, caches them, and then applies them to each row in the DataFrame. This design means you get the same API whether you run on one machine or a cluster. The distributed execution is inherited from Spark itself, so Spark NLP does not reimplement parallelism. It also supports importing models from TensorFlow, ONNX, OpenVINO, and llama.cpp GGUF, which means you are not locked into a single training framework. Supported model architectures and import paths vary, so check the specific model and runtime documentation before assuming any weights in those formats will work.
Getting It Running: Commands and Version Matching
For current installations, the project documents Spark NLP 7.0.0 with Apache Spark 3.0–3.5 and selected Spark 4 releases. Match the Maven artifact and Scala version to your runtime: Spark 3 uses Scala 2.12, Spark 4.0.0 has a dedicated Scala 2.13 artifact, and Spark 4.0.1 and the documented 4.1 releases use the standard Scala 2.13 artifact. The current quick start uses Java 8 or 11 for Spark 3, Java 17 for Spark 4, and Python 3.9 in its conda example. Check the project’s compatibility matrix before installing.
The Python example loads a pretrained pipeline with PretrainedPipeline and calls annotate() on text. The project also documents sparknlp.start() options for GPU and experimental AArch64 or Apple Silicon builds. Verify the selected package and model against the runtime you intend to deploy.
Where It Falls Down: Version Matrix and Experimental Platforms
Version compatibility needs attention. Spark NLP 7.0.0 supports Spark 3.0–3.5 and selected Spark 4 releases, with different Scala artifacts for Spark 3, Spark 4.0.0 and later validated Spark 4 versions. Choose the artifact, Java and Python versions that match your Spark distribution using the project’s current matrix.
The README marks Apple Silicon and AArch64 support as experimental, so test those deployments carefully. Pretrained pipelines may need substantial model downloads; the project documents offline use when the library and model files are provisioned in advance. For a small single-node workload, starting a SparkSession may add more overhead than a lighter local NLP library.
The Real Alternative: Direct Model Libraries
Direct model libraries such as Transformers or spaCy can be useful when a workload does not otherwise need Spark. They can also be deployed in distributed systems; their performance is not inherently single-process. Compare throughput, latency, memory, model compatibility and operational cost on the same data and hardware before choosing.
Maintenance, Upgrades, and Licensing
The repository is Apache-2.0 licensed and publishes packages through multiple channels. Model and service terms can differ, so check each artifact you use. Before upgrading, review the current compatibility matrix and release notes, pin the versions, and test pipelines and custom model imports in the target environment.
Editorial conclusion
Adopt Spark NLP if you already run Apache Spark and need NLP at scale with pretrained models in Python, Scala, or Java. Skip it if you have a single-machine workload with no distributed need, or if you cannot match your Spark version to the correct spark-nlp package. Before committing, verify your Spark and Scala versions against the current compatibility matrix and choose the matching artifact; use the Java version required by your Spark runtime, and treat Apple Silicon and AArch64 support as experimental, not production-ready.
Frequently asked questions
what is spark nlp
Spark NLP is an Apache-2.0 natural language processing library built on Apache Spark, returning annotations for machine learning pipelines. It covers tokenization, segmentation, tagging, embeddings, named entity recognition, dependency parsing, spell checking, classification, sentiment, translation, summarization, question answering, text generation, image and speech tasks and zero-shot learning, and it ships for the JVM as well as for Python on PyPI and Anaconda.
is spark nlp free
The Spark NLP repository is Apache-2.0 licensed. Check the individual model, artifact and service terms separately; the open-source code license does not establish that every related offering is free.
What are the alternatives to Spark NLP named in the repository?
None. The readme does not name competing libraries. What it does list is import support for models from four frameworks, TensorFlow, ONNX, OpenVINO and Llama.cpp in GGUF format, which is a different kind of interoperability claim: bringing your own weights rather than replacing the library.
Is Spark NLP still written in Scala?
This repository's primary language is Scala, and it publishes Maven artifacts for both Scala 2.12 and 2.13, with the binary version appended to the artifact name. It also ships a Python package on PyPI and Anaconda Cloud under the name spark-nlp, so the Scala build is the origin rather than the only way to use it.
How do I install Spark NLP for Apache Spark 4.0.0?
On Spark 4.0.0 the Maven artifact is `spark-nlp-spark400_2.13`, while on Spark 4.0.1 and the later validated versions it is `spark-nlp_2.13`, and on Spark 3.x it is `spark-nlp_2.12`. For Python the quick start pins `pip install spark-nlp==7.0.0 pyspark==4.0.0`, and Java 17 is required for Spark 4.x.
Does Spark NLP support Apple Silicon and ARM servers?
There are dedicated artifacts for both: a silicon build for Apple Silicon and an AArch64 build for Linux, each with a Spark 400 variant for Spark 4.0.0, and the start function takes an Apple Silicon flag and an AArch64 flag. The readme notes that M1 and M2 and AArch64 are under experimental support.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/johnsnowlabs-spark-nlp)Star history chart for your README
[](https://hysenlabs.com/projects/johnsnowlabs-spark-nlp)Was this useful?