dotnet/spark: C# and F# bindings for Apache Spark, and what they do not cover
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
At a glance
- What is it?
- Microsoft.Spark exposes Spark's DataFrame, SparkSQL and Structured Streaming APIs to .NET developers through a JVM bridge. It is a thin binding layer, not a reimplementation, and the boundary it draws is what determines whether it fits your workload.
- Who is it for?
- Adopt dotnet/spark if your team already writes C# or F# and your Spark work is DataFrame, SparkSQL or Structured Streaming against Spark 2.4 through 3.5. Do not adopt it if you need RDD-level control, MLlib, GraphX, or a Spark version outside that range, because the binding does not reach those surfaces.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 4 days ago.
- What is it written in?
- Mainly C#, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem dotnet/spark solves is a language boundary, not a performance one
Apache Spark is a JVM system. Its native APIs are Scala and Java, and PySpark exists because a large share of data practitioners write Python. .NET developers had no equivalent entry point, which meant a C# or F# team doing analytics work either ran a separate Python or Scala job, or kept the Spark portion of a pipeline in a language the rest of the codebase did not use. dotnet/spark, published as the Microsoft.Spark NuGet package, closes that gap by exposing Spark's DataFrame, SparkSQL and Structured Streaming surfaces to .NET code. The README frames the audience directly: developers who want to reuse "all the knowledge, skills, code, and libraries you already have as a .NET developer." That is the actual selling point. It is not a faster Spark, and nothing in the repository claims to be one. The libraries and tooling around a .NET service (dependency injection, configuration, existing domain types, the test framework the team already runs) stay usable, and the Spark job stops being a foreign artifact in the repository. If your team is not already invested in .NET, this project offers nothing that PySpark or Scala does not.
How the JVM bridge actually moves data between .NET and Spark
The mechanism is a bridge, not a port. Your .NET process does not execute Spark's query planner or its shuffle. It constructs a SparkSession and issues DataFrame and SQL operations that are serialized and handed to a JVM-side Spark process, which does the planning and execution. The .NET side is a client. This has a concrete consequence for anyone reasoning about where time goes: the expensive work happens in the JVM, and the .NET layer adds call overhead and serialization at the boundary. The README describes the APIs as "high performance," but the repository material does not include benchmark numbers, so treat that as a design intent rather than a measured claim. The architecture also explains the deployment requirement. Because executors run on remote nodes, the .NET runtime side has to be present there too, which is why the deployment documentation references a Microsoft.Spark.Worker component rather than a single NuGet reference. Any cluster you target has to be able to receive that worker. That is the first thing that breaks in practice, and it is not a code problem.
Supported Spark versions define a narrow compatibility band
The supported-versions table in the README maps release v2.3.1 of the .NET binding to Apache Spark 2.4, 3.0, 3.1, 3.2 and 3.5. Two details in that table matter more than the list itself. First, 2.4.2 is explicitly excluded, with a link to issue 60 as the reason. Second, the band is not continuous in the sense a reader might assume: 3.3 and 3.4 do not appear. If your managed Spark service has moved to a version outside this set, the binding is not certified against it, and the README gives no guidance on what happens if you try. The version of the .NET binding is independent of the Spark version it targets, which is easy to misread. Microsoft.Spark v2.3.1 is not "Spark 2.3.1"; it is the binding release that covers the Spark versions listed. Check both numbers before you plan an upgrade, because a cluster upgrade can move you off the supported band without any change to your application code.
Getting an app running: NuGet package, worker deployment, and the platform split
The README points to three getting-started guides, for Windows, Ubuntu and macOS, all built around .NET 8. The package to reference is Microsoft.Spark from NuGet. Building from source is documented separately for Windows (.NET Framework 4.8 and .NET 8) and Ubuntu (.NET 8), and the README states the whole process from cloning to running an app should take under 15 minutes. That estimate covers a local build, not a cluster deployment. The runtime story has a split worth noting: .NET 8 works on Windows, Linux and macOS, while .NET Framework is Windows only. Deployment targets named in the README are Azure HDInsight Spark, Amazon EMR Spark, and Databricks on both AWS and Azure. The repository also carries a deployment guide, and the getting-started documents are the place to look for the exact worker installation commands, since the README itself does not inline them. One structural point: the README notes an open Spark Project Improvement Proposal (SPARK-27006) to get .NET support into Apache Spark by default. Until that lands, this is an external binding you install and maintain yourself, not a component that ships with the distribution.
What the binding does not reach, and where it is the wrong tool
The README is specific about scope: the .NET APIs cover "the most popular Dataframe and SparkSQL aspects" plus Structured Streaming. That phrasing is a boundary. Work that lives below the DataFrame abstraction, including RDD-level manipulation, is not part of the advertised surface. Neither is MLlib, and neither is GraphX. If your pipeline needs a Spark component outside DataFrame, SQL and Structured Streaming, you are writing that component in Scala, Java or Python and calling across a process boundary, which erases most of the reason to use this binding. There is a second failure mode that is operational rather than API-shaped. The binding is a client to a JVM, so a version mismatch between the Microsoft.Spark package, the Microsoft.Spark.Worker on the executors, and the Spark runtime on the cluster produces failures that surface as connection or serialization errors rather than as clear compatibility messages. The supported-versions table is the only compatibility contract the repository offers, and it is a table, not a runtime check. Treat any deployment where you cannot pin all three versions as a risk you are accepting deliberately.
PySpark is the real alternative, and the difference is the bridge on both sides
PySpark occupies the same position for Python that dotnet/spark occupies for .NET: a language binding over a JVM engine, with the same client-server shape and the same worker-on-executors deployment requirement. The difference is not architectural, it is ecosystem depth. PySpark ships with Apache Spark, so its version tracks the engine rather than lagging it, and the Python data ecosystem (the notebook tooling, the plotting and dataframe libraries, the published examples) assumes it. dotnet/spark is an external project with its own release cadence, its own supported-version table, and a pending SPIP before it becomes a default part of Spark. Choosing between them is therefore a question about your team, not about the technology: if the surrounding codebase, the CI pipeline and the people maintaining the job are .NET, dotnet/spark keeps that coherent, and the cost is the compatibility band and the worker deployment. If the job is standalone analytics work with no .NET context, PySpark has fewer moving parts and a shorter path from a documentation example to a running job. There is no version of this comparison where dotnet/spark wins on ecosystem breadth.
Maintenance cost, release cadence and the MIT licence
The release history shows a slow cadence. v2.3.0 shipped in May 2025, v2.3.1-rc1 in February 2026, and v2.3.1 in February 2026. That is roughly one minor release a year, with a release candidate preceding the stable tag. For a binding that must track Spark's own release train, a slow cadence is the maintenance cost you are signing up for: when your cluster moves to a Spark version outside the supported table, you wait. Budget for the possibility of pinning your cluster to a supported Spark version rather than upgrading freely. The licence is MIT, which is permissive and imposes no copyleft obligation on your application code. Nothing in the repository material suggests any additional licensing condition on the Microsoft.Spark package or the worker. This is a description of what the licence identifier means, not legal advice; if your organization has a licence review process, run the MIT text and the NuGet package metadata through it rather than relying on this summary. The repository is not archived and the last push date is recent, so the project is active, but active is not the same as fast, and the supported-version table is the number that will actually constrain your upgrade schedule.
Editorial conclusion
Adopt dotnet/spark if your team already writes C# or F# and your Spark work is DataFrame, SparkSQL or Structured Streaming against Spark 2.4 through 3.5. Do not adopt it if you need RDD-level control, MLlib, GraphX, or a Spark version outside that range, because the binding does not reach those surfaces. Before committing, verify three things: that your cluster's Spark version appears in the supported table, that you can deploy the Microsoft.Spark.Worker to every executor node, and that your target API exists in the .NET binding rather than only in Scala or PySpark.
Community notes