# TiSpark: running Spark SQL over the TiKV data that already holds your transactions > TiSpark is a thin Scala layer that lets Apache Spark read TiKV and TiFlash directly, pushing computation down into the storage layer and using TiDB statistics for planning, so that a deployment can serve transactions and OLAP from the same cluster without an ETL pipeline in between. It is also a project with a long list of specific limitations, a dependency you must now supply yourself, and a feature table that has fallen behind its own releases. **pingcap/tispark** — TiSpark is built for running Apache Spark on top of TiDB/TiKV - Repository: https://github.com/pingcap/tispark - Stars: 890 · Forks: 250 - Language: Scala - License: Apache-2.0 - Published: 2026-10-20 · Updated: 2026-10-20 · Language: en - Canonical page: https://hysenlabs.com/projects/pingcap-tispark ## The problem: one store for transactions and one for analysis The usual data architecture puts a transactional database on one side and an analytical warehouse on the other, with an ETL pipeline moving data between them. That pipeline is the expensive part: somebody maintains the jobs, the data is stale by however long the job takes, and the two systems drift apart. TiSpark is a thin layer built for running Apache Spark on top of TiDB, TiKV and TiFlash to answer complex OLAP queries. The argument is that it enjoys the merits of both the Spark platform and the distributed clusters of TiKV and TiFlash while staying integrated with TiDB. From the perspective of data integration, TiSpark plus TiDB performs both transaction and analysis directly on the same platform without building and maintaining any ETLs, which simplifies the system architecture and reduces the cost of maintenance. The audience is a team that has already chosen TiDB as its transactional store and wants to run analytical workloads on the same data without a second copy of it. The documentation is explicit about the prerequisites: TiSpark relies on the availability of TiKV clusters and PDs, and you also need to set up and use the Spark clustering platform. So this is not a Spark distribution, it is a connector layer, and it assumes both halves are already running. The other documented use is to deploy and use the tools from the Spark ecosystem for further data processing on TiDB, for example using TiSpark for data analysis and ETL, retrieving data from TiKV as a data source for machine learning, or generating reports from the scheduling system. The licence is Apache 2.0, the language is Scala, and the last push was on 2026-08-20. ## The mechanism: Catalyst, pushdown, and a client library doing the work The implementation is described as a thin layer, and the interesting engineering is in how much of Spark's work is pushed back into the storage layer. TiSpark integrates well with the Spark Catalyst Engine, and it provides precise control of computing, which allows Spark to read data from TiKV efficiently. It also supports index seek, which significantly improves the performance of point query execution. The second mechanism is pushdown. TiSpark uses several strategies to push down computing to reduce the size of the dataset handled by Spark SQL, which accelerates query execution, and it uses the TiDB built-in statistical information for the query plan optimization. So the plan is not built from Spark's own statistics over data Spark has not seen; it is built from the statistics the transactional store already maintains, and the work is done as close to the data as possible. That is the whole thesis of the project, and it is also where the limitations come from, because anything the storage layer cannot express in a pushdown has to be handled by Spark instead, on data that has been read over the network. Most of the logic sits inside a thin layer, namely the tikv-client library, and the repository layout confirms it: a tikv-client/ directory alongside the spark-wrapper/ that adapts it to Spark's expectations, plus core/ for the Spark integration, assembly/ for the packaged jar, and core-test/ and db-random-test/ for the test suites. ## Running the stack: PD, TiKV and TiDB under Compose The repository ships Docker Compose files so you can bring the storage side up locally, and they are the clearest description of the topology TiSpark expects. The placement service comes first, and it is the component that the rest of the system asks for membership information: ```yaml pd0: image: pingcap/pd:latest ports: - "2379:2379" ``` Then TiKV, which is the storage engine Spark actually reads from, and then TiDB, which is the SQL front end and the source of the statistics the planner uses: ```yaml tidb: image: pingcap/tidb:latest ports: - "4000:4000" - "10080:10080" ``` The dependency order is declared in the compose file, with tikv0 depending on pd0 and tidb depending on tikv0, and tidb is started with --store=tikv, which is the setting that puts the SQL layer on the distributed store rather than on a local engine. Configuration is mounted from a config/ directory, so the pd.toml, tikv.toml and tidb.toml files in the repository are what shape the local deployment. There are separate compose files for a TLS setup and for the 4.0 line, so the two variants are first class rather than an afterthought. Two things to notice. The images are tagged latest rather than pinned, so a local stack is whatever upstream published that day. And the ports are the usual TiDB ones, with 2379 for the client interface of PD, 20160 for TiKV and 4000 for the TiDB SQL port. ## The dependency you now have to supply yourself One section of the project documentation exists because of a licensing decision, and it changes what you have to do at install time. The mysql-connector-java dependency will not be provided, because of the limit of the GPL licence. The versions of TiSpark's jar that no longer include it are TiSpark above 3.0.1, TiSpark above 2.5.1 for the 2.5.x line, and TiSpark above 2.4.3 for the 2.4.x line. Now TiSpark needs mysql-connector-java for writing and authentication, so it has to be imported manually when you need to write or to authenticate. There are two ways to do that. You can put the jar into the Spark jars file, or you can pass it on the command line when you submit the job: ``` spark-submit --jars tispark-assembly-3.0_2.12-3.1.0-SNAPSHOT.jar,mysql-connector-java-8.0.29.jar ``` This is the correct behaviour for a project distributed under Apache 2.0 that links to a GPL library, and the way it is handled is worth noting as a practice: instead of quietly reimplementing the connector or vendoring it, the project documents the constraint and leaves the decision, and the licensing exposure, with the user. The practical consequence is that a deployment now has a moving part it did not have before. The connector jar has to be fetched, matched to the Spark version and placed correctly, and the assembly jar has to be referenced by the same mechanism, because the second limitation in the list says the tispark-assembly dependency should not be packaged into a jar of jars file, for example when built with the spring-boot-maven-plugin, or you get a ClassNotFoundException. ## The limitations list, which is the most useful page here The limitations section is longer than most projects would publish, and for an evaluator it is the most valuable part of the documentation. The first is views. TiDB starts to support view since tidb-3.0, but TiSpark currently does not support view, and users are not able to observe or access data through a view with TiSpark. That is a direct consequence of the design: a pushdown planner cannot push a view definition into the storage layer, so the view has to be resolved somewhere it is not. The second is a Spark configuration trap. spark.sql.runSQLOnFiles should not be set to false, or you may get an Error in query: Table or view not found error, which is a confusing symptom for a configuration mistake. The third is syntax. Using the style of {db}.{table}.{colname} in the condition is not supported, for example select * from t where db.t.col1 = 1, and Null in aggregation is not supported, for example select sum(null) from t group by col1. Then there are the collation and character set limits. TiSpark does not support the GBK character set, and it does not support the whole collations rule, currently supporting only utf8_bin, utf8_general_ci, utf8_unicode_ci, utf8mb4_bin, utf8mb4_general_ci and utf8mb4_unicode_ci. On top of that, if spark.sql.ansi.enabled is false, an overflow of sum(bigint) will not cause an error but will wrap the result, unless you cast bigint to decimal to avoid the overflow. The last one is a missed optimisation. TiSpark supports retrieving data from a table with an Expression Index, but the Expression Index will not be used by the planner of TiSpark. So the data is there, the index exists in TiDB, and the query plan ignores it. ## The feature table, and the releases it does not mention The feature support table compares TiSpark 2.4.x, 2.5.x, 3.0.x and master, and the shape of the results is instructive. SQL select without tidb_catalog is supported in 2.4.x and 2.5.x but not in 3.0.x or master. SQL select with tidb_catalog appears from 2.5.x onwards. SQL delete from with tidb_catalog appears from 3.0.x onwards, which makes sense because writes need the catalogue to be consistent. DataFrame append and DataFrame reads are supported across every column of the table. What is missing is as interesting as what is there. The table has no 3.1.x column, and the repository's own releases include v3.1.6 from 2024-07-26 and v3.1.7 from 2024-11-09, alongside v2.5.3 from 2024-11-18. So the table that is supposed to tell you which version supports what predates the versions you would actually install, and it also shows that the project maintains two lines, a 2.5.x line and a 3.x line, in parallel. The release dates are worth noticing for a different reason. The three most recent tags are from July and November 2024, while the last push was on 2026-08-20. So the tree is being worked on without new tagged releases, and the newest artefact you can pull is nearly two years old. The rest of the repository is conventional and well organised: a pom.xml at the root for the Maven build, a CHANGELOG.md, an OWNERS file, a .lift.toml, a Travis configuration, and a docs/ directory with the architecture diagram and the user guide, alongside the Dev Guide and benchmark pages in the wiki. ## Licence, maintenance and where TiSpark is the wrong tool The licence is Apache 2.0, stated in the project description and in the LICENSE file at the root, and the decision to leave the GPL connector out of the distribution is an example of that licence being taken seriously rather than ignored. The maintenance picture is the one to weigh. The repository is now archived on GitHub, so it takes no further changes, and the newest releases are from November 2024 and the feature matrix has not caught up with the version line. A project that you would deploy as a connector between your transactional store and your analytics engine benefits from a recent release, because it sits on the query path of both. Where TiSpark is the wrong tool is straightforward. If you do not already run TiDB with TiKV or TiFlash, there is nothing here for you, because the project is a layer for a specific storage engine and not a distributed SQL engine in its own right. If your workload is heavy write traffic or transactional work, TiDB is the system and TiSpark is the analysis path. And if your analytical queries need views, the full collation rules, expression indexes in the plan or GBK, the limitations section is telling you to use something else. The judgement, then, is conditional but clear. For a team whose transactional store is already TiDB and whose analytical queries fit the supported set, TiSpark removes the ETL pipeline and that is a large saving. For anyone else, the same project is a connector with a long list of gaps, and the gap list is the thing to read first. ## Conclusion Adopt TiSpark if you already run TiDB with TiKV or TiFlash and want to run Spark SQL directly against that storage instead of copying data into a separate warehouse, and if your queries fit the supported feature set. Do not adopt it if you need views, GBK, the full collation rules or expression indexes in the planner, and do not adopt it expecting a current release, because the newest tags in the repository are from late 2024. Before you start, read the limitations list, add the mysql connector jar to your Spark installation yourself, and check the feature table against the version you intend to run, since the table does not cover the 3.1.x line. ## FAQ ### What does TiSpark do and what does it require? It is a thin layer for running Apache Spark on top of TiDB, TiKV and TiFlash to answer complex OLAP queries, pushing computation down to the storage layer and using TiDB statistics for query plan optimization. It relies on the availability of TiKV clusters and PDs, and you also need a Spark clustering platform. ### Why do I need to install mysql-connector-java myself? The dependency is not provided because of the limit of the GPL licence, and the jars for TiSpark above 3.0.1, above 2.5.1 in the 2.5.x line and above 2.4.3 in the 2.4.x line no longer include it. You can put the jar into the Spark jars file or pass it with spark-submit --jars, since TiSpark needs it for writing and authentication. ### What are the main limitations of TiSpark? Views are not supported, the {db}.{table}.{colname} condition style and null in aggregation are not supported, the GBK character set and the full collations rule are not supported, spark.sql.runSQLOnFiles must not be set to false, and expression indexes are retrieved but not used by the planner. The tispark-assembly dependency also must not be packaged into a jar of jars file. ### How is TiSpark put together as a codebase? Most of the logic is inside a thin layer, the tikv-client library, with a spark-wrapper that adapts it to Spark, a core/ module for the Spark integration, assembly/ for the packaged jar and core-test/ and db-random-test/ for the test suites. The repository also ships Docker Compose files for a local PD, TiKV and TiDB topology. ### What licence is TiSpark released under, and how recent is it? Apache 2.0, with the licence text in the LICENSE file at the root of the repository. The last push was on 2026-08-20, but the most recent tagged releases are v2.5.3 from 2024-11-18, v3.1.7 from 2024-11-09 and v3.1.6 from 2024-07-26, and the feature support table does not yet cover the 3.1.x line. ## Sources - [Issues](https://github.com/pingcap/tispark/issues) - [License: Apache-2.0](https://github.com/pingcap/tispark/blob/master/LICENSE) - [pingcap/tispark on GitHub](https://github.com/pingcap/tispark) - [README](https://github.com/pingcap/tispark/blob/master/README.md) - [Releases](https://github.com/pingcap/tispark/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/pingcap-tispark