Dataform Core: SQL with dependencies, tests and docs for BigQuery

What the meta-language adds to plain SQL

Dataform Core is described as an open source meta-language for creating SQL tables and workflows in BigQuery, and the value it adds over writing SQL by hand is in three areas. It provides a dependency management system, so a table knows what it must be built after and what it depends on, and the graph is checked rather than discovered by a scheduler at run time. It provides automated data quality testing, in the form of assertions attached to a model that are evaluated as part of the run. And it provides data documentation, generated from the same definitions the SQL is generated from, so the reference does not drift away from the pipeline. The framing in the README is software engineering practice applied to data, with version control and testing named explicitly, and that is the pitch: a data team gets the same workflow a code team has.

Two ways to run it, and how they differ

There is a managed path and a local path, and they are not equivalent. In Google Cloud Platform, Dataform gives a fully managed experience for building BigQuery transformations in SQL, with a cloud development environment where the assets and the version control live together, support for GitHub, GitLab and other Git providers, and a serverless orchestration environment fully integrated into the platform. The local path is the command line tool, installed globally with npm:

npm i -g @dataform/cli

After that the CLI guide explains the workflow and the CLI reference in the repository lists every command and flag. The difference in approach between the two is about who runs the pipeline. Managed means Google owns the scheduling, the retries and the state, and the cost is a dependency on that platform. Local means the same open source core runs on a laptop or in your own runner, which is the more portable option and the one the repository is actually built around.

The security note is the paragraph to read first

The README carries an explicit warning about project execution and security, and it deserves more attention than it usually gets. Dataform compiles SQLX and JavaScript project code using Node's native node:vm context, and while module resolution is scoped to the project directory, node:vm is not a security sandbox against untrusted code. The projects execute with the privileges of the local process, and the instruction is to run and compile Dataform projects only from trusted sources. The practical meaning is that a Dataform project is a program, not a configuration file. When you pull a pipeline from somebody else, or when a colleague adds a package, you are running code with your own credentials, and the module scoping is a convenience for resolution rather than a boundary. A team that treats pipeline definitions as reviewable text has to extend that review to the code inside them, and the same care applies to a pre-defined package from the public package registry.

Dependencies, assertions, scripting and packages

The reference material in the README maps onto the three features plus a scripting layer. There is documentation for creating tables and views, a page on configuring dependencies that explains how a model declares what it needs, and a page on writing data quality checks, which is where the assertions live. Scripting is enabled separately, and it is where the JavaScript API comes in, covering workflow development and code reuse, so a definition can share logic with others instead of copying SQL. On top of that sits a package mechanism, with pre-defined packages published for import and the option to create your own, which is the equivalent of a shared library of model fragments. The configuration reference covers the project settings that sit around all of this, and the whole set of documents is also readable on the project site, which is where most people will start.

The repository is a real piece of engineering

The tree shows what a mature project looks like. The build is driven by Bazel, with a build file, a module definition and a lock file, a pinned Bazel version, and a version file that carries the release number, and there are separate Cloud Build configurations for testing, for publishing and for versioning the release. The JavaScript side is a pnpm workspace with a lock file, Rollup for bundling and a strict ESLint setup that includes custom rules written in the repository, with Prettier for formatting and typedoc for the API documentation. There is a protos directory for the protocol definitions used by the editor integration, a vscode directory for the language server and the extension, a testing directory and a test credentials directory, and an examples directory with working programs such as an extreme weather example and a Stack Overflow reporter. The CLI itself lives in its own subdirectory with its own reference documentation, and there are shared common and tools directories used by both.

Where Dataform does not fit

The binding constraint is BigQuery. The core generates SQL for that engine, the dependency library is a Google Cloud BigQuery client, and the syntax helper in the dependency list is a Google SQL parser, so a team running Snowflake, Databricks or Postgres is looking at the wrong tool. The second constraint is the authoring model. Dataform projects mix SQL and JavaScript, and the SQL that runs is generated, which some reviewers dislike because the thing you read in the repository is not the thing that executes, and debugging a generated query means reading generated output. The third is the local execution story. The CLI is installed globally with npm, it compiles with node:vm as described, and there is no sandbox to fall back on, so the workflow assumes the machine you compile on is a machine you trust.

Dataform Core next to dbt

The honest comparison is with dbt, because both solve the same problem in the same shape. Dataform Core wins on the other side of the ledger. It is open source with an Apache licence, it is the same code that runs the managed Google Cloud service rather than a separate implementation, it integrates directly with the BigQuery client, the lineage and storage libraries, and with the editor tooling, and the managed path removes the orchestration work entirely. A Google Cloud team that wants no vendor to run its scheduling picks Dataform; a multi-platform data organisation picks dbt.

Licence, releases and where the code and the product diverge

The core, the CLI and the tooling are released under the Apache 2.0 licence, with the terms in the licence file at the repository root. The releases are frequent and the version numbers show a mature project in active development, 3.1.0 on 2026-10-15, 3.0.71 on 2026-09-25 and 3.0.70 on 2026-09-17, with the last push on 2026-10-19. One administrative detail matters for anyone filing issues. The README splits the reporting paths, because a bug in the core or the open source parts belongs on the GitHub issue tracker, while a bug in the managed Google Cloud Platform service goes to the Google issue tracker with a different template for bugs and another for feature requests. That split reflects the arrangement of the project itself, where the open source core and the hosted product are related but separately owned, and it tells you where a change to the managed behaviour will actually land.

Editorial conclusion

Dataform Core fits a Google Cloud team that wants its BigQuery transformations version controlled, tested and scheduled without running its own orchestration, and it fits a team that wants to leave the orchestration to Google while keeping the pipeline definitions in Git. It does not fit a multi-warehouse organisation, because the SQL it generates targets BigQuery, and it does not fit a team unwilling to accept JavaScript as the authoring language for its models. Before you compile a project you did not write, read the security note in the README, because compilation runs project code with the privileges of your own process, and the same review you would apply to any dependency applies to a pipeline definition.

Frequently asked questions

Official sources

1. dataform-co/dataform on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases