nba-monte-carlo: a single-node Modern Data Stack with an NBA season simulator

What the repository actually is: a stack, with a simulation bolted on

The project description calls it a Monte Carlo simulation of the NBA season, leveraging dbt, duckdb and evidence.dev, but the readme is clearer about what the repository is for. It serves as an end to end example of running the Modern Data Stack on a single node, and the components are designed to be hot swappable, using a Makefile to create clearly defined interfaces between discrete components in the stack.

That framing matters. The NBA season simulation is the payload that gives the stack something to do, and the readme also notes the data comes from pro football reference, sports reference for college football, basketball reference and draft kings. So the repository is a teaching artefact whose subject is the data stack, and the simulation is the vehicle.

The readme also points at the current progress, which is Serverless BI. The latest version of the project is available at mdsinabox.com, and the website embraces the notion of Serverless BI, with the pages built asynchronously with open source software on commodity hardware and then pushed to a static site. The GitHub action that automatically deploys the site upon a pull request is named deploy_on_netlify.yml.

So there are two deliveries here: the live application, which is a static site, and the source that produces it. The transformation documentation is self hosted on GitHub pages, which is the piece a reader should treat as the actual documentation, since the readme mostly points at it.

Why DuckDB and Parquet instead of Postgres

The Notes on Design Choices section is short and argues its positions directly.

On the compute engine, using DuckDB keeps install and config very simple because it is a single command and runs everywhere. The readme is also candid that DuckDB frankly covers for the sin of building a Monte Carlo simulation in SQL, and that it would be quite slow without the kind of computing that DuckDB can do. That is an honest admission that the query shape is the bottleneck, and it is the reason the choice is defensible rather than accidental.

Postgres was also considered, and the reason it was rejected is specific: it is not a great pattern to run Postgres on the same node as the rest of the data stack. On a single-node example, adding a database server changes the character of the exercise.

On storage, the project leverages Parquet in addition to the DuckDB database for file storage. The readme flags this as experimental, and says the implementation will evolve over time, especially as both the DuckDB format continues to evolve and Iceberg and Delta support is added to DuckDB. Parquet is the durable artefact and DuckDB is the query engine over it, and the readme is candid that this boundary is still moving.

The third note is about external tables. dbt-duckdb supports external tables, which are Parquet files exported to the data_catalog folder, and that allows easier integration with tools like Rill, which can read the Parquet files and transform them directly with its own DuckDB implementation. The Makefile creates that folder structure, with raw, prep, simulator and analysis subdirectories, which tells you the pipeline stages map onto it.

The Makefile is the spine: build, run, dev, serve

Everything in this repository is driven by a Makefile, and reading it tells you the whole pipeline.

The build target creates the Python environment, installs the dependencies, installs the dbt packages, installs the Evidence npm packages, and creates the four data directories:

build:
	uv venv --clear && uv sync
	cd transform && ../.venv/bin/dbt deps
	cd evidence && npm install
	mkdir -p data/data_catalog/raw

The run target then executes the two ingestion pipelines, builds the dbt models, and loads the Evidence sources:

run:
	cd dlt && ../.venv/bin/python3 nba_pipeline.py
	cd dlt && ../.venv/bin/python3 nfl_pipeline.py
	cd transform && ../.venv/bin/dbt build
	cd evidence && npm run sources

The order is the interesting part. Ingestion runs first and writes into data/data_catalog/raw, then dbt transforms that into the other directories, and only then does Evidence pull its sources. The two pipeline scripts are named for the sport, nba_pipeline.py and nfl_pipeline.py, matching the two sport reference sites in the source data list.

Two more targets complete the workflow. dev serves the Evidence app in development mode with the host set to 0.0.0.0, and serve removes the previous build, rebuilds the Evidence app, installs http-server globally and serves the build directory.

The Makefile also contains a dbt-run-backfill target that loops over a generated list of dates, running a dbt build tagged nba with a variable for the start date, and the date range is computed inline in the Makefile with a Python one-liner. The dbt docs are self hosted on GitHub pages under the #!/overview route.

The serving layer: Malloy models and an Evidence dashboard

The readme calls the Evidence.dev layer a beautiful serving layer, with three screenshots of the dashboard, and the project can be explored live at mdsinabox.com.

The choice of Evidence over a conventional BI tool is what makes this serverless BI claim work. Evidence is a markdown-based BI tool: the queries live in files in the repository, the pages are built ahead of time, and the result is a static site. There is no database connection to maintain at read time and no server to keep alive, which is exactly the property the readme is pointing at when it says the pages are built asynchronously with open source software on commodity hardware and then pushed to a static site.

The repository structure supports that reading. There is a malloy/ directory for the Malloy models, an evidence/ directory for the dashboard project with its own npm lifecycle, a sqlmesh/ directory for the data transformation framework that sits alongside dbt, a dlt/ directory holding the ingestion pipelines, a transform/ directory for the dbt project, a data/ directory for the catalog, a utils/ directory and a docs/ directory.

The presence of both dbt and sqlmesh is notable. The readme does not explain the division of labour between them in the visible text, and the to-do list contains an item to clean up environment variables and implement incremental builds, so the arrangement is still settling. The pyproject pins sqlmesh at 0.88 and dbt-core at 1.9.4, alongside dbt-duckdb, dlt with the filesystem extra, duckdb around 1.3.0, numpy, pandas, pyarrow, polars, shandy-sqlfmt with the jinjafmt extra, and the project requires Python 3.11 or above.

Getting it running: Codespaces, WSL or Docker

The Getting Started section offers three routes, and the first is the least work.

The recommended path is a GitHub Codespace. You create one and run a single command:

make build run

You then wait for the pipeline and the Evidence configuration to complete, and the readme advises a 4-core Codespace because it performs significantly better in testing. Once the build finishes you open the Evidence dashboard from the Ports tab and log in with the username and password admin and password. Codespaces also supports Docker-in-Docker, so the container route is available inside the Codespace with:

make docker-build docker-run-evidence

The Windows route is spelled out in five steps. You install WSL from an administrator PowerShell with wsl --install, restart if it was the first install, open Ubuntu and update packages with sudo apt-get update, install python3.9 with pip and venv via sudo apt-get install python3.9 python3-pip python3.9-venv, then clone the repository into a my_projects directory and run make build run. The Evidence default location is 127.0.0.1:8088, with the same admin and password credentials.

The Docker route builds an image from the Dockerfile and runs it. The image is based on nikolaik/python-nodejs with python3.11 and nodejs20 on bullseye, installs the build toolchain including gcc, g++, make, python3-venv, gfortran and musl-dev, downloads the DuckDB command line version 1.2.2 for linux-amd64, installs uv from the astral.sh installer, copies the data, dlt, transform, evidence and pyproject files in, and runs make build as a build step.

One inconsistency is worth knowing about before you follow the Docker instructions. The README's docker-run-evidence example publishes port 8088, while the Makefile's own docker-run-evidence and docker-dev targets publish 3000. Since the Makefile is what the alias runs, expect 3000 from the alias and 8088 only if you copy the README snippet literally.

What this is good for, and where its limits are

There is a real limit to what a repository like this can be, and it is worth being clear about it.

The pipeline's inputs are web pages. The data comes from pro football reference, sports reference for college football, basketball reference and draft kings, and every one of those sites can change its HTML, change its rate limiting, or disappear. A Monte Carlo simulation is deterministic given its inputs, but the inputs are scraped, and the scraping is the fragile part. The readme does not discuss failure handling for a source that stops returning what the pipeline expects, and the to-do list still contains the incremental build work that would reduce how often the whole pipeline has to re-run.

The second limit is that it is an example, not a product. The readme says the components are designed to be hot swappable and that it runs in many environments with many visualization options, but a swap in this design means changing the interface between two stages that the Makefile has hardcoded, and the Makefile is doing the orchestration. The most transferable idea in the repository is not any particular tool but the arrangement: ingestion into a raw catalog, transformation into modelled layers, a serving layer that reads files rather than a live database, and a static site at the end.

The third limit is the scale of the exercise. This is one machine, DuckDB in place of a warehouse, and a Makefile instead of an orchestrator, which is exactly what makes it runnable in a Codespace. It is a good demonstration of the shape of a Modern Data Stack, and it is not a template for running one in production.

Editorial conclusion

This repository fits a data engineer who wants one runnable example of dbt, DuckDB, dlt, Malloy and Evidence.dev composed into a real pipeline, and it does not fit an analyst who only wants the season simulation. Before building it, note that the Makefile expects a Node toolchain alongside Python, that the dbt build is driven by a data directory that the ingestion step populates first, and that the repository's data comes from public reference sites, so the pipeline's inputs are only as stable as those pages.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. matsonj/nba-monte-carlo on GitHub
4. Project website
5. README