beginner_de_project: a one-command Airflow batch pipeline you can read end to end

What the beginner_de_project pipeline is for

This repository is the code for a blog post called Data Engineering Project for Beginners, batch edition. That framing sets expectations correctly. It is a sample pipeline assembled to be read, not a framework to build on and not a service to run in production.

What it gives you is a complete path from a raw data drop to a rendered dashboard, with each stage in a piece of technology you are likely to meet in a job. Airflow schedules and orchestrates the DAGs, Postgres stores Airflow's own metadata and holds a schema representing the upstream databases, DuckDB acts as the warehouse, Apache Spark runs the processing including a classification algorithm, minio provides S3 compatible object storage, and Quarto with Plotly converts markdown into HTML you can embed in an app or serve as is.

The pipeline itself is a single DAG called user_analytics_dag, visible in the Airflow UI, and when it completes you get a rendered dashboard at ./dags/scripts/dashboard/dashboard.html.

The audience is a developer learning data engineering who wants a working example to take apart, and a technical lead who wants a starting point for a template project. The project also links a second post about data engineering projects with a free template, which covers CI/CD, infrastructure as code with terraform, make commands and automated testing.

Running the pipeline with one make up command

The local run is four commands, and they are meant to be pasted as a block:

git clone https://github.com/josephmachado/beginner_de_project.git
cd beginner_de_project 
make up
sleep 30 # wait for Airflow to start
make ci # run checks and tests

Note the sleep. The Airflow services take about thirty seconds to become ready, and the make up target includes a do-sleep step of the same length, so the explicit sleep in the README covers the remaining gap. Once it is up, the Airflow UI is on http://localhost:8080, and both the username and the password are airflow.

What make up actually does is visible in the Makefile. The up target depends on perms, docker-spin-up, do-sleep and setup-conn. The perms target creates the logs, plugins, temp, dags, tests, data and visualization directories and makes them group and world readable and writable, because Airflow runs as a different user inside the container than the one on your laptop. The docker-spin-up target builds the images and brings up airflow-init and then the rest detached. The setup-conn target runs python /opt/airflow/setup_conn.py inside the scheduler container, which is how the connection between the pipeline and its upstream database gets established.

There are also make down and make restart for stopping and recycling the stack, and a make sh target that opens a bash shell in the webserver container, which is the quickest way to poke at a failing task.

The six services and why several of them share one container

The docker-compose file is where the shape of the project becomes concrete.

Airflow runs with the LocalExecutor, so its SQL Alchemy connection points at a Postgres instance with the airflow:airflow credentials, the Fernet key is empty, DAGs are paused at creation, and example DAGs are not loaded. The API auth backend is the basic auth one, which is why the UI takes the airflow credentials. Six directories are bind mounted into the container: dags, data, visualization, logs, plugins and tests.

Postgres runs the postgres:16 image with a healthcheck built on pg_isready, and Airflow waits for that healthcheck before starting. The containers/upstream directory is mounted into the entrypoint init directory, so the upstream database schema and its data are created on first boot rather than by a separate setup step.

The simplification the documentation is upfront about is that services one to five of the list, which is to say everything except minio, are installed and run in a single container defined in containers/airflow/Dockerfile. That is a deliberate teaching choice. A production deployment would not put Airflow, DuckDB, Spark and Quarto in one image, but for a sample it means one build, one image and one place to look.

The docker-compose file also sets AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION to true, so the DAG does not fire on its own while you are still reading it. You unpause it yourself in the UI, which is a detail worth knowing before you conclude the pipeline is broken.

make ci runs isort, black, mypy, flake8 and pytest

The ci target is the part that makes this more than a demo. It is a chain of five steps:

ci: isort format type lint pytest

Each step runs inside the webserver container rather than on your host, so the tool versions are pinned by the image rather than by whatever you happen to have installed. isort sorts the imports, format runs black with the -S flag and a line length of 79, type runs mypy with --ignore-missing-imports against /opt/airflow, lint runs flake8 against the dags directory, and pytest runs the test suite with warnings disabled.

The line length of 79 is a small but real signal about the project's conventions, and so is the fact that lint only covers /opt/airflow/dags while the type check covers the whole /opt/airflow tree. Running make ci before make up would fail, since both require the containers to be running.

There is a separate .env and .env.spark in the tree, which is how the Spark job picks up its configuration, and the .dockerignore file keeps the build context small.

Running it on GitHub Codespaces instead of your laptop

The project documents a Codespaces path first, and it exists for a practical reason: the local path needs Docker with at least 4GB of RAM plus Docker Compose v1.27.0 or later, git and a GitHub account. Codespaces supplies the machine.

The steps are: create a codespace from the repository, either by cloning it or by using the Use this template button, then click Create codespaces on main. Wait for it to start, type make up in the terminal, and wait for that to complete. The documentation then says to wait another 30 seconds for Airflow to start, and after that to open the ports tab and click the link exposing port 8080 to reach the Airflow UI, with the airflow credentials.

The port forwarding detail is the part that trips people up. There is no browser on the codespace itself, so the exposed port link is how you get to the UI from your own machine.

There is a cost warning attached. The documentation asks you to switch off the codespace instance when you are done, because you have only limited free usage, and points at the GitHub codespaces pricing documentation. Running a container stack plus Spark plus a warehouse on a free tier is a reasonable thing to do once and a poor habit to keep.

Where this sample pipeline stops being useful

The first limit is structural and the project is honest about it. Airflow, DuckDB, Spark and Quarto share one container, so there is no isolation between the orchestrator and the compute. Nothing in the compose file is a production topology, and the credentials are the literal word airflow for both the Postgres user and the UI login. Fine for a laptop, wrong for anything reachable from a network.

The second limit is scope. There is one DAG, user_analytics_dag, and its output is a single HTML dashboard file. There is no incremental processing, no backfill strategy, no data quality gate and no alerting, so the interesting hard parts of batch data engineering are exactly the parts that are absent.

The third is that there is no data contract. The upstream database is created from files mounted into the Postgres init directory, and the classification algorithm in Spark is the one piece of domain logic in the project. If you want to adapt this to your own data, that Spark job and the schema in containers/upstream are where the work is.

Compare the alternative approach of running a managed warehouse with a managed orchestrator. The difference in approach is that this sample gives you every component in a form you can read and change, including the parts a hosted service would hide. What it does not give you is reliability, and it does not pretend to.

There is also a maintenance dimension. The repository is not archived, but the last push was on 2026-04-13, there are no GitHub releases, and the pinned container images will age. Anyone planning to keep this running needs to track the image tags in the compose file themselves.

Licence, and what the CI setup gives you for free

The project is released under the MIT licence, with the text in the LICENSE file at the repository root. The MIT terms permit use and redistribution, and require the copyright notice and licence text to travel with copies and substantial portions of the software, which for a template means keeping the LICENSE file if you fork it.

That permissive licence is a deliberate part of the offer. A template project that you are meant to fork, rename and put your own schema and DAGs into has to be easy to take, and the MIT terms do that with no reciprocity requirement. Read the LICENSE file yourself rather than relying on the summary, and keep the notice in your fork.

The CI story is the other thing worth taking away. The make ci target already wires together import sorting, formatting, type checking, linting and tests, and the documentation points to a separate post for setting up CI/CD and infrastructure as code. A pipeline that ships with its own quality gates and a Makefile of one-word targets is a better starting template than an empty directory with a tutorial.

The tree itself is small and legible. containers/ holds the Airflow Dockerfile and the upstream database seed, dags/ holds the DAG and its scripts including the dashboard, data/ and tests/ are bind mounted into the container, assets/ holds the images used in the documentation, and .github/ holds the workflow definitions. Nothing is hidden behind a framework.

Editorial conclusion

beginner_de_project fits a developer who wants to see a complete batch pipeline with Airflow, a warehouse, Spark and a dashboard running on their own laptop, and it is a good first repo to fork when you want to learn orchestration rather than build a platform. It does not fit production use, since the services share one container, the credentials are the literal word airflow, and the project has no GitHub releases and a last push on 2026-04-13. Before you base anything on it, run make ci and see what the pytest, mypy and flake8 checks actually cover, and read the LICENSE file, since the MIT terms require the notice to travel with any copy you redistribute.

Frequently asked questions

Official sources

1. Issues
2. josephmachado/beginner_de_project on GitHub
3. License: MIT
4. Project website
5. README