practical-data-engineering: one real-estate pipeline, seven tools

Seven tools, one pipeline

This repository contains a practical implementation of a data engineering project that spans web scraping of real estate listings, processing with Spark and Delta Lake, data science in Jupyter Notebooks, ingestion into Apache Druid, visualisation with Apache Superset, and workflow management with Dagster, all orchestrated on Kubernetes.

That sentence is a stack list, and the interesting part is what sits between the entries. A scraper produces rows. Change data capture turns those rows into a stream of changes rather than a nightly snapshot, which is the difference between a pipeline that loses a day of edits and one that does not. An S3 gateway gives the pipeline somewhere to put the raw data without caring which cloud it is. A table format with upserts and transactions gives the project somewhere to correct mistakes. A columnar database gives the dashboards something fast to query. A dashboard tool gives people a picture. A scheduler gives all of it a clock.

The domain is what makes it a project rather than a diagram. The application collects real-estate data, enriches it with various metrics, and offers insights through machine learning and data visualisation, and the stated goal is to help you find your dream properties in your area. A property dataset is a good choice for a teaching project because the shape of the data is familiar, the joins are obvious, and there is a question you could actually want answered.

The primary language of the repository is recorded as Jupyter Notebook, and the top-level listing is small, a src/ directory, images/ and the usual repository files, which tells you the work sits in notebooks and pipeline code rather than in a packaged library. The last push was on 2026-06-25 and the project publishes no releases.

What the project is trying to teach

The About section states the intent plainly: the project addresses common data engineering challenges while exploring innovative technologies, it should serve as a learning project but incorporate real-world use cases, and it is a guide to building a data application that collects data, enriches it with metrics and offers insights through machine learning and data visualisation.

The three reasons given for the project are worth reading because they describe the shape of the deliverable. A real-world application, tackling a genuine problem with real estate data to find the best properties. A wide tech stack, from web scraping and S3 storage through data processing and machine learning to visualisation and orchestration. And a hands-on learning approach, offering a way of understanding how different technologies integrate and complement each other in a real scenario.

The last one is the point of the whole thing, and it shows in how the repository is organised. The key features are listed as learnings rather than as capabilities: scraping with Beautiful Soup, change data capture mechanisms for efficient data updates, MinIO as an S3 gateway for cloud-agnostic storage, upserts and ACID transactions with Delta Lake, notebooks for data science tasks, Superset for visualisation, Dagster for orchestration and Kubernetes for a cloud-agnostic deployment.

What the project is not is a framework. There is no API to learn and no abstraction to adopt, and the README invites you to build your own version on it, with a note asking people who built their own project or forked this one to say so in the comments. The repository is a worked example you are meant to take apart.

The change data capture layer, and why it is the interesting part

Of the seven technologies, the one that is worth the most attention to a reader is the change data capture mechanism, because it is the piece that a naive project skips and that a production one cannot do without.

A scraper that runs on a schedule and writes the current state of a website gives you a table whose rows change underneath you. If a listing is edited between two runs, the edit is lost, and if a listing disappears, you cannot tell whether it was sold or the site changed. Change data capture is the pattern where the pipeline consumes a log of what changed rather than a picture of what exists now, so that updates, removals and edits all arrive as events and the downstream state is derived.

The rest of the stack supports that idea. MinIO gives a place to keep both the raw scraped payloads and the change log, in a format that is the same whether the data is local or in the cloud, which is what the README means by cloud-agnostic storage. Delta Lake then provides the upserts and the ACID transactions, so that applying a change log to the current state is a transaction rather than a delete-and-reinsert.

The notebook layer sits on top of that, and it is where the enrichment and the modelling happen, with the data science work and the machine learning both living in notebooks rather than in a scheduled job. The output of that work is what Druid ingests for the dashboards.

So the sequence is scrape, capture changes, store, apply changes transactionally, explore, model, serve to the dashboard, and schedule the whole thing. Each step has a reason to exist that is visible in the list.

The March 2024 rebuild, and the removal of Spark

The project has been rebuilt once, and the reason is the most useful thing in the README.

It started in November 2020 as a project for the author to learn and teach data engineering, and the entire project was published in March 2021, with the initial version kept on a branch called v1. The author then notes, three years later, that the tools used in the project are still used today, and that people say how fast the modern data stack changes, but if you choose wisely you see that good tools will stay the time.

In March 2024 the project was updated to the latest Dagster and representative tools versions. Most technologies were kept, with one exception: Apache Spark. The reason is given plainly, it was a nightmare to set up locally and to work with the Delta Lake SQL API, and it was replaced with delta-rs, which is implemented in Rust and can edit and write Delta Tables directly in Python.

That swap is instructive because it is a trade rather than a preference. delta-rs gives you direct, transactional access to Delta tables from Python without a Spark session, which removes a heavy dependency from a laptop, at the price of the distributed engine that a larger workload would eventually need. For a teaching project that runs on one machine, the laptop experience is the thing that decides whether anybody finishes the exercise.

The README also carries the author's next moves, which are honest about the fact that the stack keeps moving. Rill Developer might be added to analyse the data powered by DuckDB, and for a more production-ready dashboard Superset would still be the choice.

The last push was on 2026-06-25, so the tree is being touched, but the documentation and the walkthrough are the March 2024 versions.

Running it: MinIO first, then Dagster

The installation section points at the individual component directories for the detailed setup, and the quick start is four steps: clone the repository, install the dependencies, install and start MinIO, and then explore the data with the provided notebooks and Superset dashboards.

The commands live in the pipeline directory, and the order matters because the storage has to exist before the pipeline can write to it:

#change to the pipeline directory
cd src/pipelines/real-estate

# installation
pip install -e ".[dev]"

# run minio
minio server /tmp/minio/

# create the bucket `real-estate` MinIO on 127.0.0.1:9000
# Create access key/passwords, defaults are MINIO_ROOT_USER/MINIO_ROOT_PASSWORD with default values that should work without any further configuration.

# startup dagster
dagster dev

Two details are worth flagging. The bucket has to be created by hand, and the credentials have defaults, which is convenient for a local run and a bad idea anywhere else. And the last command, dagster dev, starts the scheduler's development interface, which is where the pipeline is assembled and run.

There is a disclaimer attached to the quick start, and it is the right one to read before anything else. The project is designed for educational purposes, demonstrating web scraping and data engineering practices, and you are asked to ensure you do not violate any website's copyright or terms of service, and to approach scraping responsibly and respectfully. The first stage of this pipeline fetches somebody else's web pages, and that is the part of the project where the ethics live rather than in the tooling.

What you need, and where the detail lives

The prerequisites are short, and the comments on them are the useful part.

Python and pip are needed for installing the dependencies. MinIO has to be running for the cloud-agnostic S3 storage. Docker Desktop and Kubernetes are needed for running the Jupyter Notebooks, and the README marks that as optional, if you want the machine learning capabilities. And a basic understanding of Python and SQL is recommended for effective navigation and customization of the project.

That last point is the honest signal about what this repository is. It is a project you read and modify rather than a project you run and forget, and the phrase effective navigation and customization says so. A learner who wants to see a working pipeline can follow the four quick start steps, and the value comes from taking it apart afterwards.

The repository is designed to run both locally and on cloud environments, offering flexibility in deployment and testing, which in practice means the same code has to work in two quite different places. The separate DevOps repository, linked in the resources, exists for the Druid, MinIO and other component setup, so the heavyweight infrastructure configuration is kept out of the teaching repository.

The resources list is unusually good. There is a video tutorial on building the project, the full blog post detailing the project's development, challenges and solutions, a curated list of open-source data engineering projects to explore, a vault of resources, tutorials and guides, and the author's book on the evolution of data engineering and its design patterns.

The licence field is empty, so the terms for reuse are not stated in the repository, and that is one more thing to settle before you copy any of it into work you intend to ship.

Editorial conclusion

Adopt this repository as a reference architecture if you are learning data engineering, because it wires seven tools into one pipeline you can run locally, and the write-up explains the trade-offs rather than assuming the tool choices. Do not expect a production template, since the licence is not stated, the project is an educational exercise with an explicit disclaimer about scraping, and there are no releases to pin. Before you start, decide whether you want the original Spark-based version on the v1 branch or the current delta-rs based one, and read the scraping disclaimer, because the first step of the pipeline fetches somebody else's website and the terms of that site are your responsibility, not the repository's.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. ssp-data/practical-data-engineering on GitHub