data-engineering-wiki: a CC0-1.0 knowledge base for data engineers using Obsidian

Community-authored notes for data engineers, not a textbook

The Data Engineering Wiki is a collection of notes on data engineering practice, built and maintained by the data engineering community. The notes are organized into 5 categories: Concepts, FAQ, Guides, Tools and Learning Resources. Each category is a separate folder in the repository root, with Learning Resources as an exception, stored as a single Markdown file.

The notes are, in the project's own words, "connected organically but loosely organized." That framing matters for expectations. Individual notes link to one another in a graph style rather than a linear sequence, so there is no enforced reading order and no chapter-by-chapter progression. Someone who wants a linear curriculum will need to impose that structure themselves, either through Obsidian's graph view or by following the pointers in the Learning Resources category.

The project is also available as a live website at dataengineering.wiki, which means the same content is reachable both as a hosted site and as a local Obsidian vault. The repository was last pushed on 2026-09-15 and has no GitHub releases. All 5 content categories, including all notes and support files, are covered by the CC0-1.0 license.

What each of the 5 categories covers

Concepts holds notes on core data engineering topics. FAQ answers frequently asked questions about the field. Guides go further: they are designed for decision-making, making them more opinionated than Concepts entries and closer to recommendations than definitions.

Tools covers commonly used tools for data engineering. Learning Resources collects pointers to external content recommended by the community rather than hosting that content directly. The distinction matters: Tools entries describe software the field uses, while Learning Resources entries point outward to books, courses and other materials the community finds worth reading.

Beyond those 5 main categories, the repository root includes a Community/ folder, which the homepage links to as dataengineering.wiki/Community/Community. An Index.md file sits at the top level as well. A Templates/ folder suggests contributors have some format guidance available, and the CONTRIBUTING.md file is the documented place to read contribution rules. The 20 top-level entries include Assets/, CODE_OF_CONDUCT.md, BACKERS.md and the publish.css and publish.js files used for the web version.

Opening the repository as an Obsidian vault without any CLI commands

Running the wiki locally takes 4 steps. First, download the GitHub repository. Second, download the free Obsidian desktop app (free, no affiliation). Third, run the Obsidian app and choose Open folder as vault, then click Open. Fourth, point the file browser at the folder where you downloaded the repository and click Open again.

No command-line tools are involved. The process runs entirely through Obsidian's graphical interface. Once the vault is open, Obsidian indexes the notes and you can search, follow internal links, and browse the graph view that visualizes connections between entries.

A practical detail: the files are ordinary Markdown. They remain readable in any plain text editor even without Obsidian. The vault is not a proprietary format but a set of standard files that Obsidian layers its note-linking and graph features on top of. For questions about Obsidian's own features, the documentation at help.obsidian.md is the recommended resource, since the wiki itself does not document Obsidian usage.

Foam and Roam Research as alternatives to Obsidian

Obsidian is the primary build target, but the wiki is compatible with Foam and Roam Research as well. All three support the double-bracket internal link format the wiki uses.

Foam is a Visual Studio Code extension that gives the same Obsidian-style bidirectional linking inside VS Code's editor. For teams whose workflow already centers on VS Code, Foam avoids adding a separate app. Roam Research is a subscription-based web app with a similar bidirectional linking model; it stores notes in its own cloud rather than locally.

The practical difference between the three comes down to where the files live and who controls them. Obsidian stores files locally on the user's machine, Foam keeps them in a VS Code workspace folder, and Roam Research stores them in Roam's cloud service. Whether every internal link resolves correctly across all three tools is not documented, so Obsidian is the safer starting point given that it is the stated build target.

CC0-1.0 means anyone can copy and redistribute the content without attribution

CC0-1.0 is a public domain dedication, not a traditional copyright license. Under CC0, the contributors waive all copyright claims over the content, which means anyone can copy, modify, distribute or incorporate these notes in any way and for any purpose with no attribution required and no share-alike conditions attached. This applies to the notes in the repository, not to any external resources that Learning Resources links to.

For a data engineering team, the implication is that notes from the wiki can move into internal documentation, team wikis or training materials without requiring legal review of license compatibility. A team building a custom onboarding guide could pull Concepts or Guides entries directly, adjust the wording for their stack, and publish the result without any obligation back to the wiki project.

Ongoing development is funded through sponsorship. Gold Sponsors listed in the README include dataengjobs.com, greatexpectations.io and datadriven.io. Anyone who wants to support the project can sponsor through GitHub Sponsors, and the BACKERS.md file lists the project's supporters.

Loose organization and the absence of a guided learning sequence

The weakest aspect of the wiki for a newcomer to data engineering is the absence of a guided path. Notes are connected organically, and no reading order is prescribed. No category is described as a complete introduction to the field, and there is no assessment of which notes are foundational versus which assume prior knowledge.

The FAQ and Learning Resources categories partially compensate for this. FAQ answers specific questions, which suits someone who knows what they do not yet understand. Learning Resources points to external content with its own structure, so if those materials have curated curricula, the wiki acts as a directory to reach them.

No GitHub releases exist in the repository as of the last push on 2026-09-15. New content arrives directly in the main branch without tagged milestones, which means there is no changelog showing what was added or changed in a given period. A team using the wiki as a shared offline reference will need to pull updates from the repository manually to stay current, since there is no release notification mechanism.

The r/dataengineering subreddit as a conversational counterpart

A badge in the repository header links to the r/dataengineering subreddit. The two resources serve distinct purposes.

The subreddit is a discussion forum: questions get answers from community members, topics trend with current events in the field, and the useful content spreads across thousands of threads without category structure. The wiki is a reference: notes are persistent, organized by category, and intended to remain useful after their initial writing.

For someone deciding which to consult, the distinction is the kind of question. The subreddit handles open-ended questions like which orchestration tool a team should adopt right now, because the community can respond to current tool landscapes and specific stack constraints. The wiki handles definitional questions like what a slowly changing dimension is or how a data lakehouse differs from a data warehouse, because the answers do not change week to week and benefit from being written down rather than retrieved from search.

The two are operated by the same community, and the wiki's Learning Resources category points to external content, some of which likely originates from subreddit discussions, though the README does not document a formal link between them.

Editorial conclusion

data-engineering-wiki suits self-directed learners who want a structured reference for data engineering concepts, tools and decisions that can be browsed offline and adapted into their own notes. It is a poor fit for beginners who need a guided curriculum with exercises, because the organization is deliberately loose and no structured learning path is documented. Before adopting it, open the wiki in Obsidian and browse the FAQ and Guides sections, since those give the clearest sense of whether the coverage depth matches what you need. The CC0-1.0 license means any note can be copied into your own docs or projects without attribution.

Frequently asked questions

Official sources

1. data-engineering-community/data-engineering-wiki on GitHub
2. Issues
3. License: CC0-1.0
4. Project website
5. README