# Basedosdados Pipelines: An AI-Assisted Orchestrator for Public Data Datalake Ingestion > This repository hosts the orchestration flows that capture, ingest, and treat data for the Base dos Dados datalake. Its standout feature is an AI agent that guides new dataset onboarding through 11 defined stages, with human checkpoints before production promotion. **basedosdados/pipelines** — 🔀 Orquestrador de fluxos de captura, ingestão e tratamento de dados da BD - Repository: https://github.com/basedosdados/pipelines - Stars: 52 · Forks: 23 - Language: Jupyter Notebook - License: not declared - Published: 2026-08-21 · Updated: 2026-08-21 · Language: en - Canonical page: https://hysenlabs.com/projects/basedosdados-pipelines ## What This Repository Solves and Who It Serves The repository addresses a specific operational problem: getting new datasets into the Base dos Dados public datalake. Base dos Dados is a Brazilian open-data initiative that centralizes public data, and this repository is the orchestration layer for that effort. The README describes it as the home of 'fluxos de captura e subida de dados no datalake da Base dos Dados', meaning flows for capturing and uploading data. The intended user is a contributor who wants to add a new dataset, not a general data engineer looking for a generic pipeline framework. The project is built around a Jupyter Notebook environment, which suggests an interactive, exploratory workflow rather than a production-grade scheduler. If you work outside the Base dos Dados ecosystem, this repository has little direct value, because it is tightly coupled to their datalake structure and metadata standards. ## The AI Orchestrator: A Prompt-Driven Onboarding Flow The most distinctive mechanism is the AI-assisted ingestion agent called `orchestrator`. To start onboarding a new dataset, you invoke this agent with a structured prompt. The README provides a template with fields like `dataset_slug`, `Sources`, `Drive folder`, `Architecture suggestion`, `Organization`, and `Notes`. The more context you supply, the fewer clarifying questions the agent asks. The agent then executes a sequence of 11 steps, listed in parentheses: context, architecture, download, cleaning, upload, dbt, validation, IDs, metadata dev, approval, metadata prod, and PR. The flow pauses for human approval before promoting anything to production. This design puts a human in the loop at the most sensitive transition, which is sensible for a public data repository where errors could propagate to downstream users. The prompt-driven approach means the quality of the output depends heavily on the prompt's completeness, a trade-off the README acknowledges by suggesting 'quanto mais contexto você fornecer, menos perguntas o agente fará'. ## Getting It Running: Commands and Configuration The README does not include installation commands, environment setup, or configuration keys. The only actionable instruction is the prompt template for the `orchestrator` agent. You would start by cloning the repository and opening it in a Jupyter environment, given the primary language is Jupyter Notebook. The prompt fields act as configuration: `dataset_slug` names the dataset, `Sources` points to raw files either as a URL or a local path, `Drive folder` specifies where data is stored in Google Drive, `Architecture suggestion` describes the table layout, `Organization` names the source, and `Notes` captures unusual aspects. There is no mention of dependency files, environment variables, or command-line tools. This lack of setup documentation is a gap. A new contributor would need to infer the environment from the repository's notebooks or seek guidance in the linked Discord community. The README explicitly directs contributors to the `CONTRIBUTING.md` file, which is not included in the provided material, so the actual setup steps remain unknown. ## The 11-Step Flow: A Clear Sequence with a Human Gate The 11-step sequence is the core of the onboarding process. It starts with context gathering, moves through architecture design, then download and cleaning, followed by upload and dbt transformation. Validation and ID generation come next, then metadata for development, an approval step, metadata for production, and finally a pull request. The approval step is the only explicit human checkpoint mentioned, positioned between development metadata and production metadata. This means the agent can autonomously prepare everything up to that point, but it cannot promote to production without human sign-off. The inclusion of a PR at the end suggests the code and metadata changes are reviewed through a standard version control process. The flow is sequential, which is simple to reason about but may be slow for large datasets. There is no mention of parallel execution or retry logic, so a failure at any step could stall the entire onboarding. ## Limitations and Failure Modes The most obvious limitation is the dependency on an AI agent whose behavior is not described in detail. The README does not specify which AI model powers the orchestrator, how it handles ambiguous instructions, or what happens when the agent fails a step. If the agent produces an incorrect architecture or cleaning logic, the human approval gate is the only safeguard, but the README does not explain how to review or correct the agent's output. Another limitation is the tight coupling to Google Drive and the Base dos Dados datalake. The prompt asks for a Drive folder, and the flow includes upload and metadata steps that assume the project's specific storage and schema conventions. This is the wrong tool for anyone who wants to orchestrate data pipelines for a different destination. The README also gives no information on error handling, monitoring, or debugging, so troubleshooting a failed onboarding would rely on community support via Discord. Finally, the repository has no license listed, which is a legal ambiguity for anyone considering reuse. ## Alternatives and Their Different Approaches A direct alternative is to manually perform the 11 steps without the AI agent. That is, a contributor would write their own scripts for download, cleaning, upload, and metadata creation, following Base dos Dados conventions. This approach gives full control and avoids the uncertainty of AI-generated steps, but it requires more effort and familiarity with the datalake's standards. Another alternative is to use a general-purpose orchestration tool like Apache Airflow or Prefect. Those tools are not tied to Base dos Dados and offer scheduling, retries, and monitoring. However, they would require you to build all the Base dos Dados-specific logic yourself, including metadata handling and the approval workflow. The key difference is that this repository embeds domain knowledge about the datalake into an AI-driven flow, whereas generic tools leave that domain logic to the user. For a one-off dataset onboarding, the AI agent may save time; for recurring pipelines, a dedicated orchestrator might be more robust. ## Maintenance and Upgrade Cost The README does not mention release history, versioning, or a maintenance schedule. The repository is not archived, which suggests active development, but the last push date is unknown from the provided material. The primary language being Jupyter Notebook raises concerns about maintainability. Notebooks are harder to version and test than plain Python modules, and they often mix code with outputs, which can bloat the repository. The AI agent's behavior likely depends on external model versions, so upgrades to the model could change the agent's output without notice. The repository also relies on a complex stack: dbt for transformations, Google Drive for storage, and a metadata approval process. Each of these has its own upgrade path. The absence of a license is a significant maintenance issue: without a license, you have no legal permission to reuse or modify the code, even for internal purposes. Before adopting this repository, you should check the `CONTRIBUTING.md` file and the repository's license status directly, as these details are not covered in the README. ## Conclusion Adopt this repository if you are part of the Base dos Dados community or need to onboard datasets into that specific datalake, and you are comfortable with an AI agent drafting the pipeline while humans review each stage. Do not adopt it if you need a generic, standalone data orchestration tool, because it is tightly coupled to the Base dos Dados datalake and its metadata conventions. Before relying on it, verify the current state of the orchestrator agent, the exact prompts and configuration it expects, and whether the 11-step flow matches your dataset's complexity, since the README gives no details on failure handling or rollback procedures. ## FAQ ### What is a pipeline in the basedosdados pipelines repository? Here a pipeline is one of the flows that capture and upload data into the Base dos Dados datalake. The repository holds those flows along with their dbt models, macros and tests, orchestrated on Prefect 3, with a Dockerfile for running them and a manage.py entry point. ### What is the purpose of the basedosdados pipelines? To onboard datasets into the Base dos Dados datalake through a staged agent workflow that ends in a pull request. The README states the agent pauses for human approval before promoting anything to production, so publishing is gated rather than automatic. ### What does the Base dos Dados orchestrator agent ask for? A dataset slug, the sources as a URL or a local path to raw files, a Drive folder shaped as BD/Dados/Conjuntos//, a short architecture suggestion, the source organisation name, and a notes field for anything unusual such as wide format, files split by state, or an API that requires authentication. ### Where do the basedosdados pipelines get credentials from? The example environment file points at Google application credentials for storage access, a Prefect cloud backend with its own API URL and port, and a HashiCorp Vault address with a token. The Vault pair is served by the hvac client listed in the dependency manifest. ## Sources - [Official README](https://github.com/basedosdados/pipelines#readme) - [Project repository](https://github.com/basedosdados/pipelines) - [README](https://github.com/basedosdados/pipelines/blob/main/README.md) - [Releases](https://github.com/basedosdados/pipelines/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/basedosdados-pipelines