# Querido Diário: Install and Run Gazette Spiders

> Querido Diário is a Python and Scrapy project for collecting Brazilian official gazettes from municipal publisher websites. It suits developers and contributors who need source-specific spiders, local execution, and database tooling.

**okfn-brasil/querido-diario** — 📰 Diários oficiais brasileiros acessíveis a todos     |     📰 Brazilian government gazettes, accessible to everyone.

- Repository: https://github.com/okfn-brasil/querido-diario
- Website: https://queridodiario.ok.org.br/
- Stars: 1,415 · Forks: 483
- Language: Python
- License: MIT
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/okfn-brasil-querido-diario

## Querido Diário's municipal gazette acquisition boundary

Querido Diário addresses a specific problem in Brazil's public-information ecosystem: official municipal gazettes are published across separate websites, making consistent collection difficult. This repository is the scraping component of the wider Querido Diário ecosystem. Its responsibility is to retrieve data from gazette publishers, rather than provide the public-facing website or a complete searchable archive.

The intended contributors are Python developers who can inspect a municipal publisher's pages, encode its navigation and document links in a spider, and test the resulting collection. People looking for ready-made gazette pages can use the project's public site, but this repository is aimed at acquisition code and its development. Based on the repository layout, work is concentrated under `querido_diario_raspadores`, with project guidance in `docs/`.

That boundary matters when choosing the project. It is not presented as a general-purpose downloader that accepts any website URL. The documented generator creates a spider named for a municipality and then expects the contributor to organize it by Brazilian federative unit, or UF. Each publisher can expose different HTML, URLs, file types, and navigation, so the project depends on source-specific extraction logic rather than one universal parser. That design suits contributors who understand Scrapy and can maintain code when a publisher changes.

## Scrapy spiders and the Querido Diário data path

The implementation uses Scrapy as its collection engine. A spider represents a publishing source, requests the relevant gazette pages, follows the source-specific flow, and produces collection output that the project stores locally. The README does not expose the internal item schemas or the full request pipeline, so those details should not be inferred beyond this documented boundary.

Spider files live in a `spiders` directory and are organized by UF. The project supplies a `qdtemplate` template through Scrapy's `genspider` command. This reduces initial boilerplate and gives a new municipality-specific spider a defined starting structure. The contributor still has to place the generated file in the correct UF directory and adapt its behavior for the target publisher.

The repository also includes operational tools around Scrapy. The `Makefile` defines targets for synchronizing the project environment, checking and formatting code, running a spider, opening `querido-diario.db` with `sqlite3`, launching `scrapy shell`, running tests, updating the lock file, exporting requirements, and starting a crawl from a supplied date. These targets support development and debugging rather than merely publishing a small collection script. The `shell` target is relevant when a contributor needs to inspect Scrapy's request and item environment while working on a source.

## Installing the Querido Diário development environment

The README requires `uv`, which manages the correct Python version and the project virtual environment. From the repository root, run the documented setup sequence. It installs dependencies, including development dependencies, for `querido_diario_raspadores`, and installs the code-standardization hooks used by the project.

```bash
cd querido_diario_raspadores
uv sync --all-groups
cd ..
uv tool install pre-commit
pre-commit install
```

After this sequence, the checkout should have its scraper environment synchronized and the Git pre-commit hook installed. The README says that setup for other operating systems is available in `docs/CONTRIBUTING.md` under the section named "como configurar o ambiente de desenvolvimento".

The `Makefile` defines separate targets for project synchronization, dependency locking, and requirement export. Its `sync` target uses `uv sync --project $(SRC_DIRS) --all-groups`, while `lock` uses `uv lock --upgrade`. Because the latter explicitly upgrades the lock, it should not be treated as ordinary environment installation. The documented setup remains the appropriate starting point for a first checkout.

## Generating and organizing a municipal spider

New integrations begin with Scrapy's template support. The README names the municipality through the `<uf_nome_do_municipio>` argument and supplies the publisher URL as the second argument. Run the generator from the scraper project directory:

```bash
cd querido_diario_raspadores
uv run scrapy genspider -t qdtemplate <uf_nome_do_municipio> <https://sitedomunicipio...>
```

The command creates `uf_nome_do_municipio.py` in `spiders`, with several fields already populated. It does not claim that the spider is complete. The README specifically says to move the generated file to the appropriate directory for its UF.

This workflow makes source ownership visible in the repository structure. A generated spider for one municipality is not interchangeable with one for another unless the publisher happens to share the same site structure. Contributors also need to review the project contribution documentation before submitting changes, because the template is only an initial scaffold.

Formatting and validation remain part of the development workflow. The `Makefile` defines `check`, `format`, and `test` targets. The check commands invoke `isort`, `black`, and `flake8` against `querido_diario_raspadores`, while the test target invokes `pytest tests/ -v`. A successful test suite should not be treated as proof that every municipal site remains scrapeable.

## Listing spiders and running the first crawl

For a first real use, list the spiders already registered with Scrapy and run one whose name appears there. The documented workflow starts inside `querido_diario_raspadores`:

```bash
cd querido_diario_raspadores
uv run scrapy list
```

The command prints the available spider names. Replace `<nome_do_raspador>` with one of those names when running a spider:

```bash
uv run scrapy crawl <nome_do_raspador>
```

The README identifies `ba_acajutiba` as an example spider name in its execution guidance. The placeholder form is not executable by itself, so it must be replaced with a name returned by `uv run scrapy list`.

According to the README, gazettes collected during the crawl are saved under `querido_diario_raspadores/data`. The `Makefile` also shows that this path can contain directories because its `clean` target removes directories below `querido_diario_raspadores/data`. Users should inspect that directory after execution and compare the result with the source site.

The `Makefile` provides `run_spider` for the normal crawl and `run_spider_since` for a crawl with a `start` argument. The latter expands to `scrapy crawl -a start=$(START) $(SPIDER)`. Because the README excerpt does not state the accepted `start` format, a contributor should inspect the target spider before choosing a value.

## Database inspection and recurring scraper work

Querido Diário includes local database tooling in addition to crawl commands. The `Makefile` defines a `sql` target that changes into `querido_diario_raspadores` and starts `sqlite3 querido-diario.db`. This indicates that development can involve structured local records, but the README excerpt does not document the database schema or state whether every spider populates it identically.

The repository also defines a `shell` target that changes into the scraper directory and starts `uv run scrapy shell`. This is the documented mechanism for entering an interactive Scrapy environment while working on a source. The Makefile does not explain which specific expressions a contributor should inspect, so it would be misleading to claim a particular debugging procedure beyond launching the shell.

For source maintenance, the project has recurring costs that are easy to underestimate. Publisher pages can change, and an affected spider may require selectors, request handling, or output changes. The repository provides code checks and a test target. Production scheduling, alerting, request-rate controls, and deployment are also absent from the documented workflow. It is therefore safer to classify this repository as collection code with development utilities rather than a complete deployment package.

The project is not archived, and the last push was on 2026-10-12. The repository has no GitHub releases, so there is no published release entry to anchor an upgrade policy or distinguish release versions from ordinary commits.

## Limitations and a different acquisition approach

The central limitation is dependence on publisher websites. Querido Diário is appropriate when contributors are prepared to maintain site-specific spiders for municipal sources. It is the wrong choice when the required municipality has no spider, when the publisher blocks or restricts automated collection, or when organizational review does not permit access to that source.

The README does not document robots.txt handling, request-rate limits, authentication, retries, duplicate detection, PDF validation, or a compatibility guarantee for any municipal publisher. It also does not document rollback for collected data. These omissions matter because a gazette scraper depends on both external page structure and the integrity of retrieved documents. The presence of tests does not fill those documentation gaps.

A real alternative is a publisher-provided RSS feed, API, bulk export, or another structured endpoint when one exists. Its approach is materially different: structured data avoids interpreting general HTML pages and reduces some selector maintenance. Querido Diário remains relevant when documents are exposed only through web pages or when collection must reproduce a site's publication navigation. If a structured source exists, its compatibility and access terms should be evaluated before adding or maintaining an HTML spider.

The repository is licensed under MIT, which permits reuse and modification under its terms. That software license does not itself grant access to publisher documents or establish publisher-specific reuse rights. Anyone deploying collection code must distinguish the code license from the terms that apply to the source data and retrieved documents.

## Conclusion

Adopt Querido Diário when you need to collect municipal gazettes through its Scrapy spiders or contribute a new publisher integration. Do not choose it for a complete end-to-end gazette platform, since this repository covers acquisition rather than publishing, search, or presentation. Before adoption, inspect the target publisher's terms, confirm that its page structure fits the intended spider, and verify where collected files and SQLite data are written.

## FAQ

### What is Querido Diário used for?

It is the scraping component of the Querido Diário ecosystem and is responsible for collecting Brazilian official gazettes from municipal publisher websites. It is aimed at Python and Scrapy contributors and at users of data produced by its spiders.

### How do I install the Querido Diário development environment?

Install `uv`, enter `querido_diario_raspadores`, run `uv sync --all-groups`, then install `pre-commit` from the repository root and run `pre-commit install`. The README links `docs/CONTRIBUTING.md` for other operating systems.

### Where does Querido Diário save collected gazettes?

The README says collected gazettes are saved under `querido_diario_raspadores/data`. The `Makefile` indicates that this output path can contain directories, but the excerpt does not fully document the naming scheme.

## Sources

- [Issues](https://github.com/okfn-brasil/querido-diario/issues)
- [License: MIT](https://github.com/okfn-brasil/querido-diario/blob/main/LICENSE)
- [okfn-brasil/querido-diario on GitHub](https://github.com/okfn-brasil/querido-diario)
- [Project website](https://queridodiario.ok.org.br/)
- [README](https://github.com/okfn-brasil/querido-diario/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/okfn-brasil-querido-diario