Brasil.IO turns Brazil's open data into something you can query

The product is a pipeline, not a portal

The README explains the problem in a way that is more precise than the project description. There are various public databases available in Brazil, mainly after the creation of the LAI, the law on access to information, but those data are not always accessible. If the information is not in a structured, machine readable format, using open formats, or if the documentation, the metadata, is missing, then it is closer to closed than to open.

That diagnosis is the reason the project exists. Brasil.IO was born to solve that problem, which is to make open data more accessible. The method is programs that automate the collection, cleaning and enrichment of the data, a process the project calls libertação de dados, data liberation, and then making the data available for consultation and download. On top of that the project does activism, using the LAI, so that more open data becomes available and with better quality.

So the repository is not a website. It is the backend of the Brasil.IO platform, and the README says so directly, pointing to the link on each dataset page for the collection scripts.

That structure has a practical consequence for a contributor. You are not adding a page to a site, you are adding or fixing a program that knows how to fetch a particular government source, parse whatever format it happens to publish that month, clean it, and hand it to the platform. The directory layout reflects that. There is a brazil_data/ directory for the data itself, a scripts/ directory for the collection programs, a data-server/ directory, and a dedicated covid19/ directory for one dataset that has its own history.

The licence is the GNU General Public License version 3, which is worth noting for a data project, since it means the scripts that collect public data are copyleft, while the data itself is public.

Django on PostgreSQL, with Redis doing the queueing

The requirements file describes a conventional Django stack with a few choices worth reading.

The base set is Django pinned to the 4.2 series with an upper bound below 4.3, dj-database-url for reading the database connection from an environment variable, django-extensions for the enhanced shell, django-mailer, django-redis for the cache, django-storages with the s3 extra for file storage, gunicorn as the WSGI server, psycopg2-binary, python-decouple for settings, and whitenoise for serving static files. There is also a Sentry client pulled from a GitHub archive rather than PyPI, pinned to 1.45.2, and ipython for the enhanced shell.

The custom project requirements are where the domain shows through. django-cachalot for cache invalidation on model writes, django-ratelimit and django-recaptcha for abuse and bot control, django-registration for accounts, django-rq and rq-scheduler for background jobs, djangorestframework for the API, django-localflavor for Brazilian states and cities, django-markdownx for the documentation pages, django-cors-headers, django-admin-rangefilter, and django-cache-memoize.

The data handling libraries are the giveaway about the data sources. rows is pulled from a GitHub archive of the develop branch, and openpyxl, xlrd and xlwt are all present, which is spreadsheet in and spreadsheet out. feedparser is there for the sources that publish feeds, lxml for the ones that publish HTML, requests and requests-cache for fetching, retry for the ones that fail, tqdm for progress on the long ones, and click for the command line entry points.

The poetry of it is that a public dataset in Brazil is quite often a spreadsheet, and this stack is built to survive that.

The compose file is the development environment

There is a compose.yml at the repository root, and it is the clearest picture of how the platform runs.

The web service is the Django application, built from the Dockerfile with an ENV_TYPE argument of development, mounting the project directory and the data directory, and publishing port 5000. It depends on the database being healthy and on the messaging service being started.

  db:
    env_file:
      - path: "docker/env/db"
        required: true
    shm_size: ${DB_SHM_SIZE:-8g}
    image: "postgres:17.6-trixie"

The database is PostgreSQL 17.6 on the trixie distribution, with a shared memory size that defaults to 8GB, which is a large value and tells you the data set being loaded is not small. The health check is pg_isready against the postgres user, on a three second interval with five retries, and the web service waits for it with the service_healthy condition.

The messaging service is Redis 6.2 on alpine, and the worker service is a second copy of the web image with a different command, deployed with two replicas, which is the job queue in practice. There is a separate mail worker, and a mail service using mailhog with port 8025 published, which is the local mail catcher for development.

The Makefile wraps all of this so that a contributor never has to remember a compose command. There are targets to build and pull the images, to build without cache, to run a shell inside the web container as the normal user or as root, to connect to the database shell, to clear the Django cache stored in Redis, and to run the linter.

dbshell: 				# Connect to database shell using `web` container
	$(MAIN_RUN) python manage.py dbshell

There is also a build-ci target, which strips the USER directives from the Dockerfile and the user line from the compose file so that the build runs in an environment without those features, which is a small detail that tells you the project is built in CI as well as locally.

The container build pins the database client and runs as a normal user

The Dockerfile shows a build that is careful about the parts that break in production.

The base is python:3.13-slim-trixie, with the environment set to keep the output unbuffered, a work directory of /app and a volume at /data. A group and a user are created with a gid and uid that default to 1000, the account is created with a disabled password, and the app directory is chowned to it, so the application never runs as root.

The system packages are the build essentials plus gettext, gnupg, the XML development headers and wget, and then the PostgreSQL apt repository is added and the client libraries for PostgreSQL 17 and libpq are installed with the extra packages purged afterwards. Pinning the client to 17 to match the server image in the compose file is the kind of detail that avoids a class of subtle failures.

The Python side installs with a pip cache mount, so the layer is cached between builds, and the requirements are copied in as both requirements.txt and requirements-development.txt, with the development set installed only when the environment type is not production. A git, make, ripgrep and tree installation in the same branch covers the development environment, and a custom nginx configuration is added from srv/nginx.conf.sigil, which is what serves the static files and proxies the application.

The repository layout shows the rest. There is an api/ directory, a core/ directory, a project/ directory, a dashboard/, a traffic_control/ module, a utils/ directory, a covid19/ directory for one dataset, a branding/ directory, templates/ and static/ for the presentation layer, and a Procfile and an app.json, which are the deployment descriptors for the platforms that consume them.

There is also a todo/ directory and a docs/ directory, with the development setup instructions in docs/dev-setup.md, and the README points contributors there.

Contributing means a collection script, a dataset suggestion or a donation

The README closes with a Colabore section, and it lists three distinct ways in, which is a useful signal about what the project actually needs.

The first is funding. There is a donation page at apoia.se/brasilio, and the note explains that the donations help keep Brasil.IO free, which is the project's stated goal.

The second is suggesting datasets. If you know an open dataset that is not yet in brasil.io/datasets, there is a page at brasil.io/datasets/sugira/ to propose it. This is the cheapest way to contribute and it is the one that expands coverage, which is the binding constraint for a project whose value is the number of sources it maintains.

The third is programming, and this is where the repository itself comes in. The README points at docs/dev-setup.md to learn how to run the project on your machine and start contributing.

The separation between suggesting a dataset and writing the script for it is the interesting part. A dataset suggestion is a public data source nobody has packaged yet. The script is the work: something that knows where that source lives, how it changes shape, and how to clean it. A source that publishes a different file format each month is the hard case, and that is exactly the case the platform exists to absorb.

The repository also carries AGENTS.md, a .pre-commit-config.yaml, a pytest.ini, a conftest.py, a setup.cfg and a lint.sh, so the contributor path is a conventional Python and Django one, and the topics confirm the shape of the project with opendata, dados-abertos, data and brazil.

Editorial conclusion

Brasil.IO fits a data team, journalist or researcher who needs Brazilian public records in a consistent, documented shape rather than as a pile of files with different formats per source, and it fits a developer who wants to contribute a collection script. It is the wrong tool if you need the raw bytes of one dataset, since the value here is the cleaning and the metadata rather than the download. Verify first which datasets you actually need are covered, since the README points to the dataset list rather than promising coverage, and read docs/dev-setup.md before setting up the compose environment.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. Project website
4. README
5. turicas/brasil.io on GitHub