repology-updater is the backend that keeps Repology's package data fresh

What the updater does inside the Repology project

Repology is a service which monitors a lot of package repositories and other sources and aggregates data on software package versions, reporting new releases and packaging problems. This repository contains the Repology updater code, a backend service which updates the repository information, and the documentation points to a separate web application repository for the front end. That split is the right shape for the problem. The web application answers questions about versions, and the updater is the part that goes out to hundreds of upstream repositories, downloads their package indexes, parses them into a common internal format, and pushes the result into a database. The topic list of the project is short and accurate: package, packages, repository, version. There is no user management, no web interface and no API surface in this repository, so anyone expecting a service they can call is looking in the wrong place. The project is maintained by Dmitry Marakasov, the homepage is repology.org, and the licence is GPLv3 or later.

The dependency stack and the two PostgreSQL extensions

The runtime requirements are listed in full and they explain the shape of the service. Python 3.11 or newer is the floor, alongside Jinja2 for templating, psycopg2 for PostgreSQL access, pyyaml, xxhash for fast hashing and pydantic for validation, and libversion, which is a Python module that also requires the libversion C library. PostgreSQL 16.0 or newer is required, together with two extensions: the libversion extension, and pg_trgm for trigram indexing. On top of that sits a long list of modules needed for fetching and parsing repository data, and this list is the best evidence of what the updater actually does. jsonslicer and lxml and pyparsing cover structured formats, protobuf and rubymarshal cover binary package manager formats, rpm covers the RPM ecosystem, and tomli covers packaging metadata, while requests, yarl, zstandard and brotli handle fetching and compression. git and rsync are needed too, because some repositories are cloned rather than downloaded. This is a scraper for a very heterogeneous set of formats, and the dependency list is essentially the inventory of what upstream repositories look like.

Cloning the rules, and where configuration comes from

There is a step before anything can run, and it catches out newcomers because the rules live in a separate repository. Since the repology rules live in a separate repository you need to clone it first, and the location may be arbitrary, but the rules.d subdirectory is what the default configuration file points to, so using that is the simplest way.

git clone https://github.com/repology/repology-rules.git rules.d

Configuration follows a three-level scheme, and the documentation is precise about not editing the wrong file. You may need to tune settings shared by all repology utilities, such as the directory for storing downloaded repository state or the DSN, which is the string specifying how to connect to the PostgreSQL database. The default values live in repology.conf.default, and you create a repology.conf in the same directory to override them, with an explicit instruction not to edit repology.conf.default. You can also point at an alternative config file through the REPOLOGY_SETTINGS environment variable, or override settings on the command line. By default, repology uses the ./_state directory for storing raw and parsed repository data, and a repology database, user and password called repology on localhost.

Creating the database, one psql command at a time

The database setup is given as a sequence of psql commands, and each line does something specific. You need a PostgreSQL server running, then you create the database, create the user with a password, grant that user all on the database, grant CREATE on the public schema to PUBLIC, and create the two extensions.

psql --username postgres -c "CREATE DATABASE repology"
psql --username postgres -c "CREATE USER repology WITH PASSWORD 'repology'"
psql --username postgres -c "GRANT ALL ON DATABASE repology TO repology"
psql --username postgres --dbname repology -c "GRANT CREATE ON SCHEMA public TO PUBLIC"
psql --username postgres --dbname repology -c "CREATE EXTENSION pg_trgm"
psql --username postgres --dbname repology -c "CREATE EXTENSION libversion"

The order matters. The extension commands are run against the repology database, so the database has to exist and the connection has to reach it, which is why the schema grant comes before them. The libversion extension is the interesting one, because it is the database side of the version comparison logic that the whole service is built around, and it is a separate PostgreSQL extension project rather than something bundled with the server. The documentation adds one warning at the end: if you change the credentials, do not forget to add the actual ones to repology.conf.

The dump path, and the disk cost of the update path

The database starts empty and there are two documented ways to fill it, with very different costs. The first uses a database dump of the main Repology instance, and it is described as the fastest and simplest way to fill the database.

curl -s https://dumps.repology.org/repology-database-dump-latest.sql.zst | unzstd | psql -U repology

The second is a complete update process, which includes fetching and parsing all repository data from scratch and pushing it to the database. Before you start it, note that you need more than 11GiB of disk space for the Repology PostgreSQL database, and additionally more than 11GiB space for raw and parsed repository data if you decide to run a complete update process. That figure is the single most useful number in the documentation for anyone planning hardware, because it sets the floor for a full run. The dump path skips the fetching and parsing entirely and gives you the same shape of data, which is why it is the sensible starting point for anything that is not specifically about testing the scrapers.

The update pipeline as four flags

The complete update is driven by one command with four options, and the documentation explains each of them.

./repology-update.py --fetch --fetch --parse --database --postupdate

The --fetch option tells the utility to fetch raw repository data, downloading files, scraping websites and cloning git repos into the state directory, and the documentation draws attention to a quirk: it needs to be specified twice to allow updating. The repetition exists because the first pass records the state of each repository and the second pass detects what changed, so a single fetch would only give you the current state with no way to tell a new version from an old one. The --parse option enables parsing downloaded data into the internal format, which is also saved into the state directory. The --database option pushes the processed package data into the database, and --postupdate runs optional cleanup tasks. The first run is expected to take several hours, and subsequent updates will be faster, and the same command is used to update later.

Schema checks, regression dumps and the lint targets

The Makefile shows how the project is checked, and the four targets are a good map of the codebase. The lint target runs check, test, flake8 and mypy in sequence. The test target invokes pytest verbosely with the reason summary flag, and there is a separate canonize target that adds a regression selection and a reset flag, which is the mechanism for regenerating expected outputs when a behaviour change is intended. The check target is the one specific to this project, and it runs the schema checker twice, once over every YAML file found under repos.d and once over the maintainers.yaml file, using the script repology-schemacheck.py. The check command exists because the repository definitions are YAML files whose schema is enforced, which is a different concern from the SQL schema and a different concern from the Python type checking. The mypy target covers the two top level scripts and the repology package and tests, and flake8 covers the top level Python files, the package and the tests.

Where the updater is the wrong tool

The clearest wrong case is wanting to look something up. Repology is a public service, so if the question is what is the newest version of a package in a distribution, running a scraper fleet locally to answer it is a large amount of work for a result you can get in a browser. The second wrong case is a machine that cannot host the service. The requirements are a PostgreSQL server with two extensions, more than 22GiB of disk for a full run, and a network position from which it can reach hundreds of upstream repositories, which for some environments means egress rules, and for others means the whole thing simply is not appropriate. The third is a need for real time. The update process is a batch job that takes hours on the first run, so a system that needs to know about a new release within minutes of it appearing is asking for something this design does not offer. The updater also has a hard dependency on the shape of the upstream repositories, and when a distribution changes its index format, the rules in the separate repository have to be updated, which is a maintenance commitment rather than a one-off setup.

Editorial conclusion

Adopt repology-updater if you want your own copy of Repology's package version data, for a distribution mirror, an internal package index or a research dataset on packaging drift, and if you have a PostgreSQL instance and disk to spare. Do not adopt it if you only need to look up a version, since the public service answers that question already, and do not adopt it as a scheduled job on a small machine, because the complete update path takes hours and needs more than 11GiB for the database and more than 11GiB for the raw state. Verify two things first: that the rules repository is cloned where the default configuration expects it, since the rules live in a separate repository and the default config points at the rules.d subdirectory, and that your PostgreSQL has both the libversion and pg_trgm extensions available, because the database creation steps install them explicitly. The practical first step is to load a database dump rather than run a full update, since that path gives you a working dataset in minutes instead of hours.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. Project website
4. README
5. repology/repology-updater on GitHub