We track 45,667 open-source projects on this site. Every night a job reads each one from GitHub, updates the numbers, and writes one row per project. Here is what that job does, because a daily scraper is the sort of thing that is easy to assume does more than it does.
What runs
One pass over the project list, calling the GitHub API for the repositories we have not seen in the last day. The response gives us stars, forks, open issues, the primary language, the licence identifier and the date of the last push. Those are written to the project row; the one-line description is not touched.
The star counts are also recorded as a point per project per day. That is the whole basis of the weekly ranking: two snapshots at the edges of a week, compared. The detail is on /ranking-methodology.
What it does not do
It does not read the README. The description on a project page is the repository's own GitHub description, captured when the project was first discovered. It is not a summary of the README, and it is not rewritten afterwards. When we translated 2,113 of them out of Chinese in October, the fix was a separate pass, because the snapshot never had anything to do with it.
It does not judge the project. "Active", "slowing", "inactive" and "archived" are arithmetic on the last push date and one flag. There is no model in the loop and no editorial judgement.
It does not run the code. We do not know whether a project's docker compose up works. When we recommend one on a page, it is because we read its README, and we say so.
It does not fetch releases. The last push date is not the last release date, and a project can push every week and release twice a year. We show the push date because it is one fact we can get reliably, and the earlier thread on what "maintained" means here has the reasoning.
The parts that are not free
The API is the cost. 45,667 repositories at one call per project per day is a real number of requests, so the job batches, asks for the fields it needs rather than everything, and skips the projects it has already seen today. It also writes an audit row for every run, so a day that quietly fetched half the catalogue is visible rather than assumed.
The other cost is the temptation. Every extra field is one more thing to be wrong, and the moment the snapshot starts carrying opinions the site stops being a mirror of GitHub and starts being a publication. That is a different product.