are-we-learning-yet: a scraper and a static site that track the Rust ML crate ecosystem

The problem is crate discovery, not crate quality

Searching crates.io for machine learning terms returns a mix of bindings, abandoned experiments, and libraries that are one function deep. The README frames the project as an answer to a single question: Rust is a systems programming language, but is it a machine learning language? The site at arewelearningyet.com is the public answer, and the repository is the machinery behind it. The intended audience is narrow. It is for someone who has decided to write Rust and now needs to know which ML crates exist and which ones are worth opening. It is not a benchmark suite, it does not run any crate's code, and it does not publish accuracy or throughput numbers. The README describes the goal as cataloging the Rust ML ecosystem, which is a documentation job, not an evaluation job. That distinction matters when you read the site: presence in the catalog means a maintainer added the crate to crates.yaml, and the ordering reflects a score computed from fetched metadata. Neither one is a verdict on correctness.

Two parts: a scraper and a cobalt.rs site

The README splits the repository into two key parts. The scraper reads _data/crates.yaml, fetches additional metadata about each crate from the crates.io API and the GitHub API, and generates a score used for ordering crates. It writes _data/crates_generated.yaml, which the site generation consumes. The second part is everything else: content laid out for cobalt.rs, a static site generator that produces a _site directory on cobalt build and can run a local development server that rebuilds when content changes. The data flow is therefore one-directional and file-based. Human edits go into crates.yaml. The scraper turns that into crates_generated.yaml. Cobalt reads the generated file plus the templates and writes HTML. Nothing in that chain is a database, and nothing is queried at page load. The score is the only computed judgement in the pipeline, and the README does not describe its formula, so the ordering is a black box unless you read the scraper source in _scraper. That is the first thing I would open before trusting the order of the list.

The _tmp cache and why just clean exists

Fetched data is cached in a _tmp directory. The README gives two reasons: it speeds up repeated site generation and it avoids abusing the APIs. This is the most practically important design decision in the repository. Without the cache, every local rebuild would re-hit crates.io and GitHub for every crate in the catalog, and the README already warns that a GitHub OAuth token is needed to avoid 403 rate limiting errors while generating crate data. The cache turns an expensive, rate-limited operation into a one-time cost per crate per change. The escape hatch is just clean, which drops generated data and the cached API responses. The README states that removing the cached data forces regeneration of all crate data. So the mental model is: scrape once, edit content freely, and only clear the cache when you suspect the fetched metadata is stale or when the scraper itself changed. For a catalog whose value is freshness, that is the right default, because it keeps the common edit loop fast while leaving a documented way to pay the full cost.

Getting it running: cobalt, a Rust toolchain, and a token

The README lists three prerequisites. cobalt 0.20.0 or newer, with CI building against 0.20.4. A Rust toolchain of 1.85 or newer to build the scraper. And just as a recommended task runner, or you can use the commands in the Justfile directly. The documented sequence starts with exporting GITHUB_TOKEN, with the tip that setting it in a .env file means just picks it up automatically. Then just scrape to fetch crate and repo data for sitegen, and just serve to start a dev server on port 3000. The README notes that plain cobalt serve picks an arbitrary free port, while just serve passes --port 3000, which is a small detail that saves confusion when the site does not appear where you expect. Other recipes are just build (scrape plus cobalt build), just check (the same fmt and clippy gate CI runs), and just clean. The step people will trip over is documented explicitly: cobalt serve does not trigger the scraper to rerun when crates.yaml or the scraper changes, so you must rerun just scrape to update crates_generated.yaml. Editing the YAML and refreshing the browser will show nothing new.

Publishing is a merge, and freshness is a weekly cron

The site is served by GitHub Pages. The README states that every merge into master is automatically published by a GitHub Actions job. Separately, to keep crate statistics such as download counts and stars regularly updated, the publishing task also runs as a weekly cron job, and it can be triggered by hand from the Actions tab via workflow_dispatch. This tells you the intended freshness window: a week. If you are reading the site to decide between two crates and one of them published a release yesterday, the catalog may not know yet. The manual dispatch is the answer for that case, but it is a maintainer action, not something a reader can trigger. The split between merge-triggered publishing and weekly refreshing is sensible given the API rate limits, and it also explains why the site can look stale in its statistics while its prose is current: prose changes ride on merges, numbers ride on the cron.

Where this is the wrong tool

The catalog cannot tell you whether a crate is correct, fast, or maintained in any sense beyond what the fetched metadata supports. The README describes fetching metadata and generating a score for ordering; it does not describe running tests, checking release cadence against a threshold, or validating that a crate compiles on current stable Rust. So a crate that appears high in the list is high by whatever the score measures, and a crate that is missing is missing because nobody added it to crates.yaml. That second point is the real failure mode. The catalog is hand-curated at the seed level, which means absence is not evidence of unsuitability, and coverage depends on contributions. If you need a guarantee that every relevant crate is listed, this repository cannot give it to you, because the README describes contributions as the mechanism for adding missing crates. The other wrong use is as a dependency. Nothing here is a library you import; it is a scraper and a static site, and treating it as an API means scraping HTML that was generated from a file that was generated from an API.

The alternative is the raw sources, and the difference is curation

The obvious alternative is going straight to crates.io search plus GitHub, which is what the scraper itself does. The difference is not the data, since the scraper pulls from those same two APIs. The difference is the seed list and the ordering. crates.io search is keyword-driven and ranked by its own relevance signals; this project starts from a maintained crates.yaml and applies a computed score before presenting anything. That means you get fewer results and a human decided they belong. If your question is what exists that mentions machine learning, crates.io search wins on recall. If your question is what should I look at first among Rust ML crates, the curated list is a shorter path, provided you accept that the curation is a person's judgement plus an undocumented score. A second alternative is reading the Awesome-style lists that circulate for Rust ML, and the practical difference there is the weekly refresh: a static markdown list has no cron job pulling download counts, so it drifts in a different way.

Maintenance cost and the CC-BY-4.0 licence

The licence identifier is CC-BY-4.0, which is a content licence rather than a software licence. That fits a repository whose main output is prose and generated catalog data, but it is worth noting plainly: if you copy site text or catalog content into your own material, the licence conditions attach to that reuse, and attribution is the central term. I am not a lawyer and this is not legal advice, so read the licence text itself before republishing anything. On maintenance, the cost sits in three places. The crates.yaml seed list needs human edits to stay complete. The scraper needs to keep working against the crates.io and GitHub APIs, and the README's insistence on a GITHUB_TOKEN is a reminder that those APIs rate limit. And the cobalt version floor, 0.20.0 with CI on 0.20.4, means a static site generator upgrade is part of the upkeep. The Rust 1.85 floor for building the scraper is the other moving part. None of these is heavy, but none is zero, and they are the reasons a catalog project tends to go quiet: the data pipeline keeps needing attention even when the prose does not change.

Editorial conclusion

Adopt this if you are choosing Rust ML crates and want a single curated index instead of searching crates.io by keyword, or if you want to contribute crates to that index through a pull request against _data/crates.yaml. Do not adopt it as a runtime dependency, as a benchmark, or as a live API: it is a static site generated by a scraper, and the README states that cobalt serve does not rerun the scraper when crates.yaml changes. Before relying on any entry, run just scrape locally with a GITHUB_TOKEN set, inspect _data/crates_generated.yaml for the fields you care about, and check whether the ordering score reflects what you actually need.

Official sources

1. anowell/are-we-learning-yet on GitHub
2. Issues
3. License: CC-BY-4.0
4. Project website
5. README