RosettaCodeData mirrors the Rosetta Code sample library as a git repository

A git mirror of a wiki, and the reason it exists

The repository describes its contents in one sentence. It contains almost all of the code samples available on rosettacode.org, organised by Language and Task.

That word almost carries the article. The upstream site is a MediaWiki, and a wiki is a reasonable place to publish many small examples of many tasks, but it is a poor place to consume them programmatically. There is no bulk export that preserves the language and task structure, every sample lives inside a page revision, and the pages mix prose with code in a formatting that is not uniform.

Turning that into a git repository changes what you can do with it. The top-level entries show the shape: a Lang/ directory, a Task/ directory, a Conf/ directory, a bin/ directory, plus a Makefile, a popular-tasks file and a .rc directory. The directory names are the same organisation as the wiki, but as a filesystem you can grep, script against, diff and check out at a particular revision.

The trade is that the content is no longer the live site. The README says the data is created by a Perl program called rosettacode, so this is a generated mirror with a sync step rather than a hand-curated collection, and a change on the wiki reaches this repository only when somebody runs the sync.

The primary language of the repository is recorded as REXX, which is an artefact of the corpus rather than of the tooling: the largest body of code in the mirror is REXX samples, because the upstream site collected a great many of them, and the repository metadata infers the dominant language from the file contents. The tooling itself is Perl.

The licence is not stated in the repository metadata, and the README does not discuss licensing either. For a corpus of code samples from hundreds of contributors that is worth thinking about before you build on it.

Cloning it, and the shallow clone the README insists on

The README is unusually blunt about the first thing you will hit, which is the size of the repository.

All of the data is in this repository, so you can just run:

git clone https://github.com/acmeism/RosettaCodeData

And then immediately: However... It's a lot of data!

If you just want the latest data, the quickest thing to do is:

git clone https://github.com/acmeism/RosettaCodeData --single-branch --depth=1

The difference between those two commands is the difference between a corpus you can work with and a clone that takes a coffee break. A depth-1 clone gives you one commit with all the files, and --single-branch keeps the other branches out of the pack. The full history of a repository that has been re-synced regularly is a large number of near-identical trees, which is the worst case for git's storage model.

That advice is not incidental to the project design. The data is regenerated and pushed on a schedule, so the history is a series of snapshots rather than a record of individual sample edits. If you want to diff how a particular sample changed between two syncs, the full history is what lets you do that, and the shallow clone deliberately forfeits it.

So the choice depends on what you are doing. Exploring the corpus or running analysis on the current state, take the shallow clone. Tracking how a language's samples evolved over the years, take the full one and accept the size.

The repository also has a .gitignore, a .nrepl-port file and a .rc directory. The .nrepl-port file is a Clojure tooling artefact, which points at the .clj-kondo and .lsp entries in the Makefile's clean list and suggests the project is linted and developed with Clojure-aware tooling alongside the Perl pipeline.

The sync pipeline is a Perl program and a make target

The data content is created by a Perl program called rosettacode, which you install with:

cpanm RosettaCode

And rebuild the data with:

make build

The Makefile shows how thin that layer is. It uses the Makes Makefile framework, cloning the framework itself into a .cache/makes directory on first use and including its init, ys, clean and shell modules. The build target depends on $(YS) and then runs bin/rcd-sync under a timer.

Two details in the Makefile are worth reading. The first is that the build target calls bin/rcd-sync rather than the rosettacode program directly, which means the actual synchronisation entry point is a script in the repository's own bin directory, distinct from the installed CPAN distribution. The second is that PATH is overridden with the repository's bin directory prepended and exported, so the tools in bin/ shadow any installed versions. That is a common and sensible pattern for a project whose tooling is versioned with its data.

The clean list names the artefacts a build leaves behind: Meta/, rosettacode.log and rosettacode-errors.log. The second one is the more interesting, because the README's to-do list includes addressing errors reported in rosettacode.log after running make build. The pipeline logs its own failures, and the author is treating the log as the backlog for data quality work.

The Makefile also lists Cache, .clj-kondo and .lsp as dist-clean targets, which are the fetched page cache and the Clojure language server configuration. The cache is what the 'only fetch cache pages since last pushed data update' item in the to-do list refers to: the pipeline currently refetches what it needs rather than reusing a stored copy, and the cache directory is the structure that a partial refetch would use.

The bin directory is seven small reporting tools

The README describes the bin directory as containing various tools for working with the data, and lists seven of them. They are worth reading as a set, because they describe the shape of the corpus better than any prose could.

Two of them talk to the live site. rcd-api-list-all-langs lists all the programming language names directly from rosettacode.org, and rcd-api-list-all-tasks lists all the programming and draft task names directly from rosettacode.org. The word directly matters: these do not read the local data, they query the upstream API, so they can be used to see what exists upstream as opposed to what has been synced here.

Three of them count samples. rcd-samples-per-lang shows the number of code samples per language, rcd-samples-per-task shows the number of code samples per task, and rcd-tasks-per-lang shows the number of tasks with code samples per language. A fourth, rcd-langs-per-task, shows the number of languages with code samples per task.

The fifth, rcd-new-langs, lists the RosettaCode languages not yet added to Conf. Conf is the directory holding the configuration, and lang.yaml inside it is the file the to-do list mentions as containing hundreds of guessed file extensions.

The counts are the interesting part. A task with samples in forty languages and a language with samples for two hundred tasks are very different kinds of coverage, and the fact that the toolset measures samples per language, samples per task, tasks per language and languages per task separately tells you that the corpus is irregular in both directions. A task can have many languages and few samples each; a language can have many tasks and thin coverage of each.

That is the shape of Rosetta Code itself, and the tools exist so that a user can find out where the corpus is deep before depending on it. The README's final note, that this project is not a perfect representation of RosettaCode yet, is the honest summary of all of it.

Where RosettaCodeData is the wrong tool

Three limits, and the first is the one that decides whether the corpus is usable for you.

The first is coverage. The corpus is deep for some tasks and thin for others, and it is thin for whole languages in some areas. The README's own tools for counting samples per language and per task exist because the answer varies, and if your requirement is a task implemented in a specific set of languages, you have to check whether the samples are there rather than assume it. A corpus of a thousand tasks where your task has four implementations is not a useful corpus for that task.

The second is that the data is a generated snapshot and the pipeline is known to be imperfect. The README says explicitly that this project is not a perfect representation of RosettaCode yet, that it has a few unicode issues, and that it also has to deal with various formatting mistakes in the mediawiki source pages. A unicode issue in a sample is not cosmetic if you are parsing those files to learn how a language handles a particular character, and the to-do list confirms that support for names with non-ascii characters is still to be done, alongside correcting the hundreds of guessed file extensions in Conf/lang.yaml.

The third is that the mirror lags the site, and there is no partial refresh. The to-do list includes the ability to only fetch cache pages since last pushed data update, which means that today the sync refetches rather than incrementally updating, and that your clone can differ from the live site in ways that are hard to reason about when you are chasing a bug and the sample on the wiki looks correct.

The alternative approach is to read the site directly, or to use a curated dataset. The difference in approach is that RosettaCodeData trades freshness and completeness for structure and offline access: you get the whole corpus as a tree of files that your tools can walk, at the cost of accepting the pipeline's known defects and a snapshot that may be behind. If you need one task solved in three languages, the site is faster and more accurate. If you want to analyse the corpus itself, to count, to diff, to extract patterns across hundreds of tasks and dozens of languages, nothing else offers it offline in this shape.

One more constraint deserves a mention. The samples are written by many different people under many different licences, and the repository states no licence of its own. Copying code out of it into a product raises questions the README does not answer, and for a corpus whose value is that it contains other people's code, that is a real consideration rather than a formality.

Editorial conclusion

Adopt RosettaCodeData if you need a large, offline, structured corpus of the same task implemented in many languages, because a git clone gives you the whole thing in a form you can grep and script against, which the wiki cannot offer. Do not adopt it as an authoritative reference, since the README states the project is not a perfect representation of RosettaCode and lists open unicode and formatting issues that the pipeline does not yet handle. Verify first that the clone you use is the shallow one, because the full history is described as a lot of data, that the samples you depend on are present for the language and task you care about, since the tool list includes several counting commands precisely because coverage is uneven across languages, and that you do not treat the mirrored code as licensed for reuse, since the repository lists no licence and the samples come from many different contributors.

Frequently asked questions

Official sources

1. acmeism/RosettaCodeData on GitHub
2. Issues
3. README