TimeCapsuleLLM: Training a Language Model on 1800-1875 London Text Alone

The Modern Bias Problem TimeCapsuleLLM Targets

Contemporary language models are trained on text that spans centuries and are then aligned toward present-day usage. Ask one to write as an 1830s Londoner and you typically get modern syntax dressed in archaic vocabulary. TimeCapsuleLLM takes the opposite route: it trains a model from scratch on a corpus restricted to a place and a date range, so the training signal itself contains no post-period text. The README describes the goal as emulating "the voice, vocabulary, and worldview of the era." The intended audience is narrow. This is for researchers and hobbyists studying historical language modeling, digital humanities practitioners who want a model whose errors are period-plausible, and engineers curious about what a small model learns when the corpus is deliberately bounded. It is not aimed at anyone who needs fluent, factually reliable output.

Architecture Lineage: nanoGPT, Phi 1.5 and LlamaForCausalLM

The project did not invent a new architecture. According to the README, v0 and v0.5 build on nanoGPT by Andrej Karpathy, and the README states that the core training scripts and model architecture are his work. v1 moved to Microsoft's Phi 1.5, and v2 is built on LlamaForCausalLM. That lineage matters for anyone evaluating the repository: the interesting engineering is not in the model definition but in the data pipeline and the training loop configuration. The repository is organized by version, with directories such as london_1800_1850_v0 and london_1800_1875_v0.5, each holding its own training artifacts and sample outputs. The dataset itself is published separately on Hugging Face as Historic London English (1800-1875), credited to Hayk Grigorian and Hamed Yaghoobian, and the README provides a BibTeX entry for citing it in academic work.

What the Documented Outputs Actually Look Like

The README does not hide the quality ceiling. For v0, the stated prompt "Who art Henry?" produced "I know that man, I have did not a black, the storm." The README attributes the incoherence to roughly 187MB of training data and notes that modern concepts are absent while vocabulary is mostly era-accurate. v0.5-london is described as producing Victorian writing style with proper punctuation and mostly grammatical sentences, but with a high factual hallucination rate and OCR noise such as "Digitized by Google" appearing in outputs. v1-london is the first version the README credits with connecting a real historical event to a figure in the dataset, shown through an 1834 prompt about protest and petition. v2mini-eval1-london was trained on a 15GB sample of v2's 90GB dataset to only 10K steps, and the README reports a tokenization issue that fragments words, visible in the spaced-out rendering of a Charles Dickens prompt. That fragmentation is the most concrete defect documented in the repository.

Running TimeCapsuleLLM: What the Repository Gives You

The README does not include an installation walkthrough, a requirements file listing, or a documented command for launching training. What it does give is the data path and the model path. Training data for the London corpus is referenced through the Hugging Face dataset postgrammar/london-llm-1800, and released checkpoints are collected under the Hugging Face collection haykgrigorian/timecapsulellm-1800-1875-london. Because v0 and v0.5 derive from nanoGPT and v2 from LlamaForCausalLM, the practical route is to follow those upstream projects' training entry points and substitute the period corpus. The repository layout, with one directory per version, implies that each version carries its own scripts and configuration rather than a single unified trainer. Anyone expecting a pip install and a run command will be disappointed; this is a research repository where the per-version directories are the interface.

Where TimeCapsuleLLM Breaks Down

Three limitations are visible in the material. First, corpus size caps capability: the README ties v0's incoherence directly to a 187MB training set, and even the larger v2 pipeline is sampled down to 15GB for the mini evaluation. Second, OCR contamination is a data quality problem the project has not solved, since v0.5 outputs still contain digitization boilerplate. Third, the v2mini-eval1 tokenization defect shows that a tokenizer mismatch can render otherwise learned style nearly unreadable. There is also a scope constraint: the released work covers London between 1800 and 1875. If your question concerns Paris in 1780 or New York in 1900, this project has no corpus for it, and the README does not describe a general-purpose pipeline for swapping in arbitrary periods. Treating TimeCapsuleLLM as a historical oracle would be a misuse; the README itself reports high factual hallucination in v0.5.

How This Differs From Retrieval-Augmented Historical Chat

The obvious alternative is to take a modern instruct model and constrain it with retrieval over period documents plus a system prompt. That approach keeps grammatical fluency and factual grounding, because the base model already knows English syntax and the retrieval layer supplies period text at inference time. TimeCapsuleLLM makes the opposite trade: it accepts degraded fluency in exchange for a model whose parameters contain no post-1875 text at all, which is the property that matters if your research question is about what a model learns from a bounded corpus rather than about producing good prose. A retrieval system cannot make that claim, because the base weights still encode modern usage. The two are not substitutes. One is a writing aid with citations; the other is an experimental subject.

Maintenance, Licensing and Upgrade Surface

The code is MIT licensed, which permits commercial reuse and modification of the training scripts. That licence does not extend to the training corpora. The README points to a Hugging Face dataset and provides a citation requirement for academic use, so the provenance and reuse terms of the underlying scanned texts need separate checking before any redistribution. On maintenance, the release cadence is visible in the version tags: v2mini-eval2 in December 2025, v2 in January 2026, and v3mini-eval1 in July 2026. Each version changes the base architecture, from nanoGPT to Phi 1.5 to LlamaForCausalLM, which means upgrading is not a patch but a migration to a different model family with its own tokenizer and training script. Budget for that, and pin the version directory you depend on rather than tracking main.

Editorial conclusion

Adopt TimeCapsuleLLM if you are researching historical language modeling or want a reproducible starting point for period-restricted training, and you accept that the released checkpoints hallucinate and carry OCR artifacts. Do not adopt it if you need a production text generator or a reliable historical question-answering system. Before committing, read the v2mini-eval1 model card on Hugging Face to confirm the tokenization defect described in the README, and verify the licence terms of the underlying corpora, which the MIT licence on the code does not cover.

Official sources

1. haykgrigo3/TimeCapsuleLLM on GitHub
2. Issues
3. License: MIT
4. README
5. Releases