eBible: A Verse-Aligned Parallel Corpus Builder for Low-Resource Bible Translation

What eBible Solves and Who It Serves

NLP researchers working on low-resource languages often struggle to find parallel text. The Bible is one of the most translated books on Earth, and eBible.org hosts hundreds of translations. BibleNLP/ebible curates those into a consistent, machine-readable format. The repository strips out all non-verse content: no introductions, no footnotes, no comments. What remains is the verse text alone, aligned to a canonical verse reference list. That makes it directly usable for training translation models or for cross-lingual verse retrieval. The intended user is a computational linguist or a translation technologist, not a theologian. The project's value is in the normalization: every file follows the same line-per-verse convention, so downstream code can assume a uniform structure. Without this curation, a researcher would have to scrape eBible.org and handle each translation's quirks herself.

Verse-Per-Line Format and the Canonical Reference List

The core data format is deliberately simple. Each Bible is a plain text file named like en-KJV.txt, where the language code and Paratext project name identify the translation. The file has one verse per line. A companion file, vref.txt, lists references in the form GEN 1:1, using USFM 3.0 book abbreviations. You align the two files by iterating through them in parallel: line N of the Bible file corresponds to line N of vref.txt. This design is easy to parse but has a subtle consequence. If a source Bible groups multiple verses into a single range, the combined text appears on the line of the first verse, and the subsequent lines contain the literal token <range>. So a file is not strictly one line per verse in that case. The README gives the example of Gen. 1:1-3, where the first line holds all the text and the next two lines each hold <range>. Downstream tools must handle this token or they will misalign verses. Missing verses are represented by blank lines, which is another detail to account for.

The Regeneration Script: How the Corpus Is Built

The corpus directory contains the extracted files, but the project also ships a Python script, code/python/ebible.py, that rebuilds the entire corpus from scratch. The script has no third-party dependencies, so it runs on a plain Python 3.10 installation. The README states it was tested on 3.10 but should work on later versions. To run it, you create an empty data directory, open a shell in code/python, and execute python ebible.py PATH_TO_DATA_DIRECTORY. The first run creates a directory structure with subdirectories for corpus, downloads, logs, metadata, private_corpus, private_projects, and projects. The script downloads a translations.csv file, fetches zip files for each translation, unpacks them into Paratext projects, and constructs a license file in metadata. The extraction itself is handled by bulk_extract_corpora.py from SIL's silnlp repository, which the script invokes. This is a key architectural choice: the repository does not reinvent extraction logic; it relies on an external tool. That reduces maintenance burden but also means the regeneration process depends on the silnlp project's stability.

Caching, Filtering, and Incremental Updates

The script includes practical features for repeated use. It caches downloaded zip files, suffixed with the UTC date of download, such as grc-tisch--2023-04-05.zip. By default, it reuses cached data for up to 14 days. You can override that with --max_zip_age_days 30, or force a fresh download with --force_download. The --download-only flag lets you fetch everything without building Paratext projects, which is useful for testing network behavior. For debugging, the --filter REGEX option reduces the set of translation IDs to those matching a regular expression. The README shows an example: python ebible.py -f 'grc' PATH_TO_DATA_DIRECTORY selects all IDs starting with 'grc', including grcbrent and grc-tisch. Another example uses 'gup$' to match exactly 'gup' and not 'gupk'. This filtering is a real time-saver when you only want a handful of translations. However, note that the filtering happens after the script has already downloaded translations.csv, so you still need network access for the metadata file.

Licensing Caveats and the NOASSERTION Flag

The repository's license field is NOASSERTION, which is unusual for a software project. The README explains why: the Bibles come from eBible.org, and each translation has its own license. The project claims to have ensured each resource is either in the Public Domain, under a permissive Creative Commons license, or used with permission from the rights holder. The actual license terms are stored in the metadata/licenses folder, with a file name matching the translation ID. This is a practical approach, but it places a burden on the user. Before using any specific translation, you must check its individual license. The README also states that copyright holders can request takedown via pull request or email if they believe a resource is included in error. This means the corpus is not a stable legal artifact; it can change if a rights holder objects. For commercial or redistribution purposes, you should treat each translation as a separate licensed work. The project does not provide legal advice, and the NOASSERTION field is a warning that you cannot assume a uniform license.

Regeneration Workflow and Quality Checks

The README describes a workflow for regenerating the corpus. After running the script, you inspect the newly generated corpus directory. If it looks correct, you replace the checked-in corpus directory with your new one. This is a manual step, not an automated deployment. The project provides a smoke_tests.py script to catch common issues, such as misaligned verses or missing files. The README does not specify what the smoke tests check, but it implies they are a basic sanity check. There is also a TODO note: publishing to Hugging Face is not yet documented. So the pipeline stops at the corpus directory; you must handle any upload yourself. The last push was in February 2023, and the latest release is v1.0.2. That suggests the project is not actively maintained, or at least has been stable for a while. If you need the freshest data from eBible.org, you cannot rely on the checked-in corpus; you must run the script yourself. The caching mechanism helps, but the 14-day default means you will still hit eBible.org periodically.

Limitations and When This Is the Wrong Tool

The most obvious limitation is the lack of non-verse content. If your research needs footnotes, cross-references, or introductory material, eBible is not for you. The verse-per-line format also loses the original structure of the source USFM files, so you cannot reconstruct the original document from the extract. The verse range handling is a potential pitfall: a naive parser that assumes one line per verse will produce incorrect alignments for Bibles that use ranges. Another limitation is the dependence on eBible.org's availability and the translations.csv file. If eBible.org changes its data format or access rules, the script may break. The README notes that the corpus needs regular regeneration because the source data changes over time, but the project itself has not been updated since early 2023. That means the checked-in corpus is a snapshot, not a live resource. For a user who wants a one-time download of parallel text, the corpus is fine. For a user who needs up-to-date translations or who wants to contribute new translations, the script is the only path, and it requires manual intervention.

Alternatives and How They Differ

A direct alternative is the SIL NLP project itself, which provides bulk_extract_corpora.py. That tool works on USFM files directly, without the eBible-specific wrapper. If you already have USFM files from your own sources, you can use SIL's tool to extract verse text, but you would have to manage the verse alignment yourself. Another alternative is the Hugging Face datasets hub, where the eBible corpus is published, but the README does not provide the exact dataset name or upload instructions. The difference is that eBible provides a curated, pre-aligned corpus and a script that automates the download and extraction from eBible.org. Using SIL's tool alone requires you to source the USFM files yourself. A third option is to use the raw eBible.org website, which offers USFM files but in a less uniform structure. The eBible repository saves you the effort of writing a scraper. However, if you need a specific translation that is not in the repository's filter, you might be better off going directly to eBible.org.

Editorial conclusion

Adopt eBible if you need a large, verse-aligned parallel corpus for NLP tasks like machine translation or cross-lingual scripture analysis, and you are comfortable with the mixed licensing (NOASSERTION, with permissive licenses or permission from holders). Do not use it if you need footnotes, introductory material, or a uniform license across all texts. Before adopting, verify the license for each translation in the metadata/licenses folder, and run the smoke_tests.py script on any regenerated corpus to catch alignment issues. The corpus is a snapshot from early 2023; for fresh data, you must run the regeneration script yourself.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. Release notes