Lhotse: manifest-based multimodal data preparation for speech pipelines
Tools for handling multimodal data in machine learning projects.
At a glance
- What is it?
- Lhotse is a Python library for preparing speech, audio, video, image and text data for PyTorch training. Its core abstraction is the cut, a manifest entry that describes a slice of media and can be mixed, truncated or padded at load time.
- Who is it for?
- Adopt Lhotse if your training data is speech or audio that you need to slice, mix and shuffle across multiple nodes, and you are willing to keep your corpus described in manifests. Do not adopt it if you only need to decode a handful of wav files, or if you cannot commit to regenerating manifests when your recipe changes.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Lhotse addresses: training corpora that do not fit a Dataset class
Speech corpora arrive as directories of audio files with sidecar transcripts, speaker lists, and segmentation boundaries. The naive approach is a PyTorch Dataset that opens a file per item. That works until you need to concatenate two utterances, mix noise into a recording at a specific signal-to-noise ratio, or sample cuts so that each distributed worker sees a different ordering. At that point the Dataset class becomes a small data management system, and most teams write it badly.
Lhotse targets that gap. The README describes it as a Python library aiming to make multimodal data preparation flexible, and places it alongside k2 as part of the next generation Kaldi speech processing library. The intended user is someone training speech or multimodal models in PyTorch who needs reproducible data preparation rather than ad hoc file loading. The library also ships standard data preparation recipes for commonly used corpora, which is the Kaldi inheritance: instead of every lab writing its own LibriSpeech parser, the recipe lives in the repository.
Cuts and manifests: the data model in practice
The central abstraction is the audio cut. A cut is a manifest entry that references a recording and a time interval within it, plus any supervision attached to that interval. Because the cut is a description rather than a copy, operations like mixing, truncation and padding are performed on the fly, which the README frames as a way to minimize the amount of storage required.
The data and metadata are represented in human-readable text manifests and exposed through Python classes. This is the design decision that shapes everything else. Manifests are inspectable with a text editor and diffable in version control, which matters when a preparation bug silently changes training data. The cost is that manifests are a second artifact you must keep in sync with the audio on disk. Move a corpus directory and the paths in the manifest break.
Feature extraction follows the same split. The README states that augmentation and feature extraction are supported both in pre-computed mode, with feature matrices stored on disk optionally using lilcom-compressed backends, and on-the-fly mode that computes transformations upon request. A third option, feature-space cut mixing, sits between them. That three-way choice is the part of Lhotse most likely to confuse newcomers: pre-computed features are fast to iterate on but pin you to a feature configuration, while on-the-fly extraction keeps flexibility and spends CPU on every epoch.
Lhotse Shar and the sequential I/O bet
Lhotse Shar is the storage format the README describes as similar to webdataset: optimized for sequential I/O and modularity. The motivation is that random access into many small audio files is slow on network filesystems and object stores, while reading large shards sequentially is fast. Shar trades the ability to seek to an arbitrary recording for throughput.
That trade-off has consequences the documentation does not resolve for you. A Shar shard is not a browsable directory of wav files, so debugging a single bad example means going through the Lhotse API rather than opening the file. The README points to a dedicated tutorial notebook, examples/04-lhotse-shar.ipynb, which is the right place to look before converting a corpus. If your workflow depends on tools outside Lhotse reading the same audio, Shar is the wrong layer to adopt.
The README also lists dataset blending, efficient on-the-fly bucketing, and data randomization or de-duplication for distributed multi-node training among the main goals. Bucketing matters for sequence models because padding waste scales with length variance, and doing it on the fly avoids materializing a sorted copy of the corpus. De-duplication across nodes is the less discussed feature: without it, multi-node training can show the same example to several workers in one step.
Installing Lhotse and the environment variable that changes what you get
The installation path is short. Lhotse supports Python 3.10 and later, and is on PyPI:
pip install lhotse
For the unreleased version, the README gives:
pip install git+https://github.com/lhotse-speech/lhotse
A development install clones the repository, runs pip install -e '.[dev]', and then pre-commit install to add style checks. Tests run with pytest test and linter checks with pre-commit run. The -e flag means source edits take effect on import without reinstalling, and the [dev] extra pulls in dependencies for tests, documentation builds and notebooks.
The environment variable worth reading twice is LHOTSE_REQUIRE_TORCHAUDIO. The README says that when it is set to anything other than 1, True, true or yes, Lhotse will not check for torchaudio and will remove it from the requirements. The README's own wording is that this disables many functionalities while keeping basic capabilities. So the variable is not a performance knob. It is a decision to run a reduced library, useful in constrained environments where torchaudio cannot be installed, and a trap if it is set unintentionally in a shared container image. The README text on this point is truncated mid-sentence, so the full list of what survives is something to confirm against the documentation site rather than assume.
Where Lhotse is the wrong tool
If your task is to load a folder of wav files and feed tensors to a model, Lhotse adds a manifest layer, a cut abstraction and a storage format you do not need. torchaudio's dataset classes and standard PyTorch Dataset code cover that case with less ceremony.
The second failure mode is corpus drift. Manifests are text files that point at recordings. Nothing in the material suggests Lhotse watches the filesystem for changes. If audio is re-encoded, trimmed or moved without regenerating the manifests, training will fail at load time or, worse, silently read the wrong segment. Teams that treat manifests as generated artifacts and rebuild them in CI are fine. Teams that treat them as checked-in truth are one corpus migration away from confusion.
The third case is non-speech multimodal work where the audio side is incidental. The README lists image and video support and points to examples/05-image-and-video-loading.ipynb, but the library's center of gravity is speech. If your pipeline is primarily vision with a little audio, a general data loading framework will fit better than a speech toolkit with image support bolted on.
How Lhotse differs from torchaudio and webdataset
torchaudio gives you decoding, transforms and dataset helpers. It does not give you a manifest format, a cut abstraction, or recipes for named corpora. The difference in approach is that torchaudio operates on tensors and files, while Lhotse operates on descriptions of data that are resolved later. That is why mixing and padding in Lhotse do not require writing new audio to disk.
webdataset takes the opposite route to the same I/O problem. It packs samples into tar shards and relies on the tar format and a PyTorch IterableDataset. Lhotse Shar is explicitly described as similar to webdataset, so the overlap is real. The distinction is what surrounds the shards: Lhotse pairs Shar with manifests, cuts and corpus recipes, while webdataset leaves metadata conventions to you. If you already have a webdataset pipeline that works, moving to Shar buys you the cut operations and the recipes, not raw throughput.
Against Kaldi itself, the README positions Lhotse as the successor line rather than a replacement for the older data preparation scripts, with a Python-centric design instead of shell and ark files. Icefall is named as the place where k2 and Lhotse meet, and there is a minimal ESPnet plus Lhotse example linked from the README. Both are integration points worth checking before assuming Lhotse fits your existing training stack.
Maintenance cost, releases and licence
The project is active, not archived. The most recent release listed is v1.33.0 from April 2026, following v1.32.2 in January 2026 and v1.32.1 in November 2025. Releases carry names like Himalayan Vulture and Blood Pheasant, which is cosmetic, but the cadence suggests a maintained library rather than a dormant one.
The upgrade cost is concentrated in the manifest schema and the cut API. Because manifests are text, a schema change can require regenerating them from source corpora, which is cheap if your recipes are scripted and expensive if they were built by hand. Pin the version in your training environment and regenerate manifests when you bump it.
Lhotse is licensed under Apache-2.0, which permits commercial use and modification and includes a patent grant. That is a permissive licence, but it is not legal advice: if you redistribute modified Lhotse code or bundle it into a product, read the licence text and your organization's policy on attribution and NOTICE files.
Editorial conclusion
Adopt Lhotse if your training data is speech or audio that you need to slice, mix and shuffle across multiple nodes, and you are willing to keep your corpus described in manifests. Do not adopt it if you only need to decode a handful of wav files, or if you cannot commit to regenerating manifests when your recipe changes. Before committing, verify three things: that torchaudio installs cleanly or that you have set LHOTSE_REQUIRE_TORCHAUDIO deliberately, that your corpus already has a preparation recipe or that you can write one, and that your storage layout tolerates Lhotse Shar's sequential I/O pattern.
Community notes