LitData streams datasets in chunks instead of per file

The problem is a loader that cannot keep up, not a slow kernel

LitData opens with a claim worth reading precisely: speeding up model training involves more than kernel tuning, and data loading frequently slows training down because datasets are too large to fit on disk, consist of millions of small files, or stream slowly from the cloud. That diagnosis sets the scope. The project is not trying to make your model faster, it is trying to make the step that feeds the model stop being the bottleneck. The tool provides preprocessing and optimisation that turn a dataset into a format that streams efficiently from any cloud or local source, plus a map operator for distributed data processing before that optimisation step. The stated payoff is that pipelines get faster and cloud-agnostic, with an improvement in training throughput of up to 20x. The documentation attaches a benchmark section to that claim, which is where you should go to judge it for your own data rather than taking the headline figure. The audience is teams where the data pipeline, not the model, is what is slowing the training loop.

LitData and torchdata sit at different layers

The README is unusually direct about the relationship with torchdata: different layers, not competitors. LitData is a streaming layer with its own on-disk format, plus a no-conversion path for files you already have, whereas torchdata is a toolkit of dataloading primitives with no format of its own. The comparison table makes the split concrete. On storage, LitData has its own chunked binary format via `optimize()` and also reads Parquet, MDS and raw files, while torchdata has none and asks you to bring your own. On remote data, LitData does chunk-level streaming from S3, GCS, Azure, R2 and the HF Hub with async batched downloads, prefetch, a local cache and retries, while torchdata does per-file reads through `FileLister` and `FileReader` on fsspec and smart_open, with no chunk cache or prefetch pipeline. The main APIs differ too: `StreamingDataset`, `StreamingDataLoader` and `CombinedStreamingDataset` against `torchdata.nodes` iterators you chain yourself plus `StatefulDataLoader`. Both offer mid-epoch resume through a `state_dict()` and `load_state_dict()` pair.

Two paths, one of which skips conversion entirely

The workflow choice in the quick start is between two options, and the framing matters because the second one needs no preparation step at all. Option 1 is `StreamingRawDataset`, which the README describes as streaming existing files as-is with no optimize step. You point LitData at a folder of images, audio, text or any files, local or in the cloud, and train with a normal PyTorch `DataLoader`; downloads are fully asynchronous and batched. Option 2 is the `optimize()` path, where the dataset is rewritten into the chunked binary format and then streamed. The third possibility is the `map()` operator, which handles preprocessing as a distributed job that writes a new dataset before training starts. The distinction between the two transform approaches in the comparison table is the one to sit with: LitData's `map()` and `optimize()` run preprocessing jobs before training and write a new dataset, distributed across machines, whereas torchdata's `ParallelMapper` transforms samples in the live pipeline and re-runs every epoch. Same data, different bill.

Installing it, and the version floor that bites

The install is a pip install, and the version support policy is stated right next to it.

pip install litdata

LitData supports the two most recent PyTorch minor releases, currently 2.14 and 2.13, and requires PyTorch 2.4 or newer, with CI exercising all three. Those versions are declared in `.github/torch-support.json`, which drives the CI matrix, so the support window is a file you can read rather than a promise you have to trust. There is an advanced install with extras, where on Linux and macOS the extras include an optional `uvloop` for a faster asyncio event loop used by `StreamingRawDataset`, with stdlib asyncio as the fallback when it is not installed.

pip install 'litdata[extras]'

The project also ships an agent skill for coding assistants, installed through the skills CLI and sourced from a directory in the repository.

npx skills add Lightning-AI/litData

The runtime dependency list is short and gives you a picture of what the tool actually needs: torch with a minimum of 2.4.0, torchvision, lightning-utilities, filelock, numpy, boto3, requests, urllib3 with a floor of 1.26, tifffile and obstore. The last two are the interesting ones for a storage project, since obstore is a Rust-backed object store client and tifffile handles the image formats that come out of the multimodal examples.

Where the chunk format pays off and where it does not

The value of a chunked format is entirely a function of your data's shape, so it is worth being explicit about both sides. When a dataset is millions of small files, a per-file read is a per-file round trip, and a chunk format turns that into a few large sequential reads that can be prefetched, batched and cached locally. When a dataset is one very large file, the format has less to add, because you were already streaming sequentially. When the data is small enough to sit in RAM, none of this matters. There is also a cost to the conversion step. `optimize()` writes a new dataset, so you now maintain two copies of the data and a conversion step in your pipeline, and if the source data changes you have to know that the derived copy is stale. The `map()` operator has the same property for preprocessing: because it runs before training and writes out results, the preprocessing is done once rather than every epoch, which is the point, but it also means a change to the preprocessing logic requires a re-run of that job. The README is direct that the two projects compose rather than compete: a `StreamingDataset` is an `IterableDataset`, so `torchdata.nodes.IterableWrapper` can pull straight from it.

Pausing, resuming and the rest of the feature list

The features that matter for long runs are the operational ones. Data streaming can be paused and resumed, remote data can be used without local loading at all, and raw files and large cloud datasets can both be streamed. The resume story is a `state_dict()` and `load_state_dict()` pair on `StreamingDataLoader`, which is the mid-epoch resume mechanism, so a run that is interrupted does not have to start over. Shuffling, caching and prefetching are handled by the library rather than left to you, and the README even gives a rule of thumb for choosing between the two projects: reach for LitData in most cases because it handles streaming, caching, shuffling and resume for you, and reach for torchdata when you want bare-bones primitives and are happy to build the loading pipeline yourself. The examples directory gives you the shapes to look at, with getting started, modality, multi_modal, sine_function_model_prediction and temporal_arrays, and the repository also carries a `benchmarks/` directory and a `simulator/` directory for the data preparation jobs. The licence is Apache 2.0, so embedding it in a commercial training stack carries no copyleft obligation, and the most recent release listed is v0.2.76 with the last push on 2026-10-15.

Editorial conclusion

Adopt LitData when your training data is too large to keep on disk, is made of many small files, or streams slowly from cloud storage, and you want chunk-level prefetch, caching and mid-epoch resume without writing a loader yourself. Skip it if your data already fits in memory and loads fast, since the conversion step then buys nothing, and skip it if you are on PyTorch older than 2.4. Before you start, measure where the loader actually spends its time and check that your storage backend is one of the supported ones, since chunk-level streaming is the whole mechanism.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Lightning-AI/litData on GitHub
4. README
5. Releases