# GigaDatasets: one loader for LMDB, pickle, file and LeRobot datasets

> GigaDatasets is a Python framework that treats a dataset as a set of components described by config.json files, so images, labels, videos and 2D or 3D geometry can be loaded and evaluated through the same two lines of code. Packaging is done with writer classes, and evaluation comes from a separate FIDEvaluator entry point.

**open-gigaai/giga-datasets** — GigaDatasets: A Unified and Lightweight Framework for Data Processing, Curation, and Visualization

- Repository: https://github.com/open-gigaai/giga-datasets
- Stars: 643 · Forks: 76
- Language: Python
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/open-gigaai-giga-datasets

## A dataset is a config file that points at other config files

The design of GigaDatasets starts with the file format, and the format explains most of the API.

A dataset on disk is a directory holding a config.json at its root, and that file is a small JSON document with two keys:

```
{
    "_class_name": "Dataset",
    "config_paths": [
        "labels/config.json",
        "images/config.json"
    ]
}
```

The _class_name field indicates the class type used for the dataset, which is Dataset in this case. The config_paths list points at the configuration files for the individual components of the dataset, here the labels and the images.

That indirection is the core idea. Rather than one monolithic descriptor for a dataset, each component carries its own config, and a dataset is a composition of components. A component directory in the demo giga_data folder contains its own config.json, a data.pkl holding serialised label data, and for images a data.mdb plus lock.mdb in LMDB format, with the images directory holding a config.json of its own.

The consequence is that a dataset is not a single file. It is a tree of parts, and loading one means reading a manifest and then reading each part it names. That buys you modularity, where you can swap the label backend without touching the image backend, at the cost of a layout that is more involved than a folder of images and a JSON file.

The framework describes itself as unified and lightweight, intended to make handling massive datasets simple, efficient and consistent, with a unified workflow spanning curation and packaging through to loading, evaluation and visualization.

## Loading is one function, and the package is on PyPI

The advertised interface is deliberately small. Loading a dataset is one line, and evaluation is one line:

```bash
pip3 install giga-datasets
```

```python
from giga_datasets import load_dataset

dataset = load_dataset('./getting_started/giga_data')
data_dict = dataset[0]
print('Dataset size:', len(dataset))
print('First item in dataset:', data_dict)
```

The install has a constraint that is easy to miss. GigaDatasets can be installed from PyPI, and it has to be installed in a virtual environment, venv or conda for instance. That is a sensible requirement for a package that pulls in a deep learning stack, and the from-source path is given as a separate option for the latest updates:

```bash
conda create -n giga_datasets python=3.11.10
conda activate giga_datasets
git clone https://github.com/open-gigaai/giga-datasets.git
cd giga-datasets
pip3 install -e .
```

Note the pinned Python version in that environment creation, 3.11.10, and the editable install with pip3 install -e .

The demo data and Jupyter notebooks are in the getting_started directory, with utility scripts in the scripts folder. The load_dataset example above is the getting_started/load_dataset.py file, which the documentation links to directly.

The index behaviour is worth noticing. dataset[0] returns a dictionary, because a sample in a model training or inference pipeline is represented as a dictionary with multiple fields, and len(dataset) gives the size. So the loaded object behaves like a sequence of dictionaries rather than like a table or a dataframe.

## Writers package raw data into LMDB, pickle or plain files

Packaging is the second half of the workflow, and it is where the format choices become concrete.

For an unstructured dataset you use the writer classes, which include PklWriter, FileWriter and LmdbWriter, to turn raw files into a structured format. The example given is a directory of images and matching annotation files:

```
raw_data/
├── 0.json               # Annotation file for image 0
├── 0.png                # Image file 0
├── 1.json               # Annotation file for image 1
├── 1.png                # Image file 1
├── ...
```

And the packaging code pairs a pickle writer for labels with an LMDB writer for images:

```python
image_paths = utils.list_dir(image_dir, recursive=True, exts=['.png', '.jpg', '.jpeg'])
label_writer = PklWriter(os.path.join(save_dir, 'labels'))
image_writer = LmdbWriter(os.path.join(save_dir, 'images'))
for idx in tqdm(range(len(image_paths))):
    label_path = image_paths[idx].replace('.png', '.json')
    label_dict = json.load(open(label_path))
    label_dict['data_index'] = idx
    label_writer.write_dict(label_dict)
    image_writer.write_image(idx, image_paths[idx])
label_writer.write_config()
image_writer.write_config()
label_writer.close()
image_writer.close()
```

Several details carry design intent. The file list is collected recursively with an extension filter, and the label path is derived from the image path by string replacement, which means the naming convention is a requirement rather than a coincidence. A data_index field is written into each label, so the labels stay aligned with the packed ordering even if the source directory listing was not sorted. Each writer emits its own config when write_config is called, which is what makes the component-level config.json files appear.

The two writers are then loaded back and combined into a Dataset, which is saved to the save_dir. Video packaging has its own script with three modes, since the frames are large:

```python
# package video samples in the input directory to the output directory
python getting_started/pack_videos.py --video_dir /path/to/your/raw_videos --save_dir ./giga_videos

# if you want to package videos into lmdb format for better read performance
python getting_started/pack_videos.py --video_dir /path/to/your/raw_videos --save_dir ./giga_videos --pack-lmdb

# if you want to package samples, but not copy the video files and only store the metadata and absolute paths
python getting_started/pack_videos.py --video_dir /path/to/your/raw_videos --save_dir ./giga_videos --only_path
```

The only_path mode is the one to understand for large video corpora: it stores metadata and absolute paths without copying the video files, which trades portability for storage.

## Multi format support and the field extension path

The format coverage is broader than images and labels.

The framework lists File, LMDB, Pickle and LeRobot datasets as supported, with flexible loading, and says it gives unified support for images, videos, 2D and 3D boxes, 2D and 3D points, and other structured data. LeRobot is a specific dependency rather than a generic mention, since the requirements file pins lerobot at version 0.3.2 exactly.

That pin is a constraint worth noticing. An exact version pin on a third party dataset library means the pinned version has to be compatible with your environment, and it will block an upgrade until the pin is changed.

The other dependency list is heavy. Alongside lerobot it includes beautifulsoup4, decord, ftfy, imageio, lmdb, lpips, opencv-python, pytorch-fid, scikit-image, terminaltables, torch, torchvision, transformers and tyro. torch and torchvision and transformers are the bulk of it, and pytorch-fid plus lpips are there for the evaluation path, which is the FIDEvaluator referenced in the introduction as one line of code, eval_results = FIDEvaluator(datasets)(pred_results).

Extending the schema is the third workflow. In model training or inference a sample is often a dictionary with multiple fields, and the framework is designed to be extensible to accommodate new data fields. The example adds canny maps as a new field:

```python
python getting_started/add_new_filed.py --data_dir getting_started/giga_data
```

The script name in that command is spelled add_new_filed.py, without the trailing s on field, which is a small inconsistency worth copying exactly if you run it.

## Packaging, licensing and the state of the repository

The build setup is conventional Python. The pyproject.toml declares setuptools with a build backend of setuptools.build_meta and requires setuptools 61 or newer with wheel, and there is also a setup.py, a setup.cfg and a MANIFEST.in. The setup.py carries a parse_requirements helper that reads requirements.txt and can strip version information, and it handles a few specific cases: lines beginning with -r to include another requirements file, lines beginning with -e for editable installs, and lines containing @git+ for direct references. It also splits on version operators and handles platform specific dependencies with a semicolon.

That helper exists because requirements.txt carries plain package names with almost no version constraints, with the single exception of the lerobot pin. The list is otherwise unpinned: beautifulsoup4, decord, ftfy, imageio, lmdb, lpips, opencv-python, pytorch-fid, scikit-image, terminaltables, torch, torchvision, transformers and tyro. In practice that means your install resolves whatever versions are current at the time, which is convenient for a quick start and a real reproducibility problem for a pipeline that has to run the same way twice.

The project is licensed under the Apache License 2.0, with the LICENSE file in the repository root, and it ships a citation entry for a paper:

```bibtex
@misc{gigaai2025gigadatasets,
    author = {GigaAI},
    title = {GigaDatasets: A Unified and Lightweight Framework for Data Curation, Evaluation and Visualization},
    year = {2025},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/open-gigaai/giga-datasets}}
}
```

The repository has no GitHub releases, and the last push was on 2026-08-25. The top level layout adds docs/ for the documentation sources, a .pre-commit-config.yaml for formatting hooks, a .dockerignore and a CODE_OF_CONDUCT.md. The README also notes that more usage examples and feature documentation will be added in future updates, which matches a project that is documenting a workflow as it grows it.

## Conclusion

GigaDatasets suits a robotics or vision team that keeps its data in several formats and wants a single load path plus a consistent evaluation entry point, especially when the data is large enough that a stray per file read shows up in wall clock time. It does not suit a project whose data is small and simple, since the config.json indirection buys you nothing there, and the requirements list pulls in torch, torchvision, transformers and a pinned lerobot, so the install is heavy regardless of what you use it for. Before you adopt it, check the version constraints in requirements.txt, including the exact lerobot pin, and confirm that the FIDEvaluator you need matches the prediction format your model produces.

## FAQ

### How do I install GigaDatasets?

Run pip3 install giga-datasets inside a virtual environment, since the documentation says it has to be installed in one. For the latest updates, clone the repository and run pip3 install -e . from the project directory, with a conda example using python=3.11.10.

### What dataset formats does GigaDatasets support?

File, LMDB, Pickle and LeRobot datasets, with unified support for images, videos, 2D and 3D boxes, 2D and 3D points and other structured data. The requirements list pins lerobot at version 0.3.2.

### How do I package a raw dataset with GigaDatasets?

Use the writer classes, PklWriter, FileWriter and LmdbWriter, one per component. Each writer writes its data, calls write_config so the component config.json is produced, and is then closed, after which the loaded components are combined into a Dataset and saved.

### How do I evaluate predictions with GigaDatasets?

The evaluation entry point is one line, eval_results = FIDEvaluator(datasets)(pred_results). The requirements include pytorch-fid and lpips, which is what backs the metric implementations.

## Sources

- [Issues](https://github.com/open-gigaai/giga-datasets/issues)
- [License: Apache-2.0](https://github.com/open-gigaai/giga-datasets/blob/main/LICENSE)
- [open-gigaai/giga-datasets on GitHub](https://github.com/open-gigaai/giga-datasets)
- [README](https://github.com/open-gigaai/giga-datasets/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/open-gigaai-giga-datasets