TensorFlow I/O: file systems and formats TensorFlow does not ship

The gap between what a dataset can read and what exists on disk

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. The full list of supported file systems and file formats is not in the repository readme; the project points to the API reference on tensorflow.org for that, which is a fair arrangement for a list that changes as plugins are added.

The repository description is more concise: dataset, streaming and file system extensions maintained by TensorFlow SIG-IO. The primary language is C++, and the project is Apache-2.0 licensed, with the licence file at the root of the tree and the badge in the readme matching.

The audience is machine learning engineers whose bottleneck is not the model but the input. A training run that reads its data from a remote object store, from a URL, or from a proprietary columnar format needs a way to get that data into a tf.data.Dataset, and the built-in TensorFlow support does not cover everything. TensorFlow I/O exists to fill that gap, and it does so as a set of plugins rather than as a separate pipeline framework.

The important architectural point is that it is an extension, not a fork. Your model code stays ordinary Keras or tf.data code, and the difference is which paths a dataset can open.

Reading MNIST over HTTP without downloading it first

The readme's worked example replaces the data processing part of the standard TensorFlow quickstart with a tensorflow-io call, and the point of the example is not MNIST but the URL handling.

The dataset is created from two URLs, one for images and one for labels:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

The documentation explains why this works where the ordinary tutorial has a download step. The URLs to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call, which is due to the inherent support that tensorflow-io provides for the HTTP and HTTPS file system, and that eliminates the need for downloading and saving datasets on a local directory.

There is a second detail worth knowing. Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, the compressed gzip URLs can be passed to the API call as they are.

After that point the pipeline is standard tf.data. The dataset is shuffled, mapped to convert image data from uint8 to float32, and batched at 32, and the model is a plain Keras sequential stack trained with model.fit.

Installing the wheel, the nightly and the Docker images

The Python package installs with pip:

$ pip install tensorflow-io

There is a nightly channel for people who want the unreleased build:

$ pip install tensorflow-io-nightly

The more important install detail is the extra requirement. To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install, and similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages:

pip install tensorflow-io[tensorflow]

That extra exists because of how the versioning works, and it is the reason the next section matters.

Docker images are published as well, with a stable and a nightly tag:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

There is also an R package, installed from the GitHub repository rather than from a package index, using remotes::install_github("tensorflow/io", subdir = "R-package") once the Python package is in place. That is worth noticing for a specific reason, discussed below.

The setup.py in the repository reads the version and the required TensorFlow version out of tensorflow_io/python/ops/version_ops.py at install time, and it can even print the required version on its own with the --install-require flag, which is how a packaging tool can check the pairing without parsing the readme.

The compatibility table is the real contract

TensorFlow I/O ships compiled extensions that link against TensorFlow's own C++ runtime, and that is why the version pairing is not a suggestion. The readme's table runs from the current release back to 0.1.0, and each row names a TensorFlow compatibility line and a release date.

The top of the table is the important part. Version 0.37.1 is paired with TensorFlow 2.16.x and dated Jul 01, 2024, 0.37.0 goes with 2.16.x from Apr 25, 2024, and 0.36.0 with 2.15.x from Feb 02, 2024. Going back further, 0.27.0 pairs with TensorFlow 2.10.x, and 0.10.0 with 2.0.x.

So the published release line stops at a TensorFlow I/O version compatible with TensorFlow 2.16. A team on a newer TensorFlow has to decide between an older TensorFlow I/O, which may not expose the file systems they need, or building from source.

This is not an accident of packaging, it is the coupling. A plugin that registers a file system implementation into the TensorFlow runtime has to match the internal interfaces of the version it is compiled against, and those interfaces move between TensorFlow releases. The readme's own wording is that it is recommended to install a matching version of TensorFlow I/O according to the table.

For the R package the same constraint shows up in a different place. It is installed from the GitHub repository with remotes, which means you are installing whatever is on that branch, and the readme notes that the R package requires the Python package to have been successfully installed first.

Where the GCS filesystem lives and why it is a separate package

The repository has a directory named tensorflow_io_gcs_filesystem, and the setup.py treats it as a distinct subpackage rather than folding it into the main one. There is a corresponding PyPI name in that subpackages list, tensorflow-io-gcs-filesystem.

The reason for the split is the build. A GCS client is a large native dependency, and folding it into the main wheel would make every installation of tensorflow-io, including installations that never touch cloud storage, carry it. Splitting it lets the common case stay small and lets the cloud case pull in what it needs.

The repository also carries a separate directory for a Google Cloud Storage plugin, and the third_party/ directory alongside it, which is where vendored native dependencies live. The build is Bazel based, with a WORKSPACE file, a BUILD.bazel file, a .bazelversion pin and a configure.sh, and the presence of those tells you that building from source is a real path rather than a theoretical one, even though it is much heavier than the pip install.

The docs/ directory adds tutorials and a development document, and the AVRO_TENSOR_DATASET.md file at the root documents one of the file format plugins, which gives a hint of the kind of niche formats the project takes on that have no other home.

Benchmarks, CI and the shape of the community around it

The project documents its performance benchmarking on github-pages, at the bench path of the project site, and the benchmark job is triggered on every commit to the master branch, which facilitates tracking performance with respect to commits. That is a useful setup for a project whose changes are mostly in native code, where a regression can appear in a file system plugin without touching any Python.

The CI story is table driven. The readme has a build status table with rows for each platform configuration, and the badges point at per-build status endpoints, which is the shape a matrix of platform and language combinations produces. The repository has a .kokorun/ directory, and Kokoro is the CI system that directory is named for, so the build matrix is defined there rather than in the GitHub Actions workflows used elsewhere in the ecosystem.

The governance point is that TensorFlow I/O is described as a community led open source project which depends on public contributions, bug fixes and documentation, with contribution guidelines, a development document and a tutorials directory linked from the readme. It is maintained under the TensorFlow SIG-IO special interest group, which is why the project is a separate repository with its own release cadence rather than a folder inside the TensorFlow repository.

The last push to the repository was on 2026-06-25 and the repository is not archived, so work continues even though the newest published release is 0.37.1 from 2024-07-01.

Editorial conclusion

Adopt TensorFlow I/O if your tf.data pipeline has to read from a source TensorFlow does not handle natively, most often an HTTP or HTTPS endpoint, because the HTTP and HTTPS file system lets you pass a URL straight into a dataset constructor and skip the download and unzip steps entirely. Do not adopt it if you are starting a new project on a current TensorFlow release, because the compatibility table stops at TensorFlow I/O 0.37.1 against TensorFlow 2.16.x, and the underlying reason for that pin is the same one that makes the library fragile. Before you install, check the version pairing table for your TensorFlow version, use the tensorflow-io[tensorflow] extra so pip resolves a compatible TensorFlow, and confirm that the file system you need is on the supported list, since the project docs point to the API reference rather than listing them in the readme.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. tensorflow/io on GitHub