SeqIO turns a dataset into a named, reusable, evaluable task

A task is the unit of definition, not a dataset

SeqIO is a library for processing sequential data to be fed into downstream sequence models, and the key abstraction is the Task. The documentation describes it as combining a raw data source, one or more preprocessing steps, a vocabulary to tokenize and detokenize each preprocessed feature, a postprocessor to convert detokenized model outputs into a format for evaluation, and one or more metrics to evaluate with.

That is the whole idea in one sentence. A task is not just where the data comes from, it is the data plus the tokenisation plus the decoding plus the way you measure the answer. When the definition lives in one object, a change to the tokenisation or the metric is a change to the task, not a change to every training script that touches it.

Often a task lines up with a common benchmark. The tutorial uses WMT 19 English-German translation, and the assembled task looks like this:

seqio.TaskRegistry.add(
    "wmt19_ende",
    seqio.TfdsDataSource(tfds_name="wmt19_translate/de-en:1.0.0"),
    preprocessors=[
        functools.partial(
            translate, source_language='en', target_language='de'),
        seqio.preprocessors.tokenize, seqio.preprocessors.append_eos
    ],
    output_features={
        'inputs':
            seqio.Feature(
                seqio.SentencePieceVocabulary('/path/to/inputs/vocab'),
                add_eos=False,
                dtype=tf.int32),
        'targets':
            seqio.Feature(
                seqio.SentencePieceVocabulary('/path/to/targets/vocab'),
                add_eos=True,
                dtype=tf.int32),
    },
    metric_fns=[bleu])

Tasks are usually added to a global registry, which the documentation explains makes them easier to use with model configs and flags, and means each one needs a unique string name. You can also instantiate a seqio.Task directly without registering it, if that suits you better.

Data sources are the pluggable part

Data sources are the first step in the pipeline, providing a way to load raw data in many formats as a tf.data.Dataset. All of them subclass the DataSource base class and are defined in the dataset_providers module of the repository.

The implementations listed cover the common cases. TfdsDataSource loads examples from TensorFlow Datasets, and it is what the WMT example uses, with the dataset name, the config for the language pair excluding context for the open domain setting, and the version number given as "wmt19_translate/de-en:1.0.0". TextLineDataSource loads examples from text files such as tsv. TFExampleDataSource loads tf.train.Example protos from a file such as a TFRecord. FunctionDataSource takes a custom function that returns a tf.data.Dataset, which is the extension point for anything the built-ins do not cover.

The design consequence is that a data source only has to produce a tf.data.Dataset. Whether the bytes come from TFDS, a TSV file, a TFRecord, or a function you wrote, everything downstream is the same. Preprocessors, feature conversion and evaluation do not care where the examples came from.

That is also where the framework's assumption shows. SeqIO assumes that the dataset is a sequence, and modalities such as text or audio are naturally supported. Images are supported as long as they are represented as sequences, with Image GPT given as the example. If your data is a table or a set of independent records with no ordering, this is the wrong abstraction and you will spend effort fighting it.

Preprocessors are just functions from dataset to dataset

Preprocessors are described as functions that transform one tf.data.Dataset into a new tf.data.Dataset, typically involving executing a map over the given dataset, and the preprocessors provided to a Task are executed sequentially.

The tutorial illustrates this with a translate function, which is an ordinary Python function that takes a dataset plus a source and target language and returns a dataset. Inside it there is a nested function that converts one example into a text-to-text pair.

def translate(dataset: tf.data.Dataset,
              source_language: str,
              target_language: str) -> tf.data.Dataset:
  def _translate(ex: Mapping[str, tf.Tensor]) -> Mapping[str, tf.Tensor]:
    """Convert a translation example to a text2text pair.

The example in the docstring shows the shape it is after: a dataset whose examples look like a dict with a language code per field, and the function reduces that to the two fields the model expects.

Two built-in preprocessors appear in the task definition. seqio.preprocessors.tokenize turns text into token ids using the feature's vocabulary, and seqio.preprocessors.append_eos appends the end-of-sentence token. The two-argument version of the first one, seqio.preprocessors.tokenize, is the plain one that works from the feature definition, while the tutorial's own translate step is where the example-specific reshaping happens.

The important warning in the documentation is that specifying options on a Feature does not by itself ensure the transformations are applied. You must also include the necessary preprocessors. That is a real footgun: a task that declares a vocabulary but forgets the tokenize preprocessor will produce text where the model expects ids, and the failure surfaces deep in the model rather than at task definition time.

Output features are a contract, and the vocabulary is the part that bites

The output_features field expects a dictionary mapping string feature names to seqio.Feature objects, and it defines what the task is expected to produce in its output examples. The documentation is precise about the semantics: the output examples may contain additional fields, but they must contain these fields in the specified format, or exceptions will be raised.

Each Feature has three components. A vocabulary, which must subclass seqio.Vocabulary and specifies how the feature can be tokenized and detokenized, with seqio.PassThroughVocabulary available if tokenization is not necessary. An add_eos flag stating whether the feature should end with the vocabulary's EOS token. And the output dtype, which must be a tf.dtypes.DType.

In the example, inputs and targets get separate SentencePieceVocabulary instances at different paths, different add_eos settings, and int32 dtype. The asymmetry is deliberate: the source side has no EOS, the target side does.

The note about the tasks used in T5 is worth keeping in mind. They all produce inputs and targets features to be consumed by a text-to-text model, and for a decoder-only language model only a single feature such as targets would be necessary. Nevertheless, the documentation notes that SeqIO is flexible enough to generate arbitrary output features, which will be converted into model features by the FeatureConverter later in the pipeline.

That separation is the design worth internalising. A task describes the data in its own terms, and a FeatureConverter, defined based on the model architecture, translates that into what the model actually consumes. The same task can therefore feed an encoder-decoder and a decoder-only model without redefining anything.

tf.data under the hood, JAX and PyTorch on the outside

The most important practical property of SeqIO is how it stays framework neutral while using TensorFlow internally. The documentation says it uses tf.data.Dataset to create scalable data pipelines but requires minimal use of TensorFlow, and that with one line of code the returned dataset can be transformed into a numpy iterator, hence it is fully compatible with other frameworks such as JAX or PyTorch.

That is the design decision that makes the library usable outside Google. The pipeline engine gives you input sharding, prefetching, deterministic ordering and parallelism for free, and the numpy escape hatch lets you hand the result to a model written in a different framework.

The cost of that arrangement is the dependency list. The install requirements in setup.py are absl-py, clu, editdistance, jax, jaxlib, numpy, packaging, pyglove, sentencepiece 0.2 or newer, tensorflow-text and tensorflow-datasets, and the package requires Python 3.9 or newer. So a team that wanted to avoid TensorFlow entirely still pulls in tensorflow-text and tensorflow-datasets, and jax and jaxlib as well, which is a heavier install than the README's framing of minimal TensorFlow use suggests.

For context on the lineage, SeqIO is a refactor of the t5.data library, which was used with the Mesh Tensorflow Transformer implementation to train the T5 models. If you have used t5.data, the documentation points you to a dedicated section on the differences, and that section is the one to read before you migrate.

Installing, and the three-step shape of every run

Installation from PyPI is one command:

pip install seqio

or from source, which is the same three steps any Python project needs:

git clone https://github.com/google/seqio.git
cd seqio
pip install -e .

The usage pattern is short enough to state in full. You define a Task and optionally a Mixture, you define or reuse a FeatureConverter based on the model architecture, and you use the top-level function seqio.get_dataset to obtain the tf.data.Dataset instance.

That third step is the one you will call most, and the reason for the registry design becomes clear here. Because a task is registered under a name, the model config and flags can refer to that name rather than to a constructed object, which is what lets a mixture of tasks be assembled declaratively and swapped between experiments without editing code.

The repository is a standard Python package layout with the implementation under seqio/, a test directory, setup.py, a pytest.ini, a MANIFEST.in, a CONTRIBUTING.md, a readthedocs.yaml for the documentation build, and the documentation itself published at seqio.readthedocs.io. The top-level entries also include a .github directory and the LICENSE file.

One thing the tutorial flags is a documentation gap rather than a code one. The link to the tasks used in T5 in the readme is marked as TODO, so the canonical list of T5 tasks is not reachable from that page, and you would need the readthedocs site for it.

Apache 2.0, active, and still carrying TensorFlow's weight

The licence is Apache 2.0, stated in the setup.py header and the LICENSE file, which is the permissive case with an explicit patent grant. Nothing in the packaging restricts commercial use or requires you to publish modifications.

Upkeep is healthy. The last push was on 2026-10-07, the default branch is main, and the setup.py header carries a 2026 copyright for the SeqIO authors, so the file is being maintained rather than left in place. The repository publishes no GitHub releases, so versioning is handled in the package metadata with the version read from a version module inside the package directory, following the pattern of adding that directory to the path so the import succeeds before the package is installed.

The interesting question is not whether SeqIO is maintained but whether it will remain the right dependency. The library is built around tf.data, and the trajectory of the ecosystem is toward frameworks that do not need it. The minimal TensorFlow use and the numpy iterator are the hedge, and they have held, but the dependency list still includes tensorflow-text and tensorflow-datasets.

The audience is a research organisation running multiple sequence models over shared corpora. For a single training run with one dataset, a plain tf.data pipeline or a Hugging Face datasets script is less machinery. SeqIO earns its complexity when you have many tasks, several model architectures reading the same tasks, and metrics that must stay consistent across all of them.

Editorial conclusion

SeqIO suits a research team that trains several sequence models over the same corpora and wants one definition of each task reused across model architectures and frameworks, with metrics attached to the data rather than to the training script. It is the wrong choice for a project that wants a light data loader, since the vocabulary, postprocessor and metric machinery only pays off when you have many tasks to keep consistent, and the wrong choice if you need to avoid TensorFlow entirely, since tf.data remains the pipeline engine. Before adopting it, define one real task end to end with a data source, a vocabulary and a metric, and check the differences from t5.data section if you have used that library, since the migration path is the part most likely to surprise you.

Frequently asked questions

Official sources

1. google/seqio on GitHub
2. Issues
3. License: Apache-2.0
4. README