TensorFlow Text moves tokenisation inside the graph so training and inference agree

Text as a first class citizen in the graph

TensorFlow Text provides a collection of text related classes and ops ready to use with TensorFlow 2.0, and the library can perform the preprocessing regularly required by text-based models, plus other features useful for sequence modelling that are not provided by core TensorFlow.

The description is making text a first-class citizen, and the implementation language recorded for the repository is C++, which is the point. These are native ops, not Python wrappers around string methods, so they run inside the graph, on the same device as the rest of the model, and they are part of the exported graph rather than a preprocessing step that happens before it.

The licence is Apache-2.0, the last push was on 2026-10-15, and the release history is on the TensorFlow cadence, with v2.21.1 from 2026-09-09, v2.21.0 from 2026-08-21 and v2.20.1 from 2026-03-10.

The badges show a PyPI release and a PyPI nightly, a supported Python versions badge, an API reference link, a contributions welcome badge and the licence badge, and there is a commented out block for the continuous integration status table for Linux, macOS and Windows with a note that the badges are not public yet, which is a small reminder that this is a project where the build matrix is not fully visible from the readme.

Why preprocessing belongs in the graph

The Introduction section states the benefit plainly, and it is the argument for the whole library.

The benefit of using these ops in your text preprocessing is that they are done in the TensorFlow graph. You do not need to worry about tokenization in training being different than the tokenization at inference, or managing preprocessing scripts.

That is a real class of bug. The classic failure is a preprocessing script that lowercases, or strips punctuation, or splits on a different whitespace set, and that gets applied to the training data but not to the requests at serving time. The model then works and the production accuracy does not, and the cause is a few lines of Python that nobody looked at because they are not part of the model.

Putting the same ops in the graph removes the possibility rather than the likelihood. The preprocessing becomes a saved artefact with the model, it runs on the same hardware, and it is exercised by the same test.

The library also includes features useful for sequence modelling that core TensorFlow does not provide, which is the other half of the value, since a sequence model needs offsets and a ragged representation of variable length text, and neither is convenient to build on top of plain string tensors.

Unicode: assume UTF-8, transcode if not

The Unicode section is short and it is the prerequisite for everything else.

Most ops expect that the strings are in UTF-8. If you are using a different encoding, you can use the core tensorflow transcode op to transcode into UTF-8, and you can also use the same op to coerce your string to structurally valid UTF-8 if your input could be invalid.

docs = tf.constant([u'Everything not saved will be lost.'.encode('UTF-16-BE'),
                    u'Sad☹'.encode('UTF-16-BE')])
utf8_docs = tf.strings.unicode_transcode(docs, input_encoding='UTF-16-BE',
                                         output_encoding='UTF-8')

The second use is the one that matters more than the first. A string that is not valid UTF-8 is not a slow input, it is a crash somewhere further down, usually in a component that has no idea what it is looking at. Coercing it at the boundary with the transcode op is cheap and it turns a class of late failure into a handled one.

The note that this is the core tensorflow op rather than a TF Text op is worth noticing, because it means the encoding step does not add a dependency on this library. You can and should normalise the encoding of your data before it reaches any of the text ops.

Case folding and NFKC normalisation

The Normalization section deals with the problem of the same word appearing in different forms, which is the first thing that breaks a naive text pipeline.

When dealing with different sources of text, it is important that the same words are recognized to be identical. A common technique for case-insensitive matching in Unicode is case folding, similar to lower-casing, and the note is that case folding internally applies NFKC normalization.

print(text.case_fold_utf8(['Everything not saved will be lost.']))
print(text.normalize_utf8(['Äffin']))
print(text.normalize_utf8(['Äffin'], 'nfkd'))

The library also provides Unicode normalization ops for transforming strings into a canonical representation of characters, with Normalization Form KC being the default.

The difference between the two calls in that snippet is the interesting part and the reason NFKC is the default. NFKC folds compatibility characters into their canonical form, so the precomposed and decomposed spellings of the same character become the same string, and the output shows it, with the first call returning the composed form and the nfkd call returning the decomposed form with the combining diaeresis separated. Case folding on top of that gives you a key that matches regardless of how the text was typed.

The same word recognised as identical is the whole point, and a pipeline that does not do this has a vocabulary split across spellings that you will only notice in a confusion matrix.

Tokenisers, offsets and the RaggedTensor output

Tokenization is the process of breaking up a string into tokens, commonly words, numbers and punctuation, and the section explains the interfaces and the shape of the result.

The main interfaces are Tokenizer and TokenizerWithOffsets, which each have a single method, tokenize and tokenizeWithOffsets respectively. Multiple implementing tokenizers are available, and each of them implements TokenizerWithOffsets, which extends Tokenizer, and that includes an option for getting byte offsets into the original string. This allows the caller to know the bytes in the original string the token was created from.

All of the tokenizers return RaggedTensors with the inner-most dimension of tokens mapping to the original individual strings, and as a result the resulting shape's rank is increased by one. The readme points at the ragged tensor guide for anyone unfamiliar with them.

The WhitespaceTokenizer is the basic one, splitting UTF-8 strings on ICU defined whitespace characters such as space, tab and newline.

tokenizer = text.WhitespaceTokenizer()
tokens = tokenizer.tokenize(['everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])
print(tokens.to_list())

The UnicodeScriptTokenizer splits based on Unicode script boundaries, using the International Components for Unicode UScriptCode values. In practice it is similar to the whitespace tokeniser, with the most apparent difference being that it will split punctuation, which is USCRIPT_COMMON, from language texts such as Latin or Cyrillic, while also separating language texts from each other.

For languages that do not segment words with whitespace, the readme points at the unicode_split op in core TensorFlow, which simply splits by character.

Version pinning is not optional here

The IMPORTANT note at the top of the readme is the operational fact about this library.

When installing TF Text with pip install, please note the version of TensorFlow you are running, as you should specify the corresponding minor version of TF Text, for example for tensorflow==2.3.x use tensorflow_text==2.3.x.

That is because the TF Text ops are C++ kernels compiled against a specific TensorFlow ABI, and the minor version has to match. The readme uses TensorFlow 2.3 as the example, and the current release line here is 2.21, so the same rule applies with a much larger number.

The practical consequence is that upgrading TensorFlow means upgrading TF Text in the same lock step, and that a pip install of tensorflow-text without pinning can quietly install a version built for a different TensorFlow than the one you have.

The documentation section points at tensorflow.org/text for the API docs, the guides and the tutorials, and the readme also lists a set of other text operations, including Wordshape for extracting properties such as case, script and punctuation, and n-grams and a sliding window for sequence modelling.

The repository layout backs the build story. There is a WORKSPACE file for the Bazel build, a tensorflow_text/ directory for the library, third_party/ for the vendored dependencies, oss_scripts/ for the release scripts, docs/ for the guides, and examples/ with a Keras example notebook, keras_example_174.ipynb, which is the quickest way to see the ops used the way a model would use them.

Editorial conclusion

Adopt TensorFlow Text when you are training text models in TensorFlow and the preprocessing has drifted between the training pipeline and the serving path, because the ops run in the graph and remove a class of mismatch. Do not adopt it for a project that does not use TensorFlow, and do not treat the tokenisers as a general purpose NLP toolkit, since they cover normalisation and segmentation rather than modelling. Verify first that the version of TensorFlow Text you install matches the minor version of TensorFlow you run, since the readme calls that out specifically, and that the UnicodeScript tokeniser gives the segmentation your language needs before you commit to it.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. tensorflow/text on GitHub