# simdutf: validating and transcoding Unicode without allocating

> simdutf gives C++ programs validating UTF-8, UTF-16 and UTF-32 transcoding plus Base64 across seven instruction sets. Its functions never allocate and never throw, which pushes buffer sizing onto the caller.

**simdutf/simdutf** — Unicode routines (UTF8, UTF16, UTF32) and Base64: billions of characters per second using SSE2, AVX2, NEON, AVX-512, RISC-V Vector Extension, LoongArch64, POWER. Part of Node.js, WebKit/Safari, Ladybird, Chromium, Cloudflare Workers,  Ghostty and Bun.

- Repository: https://github.com/simdutf/simdutf
- Website: https://simdutf.github.io/simdutf/
- Stars: 2,001 · Forks: 163
- Language: C++
- License: Apache-2.0
- Published: 2026-10-14 · Updated: 2026-10-14 · Language: en
- Canonical page: https://hysenlabs.com/projects/simdutf-simdutf

## Why an unchecked UTF-8 string is a security problem

Not every byte sequence is a valid Unicode string, and UTF-8 is the de facto format on the web, in JSON and in HTML. The README puts two requirements on any code consuming those bytes: validate before use, and make conversion between encodings a validating operation, so an incorrect string is refused instead of transcoded. That second requirement is the unusual one. Plenty of transcoders substitute a replacement character and continue, which suits a text editor and does not suit a server. The audience here is C++ code taking bytes from outside its own control.

The adoption record is the reason to trust the edge cases. Node.js carries it at 18.15, 19.4.0 and 20.0 or better, an even and an odd branch named side by side. Bun, WebKit behind Safari on iOS and macOS, Chromium behind Google Chrome, Microsoft Edge and Brave, Ladybird, Cloudflare workerd, StarRocks, Couchbase, Oracle GraalVM JavaScript, haskell/text and klogg all sit in the usage list. The project description goes one name further, naming Ghostty, which the list itself leaves out.

## One function per encoding pair, four ways to ask for it

The function list is a matrix, not a menu of odds and ends. Latin1 converts to and from UTF-8, UTF-16LE/BE and UTF-32. UTF-8 converts onward to UTF-16LE/BE and UTF-32. UTF-16LE/BE converts to UTF-8 and UTF-32. UTF-32 converts back down to each of the other three encodings. Every cell of that grid arrives in four shapes: with validation, without validation, with error identification, with both.

The pair in the middle is worth understanding. Dropping validation is a promise that the input is already correct, and that promise fails on hostile bytes. Dropping error identification keeps the refusal but hands back a boolean instead of the position of the offending byte, so a client can be told where its own data went wrong. Validation also stands alone: ASCII, UTF-8, UTF-16LE/BE and UTF-32 each validate on their own, with and without error identification, so a program needing only a yes-or-no answer never has to pick a conversion. UTF-16 endianness change appears as its own operation, swapping UTF16-LE for UTF-16-BE and back.

## Size queries remove the guesswork before you allocate

Because nothing allocates, something else has to. The list includes a family of size functions that take a UTF-8 string and return the length its Latin1, UTF-16 or UTF-32 equivalent needs, plus the mirror set starting from UTF-16LE/BE and UTF-32. Character counting for UTF-8 and UTF-16LE/BE sits in the same place, so you can bill a caller in characters without decoding a single code point. The workflow that follows is mechanical: ask for the destination size, allocate the buffer yourself, call the conversion, keep the byte count it returns to you.

The speed claim carries a length condition, and the condition is easy to skim past. The README states that when strings contain hundreds of characters, transcoding can run past a billion characters per second, and it expects that behaviour for Chinese, Japanese, Arabic and the rest of the range including emoji, not only ASCII. That figure is stated for long strings and not claimed for short ones. The compiled footprint is a few hundred kilobytes, and the API is exception-free.

## Base64 decodes forgivingly, with a URL-safe variant

Base64 sits inside the same library rather than beside it, and the project description makes that explicit by listing Base64 next to the Unicode routines. Two directions are on offer, binary to base64 and back, and both take a URL encoding flag, so one call covers the padded standard alphabet and the URL-safe form. The decoder is tied to a named specification: the link in the function list points at WHATWG forgiving-base64-decode in the Infra standard, and the README does not restate the tolerance rules. Anyone depending on exactly which malformed inputs are accepted has to read that specification page rather than trust the word forgiving.

## Building from the root, or from a single amalgamated header

The table of contents names the ways in: Usage (Usage), Usage (CMake), a Single-header version, a Single-header version with limited features, and Packages. The README body published on the project page stops inside the Real-World Usage list, before any of those sections open, so it prints no install command to copy. What can be read directly is the shape of the repository root.

```bash
CMakeLists.txt
cmake/
Makefile.crosscompile
```

CMakeLists.txt with its cmake/ directory is the CMake build the table of contents refers to, and Makefile.crosscompile is named for the job it does. The library ships in two forms, which the single-header section titles hint at:

```bash
include/
src/
singleheader/
simdutf.pc.in
```

The split include/ and src/ tree is the normal build, while singleheader/ amalgamates the library into one header for projects that cannot add build rules. simdutf.pc.in is the templated pkg-config file that travels with them. That arrangement carries a cost the section title Single-header version with limited features makes plain: the amalgamated form is not feature-complete, so check which calls you need before committing to it.

One more pair of root files deserves a second look. The project is described as Apache-2.0, yet LICENSE-APACHE and LICENSE-MIT sit in the root at the same time, and which one binds you is not something the description settles.

## Dispatch across SSE2 to AVX-512, and one name that does not match

At runtime the library picks a kernel for whatever CPU it finds. The project description names SSE2, AVX2, NEON, AVX-512, RISC-V Vector Extension, LoongArch64 and POWER. The README body names the same family in different words: ARM NEON, SSE, AVX, AVX-512, RISC-V Vector Extension, LoongSon and POWER. LoongArch64 in one document and LoongSon in the other, with SSE2 against SSE and AVX2 against AVX, means a CPU filter or build script written from one of them will not match the other.

Two root paths show where the newer targets live. README-RVV.md documents the RISC-V Vector Extension, and riscv/ holds its sources. The table of contents also promises Manual implementation selection for choosing a kernel yourself, a section on SIMDUTF_USE_STATIC_INITIALIZATION, and Compiling without the C++ standard library for freestanding builds. Those are knobs a platform engineer reaches for and a casual user never touches.

## Where the non-allocating design helps and where it bites

The promise is small and predictable: a few hundred kilobytes of code, exception-free and non allocating. That has a direct consequence. Every conversion takes a buffer you own, and the size functions exist so you can size it exactly once. Code expecting a std::string back, or catching an exception on malformed input, has to be reshaped around that. The C API is listed as C11 or better, and the table of contents pairs a C++20 and std::span section with one on compiling without the C++ standard library, so a freestanding core and the C++ conveniences are separate surfaces.

Two cases make it the wrong tool. The speed claim is tied to strings containing hundreds of characters, so checking a short field name or a protocol header gains nothing from vectorised kernels. And nothing in the enumerated list performs normalization, case folding or grapheme cluster segmentation, so a pipeline needing those still carries another dependency. If the target is a Node.js service, check first: the runtime has carried simdutf since 18.15, 19.4.0 or 20.0, so vendoring it yourself buys nothing.

## Tests, fuzzers and command-line tools sit beside the sources

The verification story is visible in the directory listing rather than merely promised. The README claims exhaustive tests and an elaborate fuzzing setup, and both appear as directories:

```bash
tests/
fuzz/
benchmarks/
tools/
man/
doc/
```

benchmarks/ is where the speed numbers come from, tools/ and man/ cover the command-line tools the table of contents advertises, and doc/ together with Doxyfile and doxygen.py is how the API reference gets generated. Governance files sit alongside them: CONTRIBUTING.md, AUTHORS, CONTRIBUTORS, SECURITY.md and AI_USAGE_POLICY.md, the last two worth reading before you automate anything against this repository. On upkeep, the last push landed on 2026-10-05, and three tags went out inside two weeks in September 2026: v9.1.2 on 2026-09-11, v9.2.0 on 2026-09-15 and v9.2.1 on 2026-09-23.

## Conclusion

simdutf is the right dependency when C++ code must treat outside bytes as text and refuse the bad ones, and it is already inside Node.js, WebKit and Chromium for anyone working in those runtimes. It is the wrong pick for text work that needs normalization, case folding or grapheme clusters, none of which appear in its function list, and its speed claim is attached to strings of hundreds of characters, not short fields. Verify which ISA build your fleet actually runs, and whether the CMake build or the limited single-header form fits your link rules.

## FAQ

### What is simdutf used for in Node.js and WebKit?

Node.js carries simdutf at 18.15, 19.4.0 and 20.0 or better, and WebKit uses it as the engine behind Safari on iOS and macOS. Bun, Chromium, Ladybird and Cloudflare workerd are named alongside them, with Chromium serving Google Chrome, Microsoft Edge and Brave.

### Does simdutf allocate memory or throw exceptions?

Its functions are exception-free and non allocating. The caller owns the output buffer, and the size functions exist so you can compute the destination length before allocating one.

### Can simdutf be used at compile time?

With C++23 or newer there is experimental support for using the library at compile time through constexpr. The project calls that support experimental, so treat it as a preview rather than a stable path.

### Which CPU instruction sets does simdutf support?

The project description names SSE2, AVX2, NEON, AVX-512, RISC-V Vector Extension, LoongArch64 and POWER. The README body lists ARM NEON, SSE, AVX, AVX-512, RISC-V Vector Extension, LoongSon and POWER instead, so the two documents use different names for the same family.

### How do I build simdutf from the repository?

The README table of contents promises a Usage (CMake) section, a single-header version and a single-header version with limited features. The root holds CMakeLists.txt, a cmake/ directory and Makefile.crosscompile, and the project homepage is where the usage instructions live.

## Sources

- [License: Apache-2.0](https://github.com/simdutf/simdutf/blob/master/LICENSE)
- [Project website](https://simdutf.github.io/simdutf/)
- [README](https://github.com/simdutf/simdutf/blob/master/README.md)
- [Releases](https://github.com/simdutf/simdutf/releases)
- [simdutf/simdutf on GitHub](https://github.com/simdutf/simdutf)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/simdutf-simdutf