# RE/flex: a Flex++ successor that generates C++ scanners and matches in linear time

> RE/flex is a C++ regular expression library and lexical analyzer generator that extends Flex++ with Unicode support, indentation anchors, lazy quantifiers and error reporting, and it is used in production by the Tiger compiler, by ugrep and as a build dependency for a GCC 3.3 port. The README's own benchmark table is the most honest look at how the alternatives compare.

**Genivia/RE-flex** — Yet another high-performance C++ regex library and lexical analyzer generator like Flex: extends Flex++ with Unicode support, indent/dedent anchors, lazy quantifiers, functions for lex and syntax error reporting and more. Seamlessly integrates with Bison and other parsers.

- Repository: https://github.com/Genivia/RE-flex
- Website: https://www.genivia.com/doc/reflex/html
- Stars: 599 · Forks: 97
- Language: C++
- License: BSD-3-Clause
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/genivia-re-flex

## The benchmark table is the most useful page in the README

RE/flex is described as a high-performance C++ regex library and lexical analyzer generator with Unicode support, and the README leads with three use cases. A RE/flex generated tokenizer is used by the Tiger Compiler, the RE/flex C++ regex engines are used by ugrep, and the library is a build dependency for a 2026 port of the open source PSPSDK to GCC 3.3.6, a version used in the official Sony PlayStation Portable development kit.

The third of those is worth a second look, because it is a build dependency in a toolchain rather than a demo. A lexer generator that has to keep working across a compiler port is a different reliability bar from a script.

The performance table then compares tokenizing a 2 KB representative C source file into 244 tokens, and the numbers are in the same units the author publishes the test set for. The two fastest entries are RE/flex with the fast and noindent options at 8.7 and 8.9 microseconds, and Flex itself with full scanning is at 9.8. Then the field spreads out: the RE/flex full option is at 20.6, PCRE2 with JIT is at 60.8, Hyperscan is at 129, RE2 pre-compiled is at 218, PCRE2 without JIT is at 500, and at the bottom the GNU C POSIX regex is at 4932 and C++11 std::regex at 6490.

The note under the table explains the conditions, which are a 2.9 GHz Intel Core i7 with 16 GB of memory on macOS 12.6.9 with clang 12.0.0 at -O2, over a thousand to ten thousand benchmark runs. It also explains why Hyperscan is in the table at all, since it is disqualified as a scanner because of its all matches reported semantics producing 1915 matches for the test, and because of its event handler requirements.

That is a good sign about the project. A benchmark that explains how it disqualifies a competitor is a benchmark somebody is arguing from a position of having measured it properly.

## The DFA transformation is what makes lazy quantifiers possible

The distinguishing technical claim is one sentence in the README: only RE/flex supports backtrack-free regex lazy matching in linear time using an advanced DFA transformation algorithm, invented by Dr. Robert van Engelen.

The phrase to unpack is backtrack-free. Most regular expression engines that support lazy quantifiers do it by backtracking, and backtracking turns a pattern that looks linear into one that can be exponential on a crafted input. The DFA transformation approach lets the lazy quantifier exist in a matcher that still runs in linear time, which is why the README says the library is faster than Flex and much faster than Boost.Regex, C++11 std::regex, PCRE2 and RE2.

The default is a DFA-based extended regular expression matching in linear time, described as the fastest method, and PCRE2 and Boost.Regex are integrated as optional engines for Perl matching. The note at the end of the features list is explicit that neither is a dependency: they are available if you want Perl semantics, not required if you do not.

The other performance-related details are about what the matcher tracks. The RE/flex matcher tracks line numbers, column numbers and indentations, whereas Lex and Flex do not with the noyylineno option, and the other matchers in the table do not either, except PCRE2 and Boost.Regex when used with RE/flex. The README is upfront that tracking this information incurs some overhead, which is a fair trade and an honest admission.

Auto-detection of UTF-8, UTF-16 and UTF-32 input is another one of those conveniences, along with accepting std::istream, strings and wide strings as input, and supporting file encodings from ISO-8859-1 through ISO-8859-15, CP 1250 through 1258, CP 437, CP 850, CP 858, KOI8, MACROMAN, EBCDIC and custom code pages.

## Flex compatibility is the strategy for adoption

The library is described as a lexical analyzer generator that extends Flex++ with Unicode support, indent and dedent anchors, POSIX regex lazy quantifiers, word boundaries, functions for lex and syntax error reporting, lexer rule execution performance profiling and other new features.

The compatibility with Flex and Bison is stated as the reason to adopt it rather than to fear it. The project is compatible with Flex and Bison to eliminate a learning curve, making a transition from Flex++ frustration-free, it is IEEE POSIX P1003.2 compliant like Lex and Flex but generates C++, and it auto-generates code that integrates with Bison Reentrant, Bison-Bridge, Bison-Locations, the Bison 3.0 C++ interface with the lalr1.cc skeleton, and Bison Complete Symbols.

The specific Bison skeletons matter to anyone who has configured a parser generator by hand, because choosing the wrong one is the usual cause of a scanner and parser that will not talk to each other.

The features list also spells out the ways the generated code is better structured, and the last entry is the one a Flex user notices first. yypush_buffer_state saves the scanner state, meaning the line, column and indentation positions, not just the input buffer. There is no input buffer length limit, where Flex has a 16KB limit. And line() returns the current line, which is exactly what you want for error reporting and is not something Flex gives you.

The repository has the shape of a mature autotools project with a CMake path alongside it. There is a configure.ac and aclocal.m4, Makefile.am and Makefile.in, a bootstrap.sh and a build.sh, config.guess and config.sub, a reflex.pc.in and reflexmin.pc.in for pkg-config, a cmake/ directory, a Makefile.am in the examples, and lex.vim for editor support. There is also a fuzzy/ directory, which suggests fuzz testing, and a unicode/ directory.

## The feature list that matters is about indentation and Unicode

Three entries in the feature list change what you can write in a lexer specification, and they are the ones beyond plain Unicode support.

The indent, nodent and dedent anchors match indentation levels to tokenize, which is a language feature rather than a text-processing feature. A lexer for a language where indentation is significant, Python being the obvious case, needs to know when the indentation level goes up and when it comes down, and a generator that gives you that as a pattern anchor removes the usual hand-rolled counter from the specification.

Word boundary anchors and the freespace mode option to improve the readability of lexer specifications are more modest but still useful, the latter being a whitespace mode that lets a specification be written with the same layout as the language it describes.

The Unicode support is described as complete, including Unicode properties such as the general category escape, and including Unicode identifier matching for C++11, Java, C# and Python source code. That last point is a good indicator of how thorough the work is, because identifier matching is where Unicode handling usually falls apart, since the rules for what may start an identifier and what may continue one differ from letter categories.

The rest of the list covers the generator side. There is %class and %init to customize the generated Lexer classes, %include to modularize lexer specifications, and the ability to combine multiple lexer classes in one application, for example by multiple threads in a thread-safe manner. Configurable Lexer class generation customises the interface for various parsers including Yacc and Bison. And the tool generates Graphviz files to visualise the finite state machines with the Graphviz dot tool, which is one of those debugging affordances that turns a generator from a black box into something you can inspect.

## An extensible hierarchy of matcher engines, and a maintained single author

Two aspects of the project are worth separating, because they carry different risk.

The first is the matcher engine hierarchy, which the README describes as extensible, with a choice of regex engines including the RE/flex engine, PCRE2 and Boost.Regex. The separate RE/flex regex library is what makes C++11 std::regex, PCRE2 and Boost.Regex easier to use for pattern matching on strings, files and streams, in the sense that they get the same interface and the same Unicode handling. So the library is usable as a drop-in general regex API as well as a scanner generator.

The second is the examples. The repository ships a large set of them, and the README names a mini C compiler to Java bytecode, tokenizers for C and C++ source code, for Python source code and for Java source code, and for Lua, JSON, XML and YAML. The examples directory in the repository listing backs that up with files like calc.l and calc.y, cards.cpp, ctokens.l, cow.l, cows.l, csv.l, cdefs.l and a pair of fastsearch files that pair a lexer with a fastsearch.cpp, plus a fastsearch_fsm.cpp for the generated state machine.

The maintenance picture is steady and narrow. The last push was on 2026-09-15, and the release history shows v6.5.0 on 2026-09-15, v6.4.0 on 2026-08-27 and v6.3.0 on 2026-05-17. The repository is not archived and the project is BSD-3-Clause licensed.

The single author is visible in the README, since the DFA transformation algorithm is credited to Dr. Robert van Engelen, who is the author of RE/flex. That is a normal shape for a tool of this kind, and the use in the Tiger compiler, in ugrep and in a compiler toolchain port is evidence that it holds up outside a single person's projects, but it does mean the roadmap is one person's time.

## Conclusion

RE/flex is worth adopting if you are writing a lexer or a C++ tool and want a Flex-shaped workflow without the Flex++ constraints, because the compatibility is the point, the generated code is C++ rather than C, and the lazy quantifier support removes the usual workarounds people write around greedy repetition. It is the wrong tool if you need RE2's guaranteed linear time on patterns with backreferences, since RE/flex achieves linear time through a DFA transformation and not through the same restrictions, and it is the wrong tool if you need a maintained drop-in replacement with no feature work. Before you commit, verify three things yourself: that the optional PCRE2 and Boost.Regex integrations are what you want rather than the default DFA engine, since those are the only paths where Perl-style behaviour is preserved, that your patterns do not rely on backreferences, and that you can live with the project being a single maintainer's tool, since the release history is a steady line of version bumps by one author rather than a team effort.

## FAQ

### What is RE/flex?

It is a high-performance C++ regex library and lexical analyzer generator with Unicode support, extending Flex++ with indent and dedent anchors, POSIX lazy quantifiers, word boundaries and functions for lex and syntax error reporting. It is BSD-3-Clause licensed and is compatible with Flex and Bison while generating C++ rather than C.

### How does RE/flex handle lazy quantifiers?

The README states that only RE/flex supports backtrack-free regex lazy matching in linear time, using an advanced DFA transformation algorithm invented by Dr. Robert van Engelen. By default RE/flex uses DFA-based extended regular expression matching in linear time.

### Which projects use RE/flex?

The README gives three examples: a RE/flex generated tokenizer used by the Tiger Compiler, the RE/flex C++ regex engines used by ugrep, and RE/flex as a build dependency for a 2026 port of the open source PSPSDK to GCC 3.3.6, the version used in the official Sony PlayStation Portable development kit.

### What are the improvements over Flex++?

The README lists several: yypush_buffer_state saves the scanner state including line, column and indentation positions rather than just the input buffer, there is no input buffer length limit where Flex has a 16KB limit, and line() returns the current line for error reporting. There is also full Unicode support including identifier matching for C++11, Java, C# and Python.

### Does RE/flex require PCRE2 or Boost.Regex?

No. The README notes that neither is a dependency. They can be used as optional regex engines in addition to the RE/flex engine, for Perl-style matching, while the default is DFA-based extended regular expression matching in linear time.

## Sources

- [Genivia/RE-flex on GitHub](https://github.com/Genivia/RE-flex)
- [License: BSD-3-Clause](https://github.com/Genivia/RE-flex/blob/master/LICENSE)
- [Project website](https://www.genivia.com/doc/reflex/html)
- [README](https://github.com/Genivia/RE-flex/blob/master/README.md)
- [Releases](https://github.com/Genivia/RE-flex/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/genivia-re-flex