simple: a SQLite fts5 tokenizer for Chinese and pinyin

The problem a Chinese tokenizer actually has

simple is a SQLite3 fts5 extension supporting Chinese and pinyin. The README states that it fully implements solution four from an article on the polyphonic character problem in full-text search on WeChat mobile, and that it supports Chinese and pinyin search simply and efficiently.

The problem worth naming is polyphony. Chinese has many characters that are pronounced differently depending on context, and a search system that indexes one reading per character gets the wrong answer for a large share of real queries. A user who types pinyin for a character they have in mind expects to find the character, and the index has to have recorded that reading to make the match possible. The reverse is also true, a pinyin query has to be expanded back into the characters that could carry that reading.

So a Chinese fts5 tokenizer has to do two things at once, segment the text into units that can be matched, and keep a mapping between characters and readings so that a query in one script can find text in the other. simple does both, and the pinyin mapping is data rather than code, which is the detail that matters for the rest of this article.

The project is C++14, described in the topics as a tokenizer for fts5 and sqlite3, with the default branch on master. The release history is recent and tight: v0.6.1 on 2026-02-15, v0.7.0 on 2026-02-20 and v0.7.1 on 2026-02-23. The last push was on 2026-05-17.

Loading the extension and running the first query

The usage path is the standard one for a SQLite extension, and the README starts by checking that the SQLite build you have supports fts5 at all:

select fts5(?1);

After that you load the extension and create a virtual table with the simple tokenizer, and the session in the README shows the whole shape of the thing:

sqlite> .load libsimple
sqlite> CREATE VIRTUAL TABLE t1 USING fts5(text, tokenize = 'simple');
sqlite> INSERT INTO t1 VALUES ('中华人民共和国国歌');
sqlite> select simple_highlight(t1, 0, '[', ']') as text from t1 where text match simple_query('中华国歌');
[中华]人民共和[国国歌]
sqlite> select simple_highlight(t1, 0, '[', ']') as text from t1 where text match jieba_query('中华国歌');
[中华]人民共和国[国歌]

Those two results are the whole argument for the extension, and they are worth reading closely. The indexed text is the national anthem of the People's Republic of China, and the query is a shortened form, 中华国歌. The simple tokenizer splits it its own way and matches 中华 and 国国歌, which highlights as a strange doubled word. The jieba query segments the same query as 中华 and 国歌 and matches the real phrase, so the highlight is [中华]人民共和国[国歌].

The second query in the README is the inverse case, and it shows the polyphony handling from the other side:

sqlite> select simple_highlight(t1, 0, '[', ']') as text from t1 where text match simple_query('中华人民共和国');
[中华人民共和国国]歌
sqlite> select simple_highlight(t1, 0, '[', ']') as text from t1 where text match jieba_query('中华人民共和国');
[中华人民共和国]国歌

Here the simple tokenizer matches a longer run and attributes 国 to the previous word, while the jieba segmentation puts the boundary where a reader would put it. Neither is wrong, but the second is what someone searching for that phrase expects to see, and that difference is the feature.

Five functions, and one switch for jieba

The feature list in the README is the API surface, and it is a small, deliberate one.

The tokenizer supports Chinese and pinyin segmentation, with a switch to control whether pinyin support is needed. simple_query() assembles the match query automatically, so the user does not have to learn fts5 query syntax. That single function is often the reason to adopt the extension at all, because fts5's MATCH syntax is expressive and unfriendly, and a typo in it is an error at query time rather than an empty result.

Three functions then handle presentation. simple_highlight() highlights the matched words and works like SQLite's own highlight, except that it groups consecutive matched words into one run, which the README argues is what users actually need. simple_highlight_pos() returns the positions of the matches and leaves the decision about how to use them to the caller. simple_snippet() cuts out the matched region, and works like SQLite's snippet with the same grouping of consecutive matches.

The jieba path is layered on top. jieba_query() gives the effect of jieba word segmentation, and the point is that it does this with the index unchanged, so a more accurate match is available without reindexing the corpus. It can be switched off at build time with -DSIMPLE_WITH_JIEBA=OFF. jieba_dict() sets the dictionary directory, has to be called once, and has to be called before jieba_query() is used.

And pinyin_dict() accepts a custom pinyin file path, switching the mapping immediately on success, and returning an error without changing the current mapping if the file format is wrong. The error behaviour matters more than it sounds, because a half-applied mapping would silently change what your index means.

The pinyin table is data, and changing it invalidates your index

By default the extension uses the pinyin table built into the shared object, contrib/pinyin.txt. If you want your own table, you set it before querying:

select pinyin_dict('/path/to/pinyin.txt');

The file format is one entry per line, with the Unicode code point and the readings separated by commas, for example:

U+3007: líng,yuán,xīng

The README notes that a trailing comment is supported, with a space before it, and that the same format is used as the default file. The entries carry several readings for one character, which is exactly the polyphony data that the tokenizer needs.

Two warnings come with it, and both are operationally important. It is advised to call pinyin_dict() once before building the index and before querying. And if you replace the pinyin mapping, the pinyin tokens already in an existing index are not rebuilt automatically, so you have to rebuild the index according to your own policy.

That second warning is the one to design around. A custom pinyin table is a schema change, not a configuration change, and if your application changes it at runtime the search results will be inconsistent until you reindex. The same applies to the jieba dictionary, though the jieba path does not change the index at all, which is the main argument for it.

Building it locally, or for iOS, or for OpenHarmony

The build instructions cover three targets, and the first one is a single script. A compiler supporting c++14 or newer is enough, and running ./build-and-run in the root directory compiles everything needed and runs the tests, with the build output in the output directory.

The manual route is plain CMake:

mkdir build; cd build
cmake ..
make -j 12
make install

For iOS there is a script, ./build-ios.sh, and a dynamic variant, build-ios-dynamic.sh. For OpenHarmony the instructions are longer, because the SDK has to be unpacked first. You download an SDK for your target platform from the official release channel, then:

tar -zxvf ohos-sdk.tar.gz
cd $OHOS_SDK/Linux
for i in *.zip;do unzip ${i};done

and then run ./build-ohos.sh to build.

The presence of a Windows script, build.bat, and a vcpkg directory, alongside CMakePresets.json, suggests the project is packaged for consumption rather than only for its own CI. The top-level listing also has a build-and-run-no-jieba script, which is the jieba-free build matching the -DSIMPLE_WITH_JIEBA=OFF option, and a .gitmodules file for the dependencies, which is how cppjieba is brought in.

The source layout is small and maps directly onto the features. src/entry is the entry point that registers the SQLite tokenizer and the functions, src/simple_tokenizer is the tokenizer implementation, src/simple_highlight is the highlight function, modified from the built-in one so that adjacent matched words are highlighted continuously, and src/pinyin holds the Chinese to pinyin conversion and the logic for splitting a query into pinyin.

Two milliseconds to load, half a second to use pinyin, four to use jieba

The README's completed to-do list ends with performance figures, and they are the numbers to plan around: loading the extension takes under 2 milliseconds, first use of the pinyin feature needs to load the pinyin file and takes about 500 milliseconds, and first use of the jieba feature needs to load the jieba file and takes about 4 seconds.

Those three numbers describe a design in which the extension loads fast and the data loads lazily. The tokenizer itself is small and its registration cost is negligible, which is what you want for a loadable SQLite extension that any connection can pull in. The dictionaries are the expensive part, and they are only paid for when a query actually needs them.

The jieba number is the one that matters operationally, and it does not go away after the first call, because the dictionary stays loaded. Four seconds is a long time to hold a request, and it is a long time to hold a worker. If your service exposes search to end users, that cost is either paid on a warm path or designed out with a dedicated process that loads the dictionary once.

The pinyin number, about half a second, is more forgiving but not free, and it is the same kind of problem at a smaller scale. A system that has a pinyin table for a large set of characters and queries on every request needs the table warm, or it needs the loading to happen at startup rather than at first query.

The jieba-free build is the practical escape hatch. The repository ships build-and-run-no-jieba, and the switch -DSIMPLE_WITH_JIEBA=OFF removes the dependency entirely, which is the version to consider if the four seconds is unacceptable and the simpler segmentation is good enough.

Where the code lives, and how it is licensed

The examples directory is the clearest statement of who this extension is for. It holds examples for C++, Python, Go and node-sqlite3, and a separate document for React Native with op-sqlite. The README also points at community examples for iOS, including a pull request and two demo repositories, for Rust, for Java, for C# and for Android with Flutter.

That is a wide surface, and it is a consequence of the design. A loadable SQLite extension is the same binary everywhere, so the only thing each platform has to solve is how to load a shared object and what SQLite build it links against. The examples directory also holds example.sql and example-jieba.sql at the repository root level as the reference queries.

The licence needs a note. The repository's own metadata gives no standard classification for it, and the README badge states the terms as MIT or GPL v3 later, dual licensed. The LICENSE file in the repository root is the reference, and for a dual licence the choice is usually the caller's, which matters for a library that is linked into applications with different distribution models.

The maintenance picture is straightforward. Three releases in eight days in February 2026, from v0.6.1 to v0.7.1, with the last push on 2026-05-17 and the repository not archived. The project site at wangfenjin.com hosts the implementation write-up and a second article on the jieba tokenizer, and the topics list the compatibility markers, fts5, sqlite3, tokenizer, chinese, pinyin and cpp14, which is everything you need to know about what it is and what it needs from you.

Editorial conclusion

Adopt simple if your application searches Chinese text through SQLite and you need pinyin input to find Chinese content, because the extension handles the polyphonic character problem at the tokenizer level and gives you simple_query so callers do not have to learn fts5 query syntax. Do not adopt it if the corpus is enormous and the first-call latency matters, since the first pinyin call spends about 500 milliseconds loading the mapping and the first jieba call about four seconds loading its dictionary. Before you commit, decide whether you need jieba at all, because the jieba build and the no-jieba build are separate and the segmentation path is what costs the four seconds, check the pinyin table you ship, and remember that replacing the pinyin mapping does not rebuild the tokens already in your index, so plan a reindex when you change it.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. wangfenjin/simple on GitHub