zhparser: teaching PostgreSQL to index Chinese text one word at a time

PostgreSQL's default parser treats a Chinese paragraph as one token

PostgreSQL's built-in text search splits text on whitespace and punctuation, so a run of Han characters lands in the index as one enormous token. Put a Chinese article into a tsvector column and searching it becomes equivalent to matching whole paragraphs. zhparser exists to remove exactly that gap. It is a C extension that registers a Chinese parser with PostgreSQL's text search framework and splits Mandarin sentences into dictionary words before anything is indexed. The audience is narrow: a team that already keeps Chinese content in Postgres and wants tsvector, tsquery and ranking to behave on that content, without standing up a separate search service. The project describes itself as a PostgreSQL extension for full-text search of Chinese language, and its implementation leans on the SCWS segmentation library rather than inventing its own word boundary logic.

SCWS supplies the dictionary, the Makefile supplies the rules

zhparser does not carry its own segmentation engine. The README states it implements the Chinese parser on the basis of Simple Chinese Word Segmentation, and the repository Makefile links the extension against the libscws C library. That Makefile resolves the library in a defined order: an explicitly set SCWS_HOME wins, otherwise pkg-config flags are used when scws is registered there, and the fallback is /usr/local. Dictionary files travel with the extension. The Makefile installs dict.utf8.xdb and rules.utf8.ini as tsearch data, so the word list and the rule set arrive with the package instead of being something you assemble by hand.

Two statements then wire the parser into a text search configuration. CREATE TEXT SEARCH CONFIGURATION names zhparser as the parser, and ADD MAPPING binds the token categories to the simple dictionary so tokens are indexed as plain words. The ts_parse call in the documentation shows the raw output before any indexing happens, which is the fastest way to see what segmentation you actually get:

SELECT * FROM ts_parse('zhparser', '保障房资金压力');

Each row carries a tokid and a token, and the ids differ by category, with numbers, Han words and punctuation kept apart. A text search query over the same configuration is written the same way as any other:

SELECT to_tsquery('testzhcfg', '保障房资金压力');

Running the container is the fastest way to see the tokens

The published container is built on the official PostgreSQL image, so nothing on your host needs to change while you evaluate. The documented tag pins PostgreSQL 16 on bookworm. Start it detached under a name you will recognise:

docker run --name pgzhparser -d -e POSTGRES_PASSWORD=somepassword zhparser/zhparser:bookworm-16

When the container is accepting connections, attach a psql session as the postgres superuser:

docker exec -it pgzhparser psql postgres postgres

Inside that session, install the extension, create a configuration bound to the parser, map the token categories to the simple dictionary, and look at the tokens the parser produced:

CREATE EXTENSION zhparser;
CREATE TEXT SEARCH CONFIGURATION testzhcfg (PARSER = zhparser);
ALTER TEXT SEARCH CONFIGURATION testzhcfg ADD MAPPING FOR n,v,a,i,e,l WITH simple;
SELECT * FROM ts_parse('zhparser', 'hello world! 2010年保障房建设在全国范围内获全面启动');

The documented output for that sentence is fourteen rows, mixing ASCII words, the number 2010, punctuation, and Chinese tokens such as 保障, 房建, 设在 and 全国. That listing is worth reading closely, because it shows segmentation behaviour you inherit wholesale: 房建 and 设在 are produced from the shipped dictionary and rules, not from your data. The Docker Hub page for zhparser/zhparser carries further image details, and the image follows the conventions of the upstream PostgreSQL image.

Source installs need SCWS, a pg_config and the right make

The source route has real prerequisites. zhparser supports PostgreSQL 9.2 and newer, and on Red Hat and CentOS you need the development libraries and headers, which the documentation says are generally in the postgresql-devel package. SCWS goes in first, and it is the part most people miss:

wget -q -O - http://www.xunsearch.com/scws/down/scws-1.2.3.tar.bz2 | tar xf -
cd scws-1.2.3 ; ./configure ; make install

If you took the SCWS source from GitHub rather than the tarball, the configure script has to be generated first:

touch README;aclocal;autoconf;autoheader;libtoolize;automake --add-missing

On FreeBSD 10 and later, configure needs the --with-pic option. Then the extension itself:

git clone https://github.com/amutu/zhparser.git

make && make install

When SCWS sits outside /usr/local, set SCWS_HOME before building. When several PostgreSQL versions are installed, point PG_CONFIG at the one you want:

PG_CONFIG=/usr/lib/postgresql/9.5/bin/pg_config make && make install

On the BSDs the build uses gmake rather than make. The last step is a database change, not a file copy:

psql dbname superuser -c 'CREATE EXTENSION zhparser'

Custom dictionaries moved out of the install directory and into the database

The dictionary story changed with the 2.1 custom dictionary scheme, and that change decides how you extend the vocabulary. In the 2.0 arrangement, extra dictionaries were files placed in share/tsearch_data and selected with the zhparser.extra_dicts setting, where a .txt extension means a text format and .xdb means the binary format. Several files are separated by commas, and their priority runs from low to high.

Under 2.1 the custom word list is stored in the database instead. The documentation is explicit that these operations need superuser rights, that the custom dictionary is per-database rather than per-instance, and that the data lives under the data directory in base, under the database ID, where version 2.0 kept it in share/tsearch_data. The workflow is a table insert followed by a sync call:

insert into zhparser.zhprs_custom_word values('资金压力');

select sync_zhprs_custom_word();

The documented result of ts_parse changes after the sync and a fresh connection, with 资金压力 appearing as one token where it had previously been split into 资金 and 压力. The connection matters: the example reconnects before parsing again. Entries can carry attributes, and setting the word attribute to a bang marks the word invalid even if a core dictionary contains it.

Two settings only apply to a backend that is started fresh

Most of the behaviour switches are ordinary PostgreSQL settings that default to false, and they are named after the SCWS options they mirror:

zhparser.punctuation_ignore = f
zhparser.seg_with_duality = f
zhparser.dict_in_memory = f
zhparser.multi_short = f
zhparser.multi_duality = f
zhparser.multi_zmain = f
zhparser.multi_zall = f

They cover punctuation handling, two-character aggregation of loose text, loading the whole dictionary into memory, and the several compound-word modes. What matters operationally is scope. The documentation states that zhparser.extra_dicts and zhparser.dict_in_memory must be set before the backend starts, by editing the configuration file and reloading, after which new connections pick them up. Every other option can be changed inside a live session and takes effect immediately.

Text dictionaries have their own format rules. Each line holds one record, lines beginning with # or a semicolon are comments, and a record has four fields separated by spaces or tabs: the word, TF, IDF and a part of speech tag. Everything except the word may be omitted, in which case TF and IDF default to 1.0 and the tag defaults to @. A text dictionary is watched for modification times and converted to xdb in the system temporary directory at load time, which is why the documentation advises keeping TXT dictionaries small and points at scws-gen-dict for large binary XDB files instead.

A permissive licence text that GitHub could not classify

The licence is listed as Other, and the text sits in the COPYRIGHT file at the root of the repository. It is a permissive grant in the BSD mould: permission to use, copy, modify and distribute the software and its documentation for any purpose, without fee and without a written agreement, provided that the copyright notice and the following paragraphs appear in all copies. The following paragraphs disclaim warranties and limit liability, and they name the University of California even though the copyright line reads Portions Copyright (c) 2012-2017, Jov([email protected]). That mix is worth reading directly in the file before you redistribute it, and this is not legal advice.

Maintenance timing is uneven. The last push was on 2026-06-29 and the repository is not archived, but the tagged releases are older: v2.3 on 2025-01-24, v2.2 on 2023-06-09, and v0.2.0 on 2017-05-28. The tree already carries zhparser--2.4.sql and the zhparser--2.3--2.4.sql upgrade script, and the Makefile lists both, so the SQL for a 2.4 path is present in the source even though no 2.4 release appears in the release history.

Where zhparser is the wrong tool

Start by ruling out the wrong category of problem. zhparser segments Chinese text for a lexical index inside PostgreSQL. It does not translate, embed, summarise or match across languages, and nothing in the project describes any such behaviour. It is also the wrong answer if your database is not PostgreSQL, since the whole thing is a shared library built against pg_config and libscws.

Even inside PostgreSQL the edges bite. The database-level custom word list needs superuser rights plus a new connection after sync_zhprs_custom_word(), which does not suit a managed instance where you control neither. The file-based dictionaries in share/tsearch_data are converted to xdb in the system temporary directory at load time, so large text dictionaries are a poor fit while the binary XDB route needs the scws conversion tool. The project does not describe a rollback procedure for custom word entries, and it does not describe a way to pin segmentation so the same corpus always splits the same way.

Practical advice follows from that: if your users search for product names, acronyms or internal project names, put those exact strings through ts_parse first and check what comes out. The dictionary decides the outcome, not the query you write.

Editorial conclusion

zhparser suits teams already running PostgreSQL that need Mandarin keyword search without a separate search engine, and whose licence review accepts the permissive terms in the project COPYRIGHT file. It does not suit teams needing semantic matching, running anything other than PostgreSQL, or lacking superuser access to the instance. Before adopting it, check that segmentation of your own domain vocabulary matches what users type, and that your environment lets you set zhparser.extra_dicts and zhparser.dict_in_memory before the backend starts.

Frequently asked questions

Official sources

1. amutu/zhparser on GitHub
2. Issues
3. README
4. Releases