go-ego/gse: Chinese, Japanese and English text segmentation in Go

What gse solves for Go services that handle CJK text

Splitting Chinese or Japanese text into words is not whitespace tokenization. A Go service that indexes Chinese documents, counts term frequencies or extracts keywords needs a dictionary-driven segmenter, and the usual answer is jieba, which is Python. gse is a Go implementation of jieba with added NLP support, which means the tokenizer runs inside your Go process instead of behind an RPC call to a Python worker. The README states this directly: "Gse is implements jieba by golang, and add NLP support and more features."

The audience is narrow and specific. You are writing Go, you have text in Chinese, Japanese, English or a mix, and you want tokens rather than characters. The README also lists integrations with riot, zincsearch, elasticsearch and bleve, so a second audience is anyone wiring a search backend that needs a CJK analyzer written in Go. If your pipeline is Python, you should use jieba itself and skip the port.

Double-array trie, DAG shortest path and HMM Viterbi

Two structures do the work. The dictionary is stored in a double-array trie, implemented in dictionary.go. Segmentation runs on a DAG: the README describes the segmenter algorithm as "the shortest path (based on word frequency and dynamic programming), and DAG and HMM algorithm word segmentation." In practice that means the segmenter builds a directed acyclic graph of every dictionary match over the input, then picks the lowest-cost path using word frequencies. Words the dictionary does not cover can be handled by the HMM path, which the README says uses the Viterbi algorithm.

The mode you choose changes the output shape, and this is the part that matters more than the algorithm names. Cut returns a standard segmentation. CutSearch is tuned for search-engine indexing, where long words are also emitted in shorter pieces so a query for a substring still matches. CutAll emits every dictionary word found, which produces many overlapping tokens. CutDAG accepts a regular expression, and the README example uses one to pull out dates, Latin runs, Hangul runs and decimals before cutting. There is also an Analyze call that returns segment information, and Trim and stop-word helpers for filtering.

The dictionary is not fixed. LoadDict with no argument loads the default, and the README lists zh_s for Simplified Chinese, zh_t for Traditional Chinese and jp for Japanese as named variants. LoadDictEmbed loads a dictionary embedded in the binary, which removes the need to ship a data file alongside the executable. The AlphaNum flag controls whether Latin and numeric runs are kept as single tokens, and ToLower controls case folding. Those two settings alone will decide whether "Helloworld" comes back as one token or two.

Installing gse and cutting your first mixed-language string

The module path is github.com/go-ego/gse and go.mod declares go 1.24. With module support the README says to import the package directly; without it, run the go get command. Both are shown below.

go get -u github.com/go-ego/gse

import "github.com/go-ego/gse"

Once the import resolves, the smallest working program declares a gse.Segmenter, loads a dictionary, and calls Cut. The README example uses a test dictionary from the repository, but LoadDict with no argument loads the default dictionary, which is what you want in a real service. LoadDict returns an error, so check it.

package main

import (
	"fmt"

	"github.com/go-ego/gse"
)

func main() {
	var seg gse.Segmenter
	if err := seg.LoadDict(); err != nil {
		fmt.Println("Load dictionary error: ", err)
	}

	text := "Hello world, Helloworld. Winter is coming! こんにちは世界, 你好世界."
	fmt.Println(seg.Cut(text))
}

After the dictionary loads, Cut returns a slice of strings. The README's own example output for a similar mixed string shows the tokens separated by commas and spaces, with the Japanese and Chinese portions split into words rather than characters. If you need the embedded dictionary instead of a file on disk, swap LoadDict for LoadDictEmbed; the README shows LoadDictEmbed being called with dictionary text concatenated from embedded variables. For Traditional Chinese or Japanese, pass the variant name: seg.LoadDict("zh_t") or seg.LoadDict("jp").

One detail worth setting before you cut anything is AlphaNum. In the README's first example, seg2 sets AlphaNum = true and the input "Hiworld, Helloworld!" comes back as "[hi world , hello world !]", meaning the camel-case run was split into two tokens. With AlphaNum left at its default, the same input behaves differently. Decide which you want before you write tests against the output.

Where gse is the wrong tool

The README is candid about scope, and two entries in the feature list are marked "in work": NLP by TensorFlow and Named Entity Recognition. If your requirement is entity extraction, sentiment classification or any trained model output, gse does not ship that today. The crf/ and tf/ directories exist in the repository, but the README does not document them as usable features, and nothing in the install or use sections references them. Treat them as unfinished.

There is a second limit. Segmentation quality is a function of the dictionary, and the README does not describe how to train, extend or evaluate one beyond loading a user dictionary. Domain vocabulary (product names, drug names, legal terms) will not be in the default dictionary unless you supply it, and the README gives no guidance on dictionary format beyond the test data files it points at. If your corpus is heavy on jargon, budget time for dictionary work that the documentation does not walk you through.

Finally, the speed numbers in the README are the project's own measurements on a 2-core, 4-thread MacBook Pro: 9.2MB/s single-threaded, 26.8MB/s with goroutines, and 3.2MB/s for HMM segmentation single-threaded. Those are useful for order-of-magnitude planning, but they are single-machine figures with no stated corpus, and they should not be treated as a guarantee for your input.

How gse differs from jieba and from a full NLP framework

The closest alternative is jieba, and the difference is the runtime, not the algorithm. gse is a port of jieba, so the segmentation approach (dictionary plus DAG plus HMM) is the same family. Choosing between them is choosing between a Go binary and a Python dependency. If your service is Go and you want one process, gse removes the Python interpreter, the cross-language call and the deployment surface that comes with it. If your service is Python, or if you need the wider Python NLP ecosystem around the tokenizer, jieba is the direct answer and gse is not.

The second comparison is against a full NLP framework. A framework gives you tokenization plus tagging plus models in one package, at the cost of a heavier dependency. gse is a segmenter with POS tagging and analysis helpers attached, and the README lists the model-based features as unfinished. That is a smaller commitment and a smaller capability set. If you only need tokens, the smaller commitment is the point. If you need tokens plus predictions, you will be pairing gse with something else or waiting.

Maintenance, licence and what an upgrade costs

The repository is not archived and the last push was on 2026-09-12, four days before this writing. Releases are sparse: v1.0.0 on 2025-12-05, v1.0.1 on 2026-03-02 and v1.0.2 on 2026-03-12. That cadence suggests a project that changes slowly, which for a tokenizer is closer to a feature than a problem, because the dictionary format and the Cut signature are the parts you build against.

The dependency surface is small. go.mod requires only github.com/vcaesar/cedar and github.com/vcaesar/tt, both by the same author as the elasticsearch and bleve integrations. A short dependency list means fewer transitive upgrades to chase, but it also means those two packages are single-maintainer dependencies and worth checking on your own.

The licence is Apache-2.0. That is a permissive licence with an explicit patent grant and a requirement to preserve notices; it is compatible with commercial use. This is a description of the licence identifier, not legal advice, and if you redistribute gse inside a product you should read the LICENSE file in the repository rather than this paragraph.

Upgrade cost is the open question. The README does not document rollback, a migration guide or a compatibility policy across the 1.0.x line, and the release notes are not reproduced here. The practical check is to pin a version, run your own corpus through it, and diff the token output before bumping. Segmentation output is the contract, and that is what a version change can move.

Editorial conclusion

Adopt gse if you are already writing Go and need Chinese, Japanese or English tokenization inside the same binary, whether for search indexing, keyword extraction or a preprocessing step. Do not adopt it if you need a trained NLP pipeline today: the README lists TensorFlow-based NLP and named entity recognition as in work, and the CRF and tf directories are present but not documented as finished features. Before committing, verify three things on your own corpus: which dictionary variant you need (zh, zh_s, zh_t or jp), whether Cut, CutSearch or CutAll matches your downstream consumer, and how much memory the dictionary holds once loaded. The repository was last pushed on 2026-09-12 and is not archived, so the code is current, but the README does not document rollback, migration or a version compatibility policy, which is the gap to close before you depend on it.

Frequently asked questions

Official sources

1. go-ego/gse on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases