jdkato/prose: a pure-Go NLP library for tokenizing, tagging and extracting entities

What jdkato/prose solves, and who it is for

Most NLP tooling assumes Python and a model server. If your service is written in Go, that means either shelling out to a Python process, calling an HTTP endpoint, or linking a C library through cgo. Each of those adds a deployment dependency you did not have before. prose takes the other route: it is pure Go, with no cgo and no external services, and it ships its own trained models inside the module. The README states it is English only, which is a boundary rather than a footnote.

The intended user is a Go engineer who needs the classical pipeline stages on English text: splitting a document into words and punctuation with byte offsets, splitting it into sentences with the Punkt algorithm, assigning Penn Treebank part-of-speech tags, pulling out people, places and organizations, and computing readability metrics such as Flesch-Kincaid, Gunning fog and SMOG. The module also includes strcase for AP and Chicago title case, which is a small utility that has nothing to do with NLP but sits in the same import path.

It is not a general text analytics platform. There is no training API, no embedding output, no dependency parser and no language detection. The README lists exactly six capabilities, and the package table names eight packages, which tells you the scope is deliberately narrow.

How the pipeline is put together

The root package exposes a Document, which is the whole pipeline applied to one input string. NewDocument runs the enabled stages and the resulting tokens, sentences and entities are retrieved through accessor methods. The stages have a strict dependency order that the README makes explicit: extraction requires tagging, and tagging requires tokenization, so enabling a later stage enables the earlier ones automatically. The configuration options are WithTokenization, WithSegmentation, WithTagging and WithExtraction, each taking a bool, so you can turn off work you do not need. The README notes that skipping entity extraction avoids most of the work.

The design choice worth noticing is the offset contract. Every token, sentence and entity carries a byte offset into the source, and the README states the invariant src[tok.Start:tok.End()] == tok.Text, including for multi-byte input, repeated words and irregular whitespace. That invariant is why the library never normalizes text before tokenizing: replacing curly quotes with straight ones, or an HTML entity with an apostrophe, changes byte lengths and shifts every offset that follows. Normalization is left to the caller, before prose sees the text. This is a real constraint on how you integrate the library, not a stylistic preference.

Underneath, the module depends on github.com/dlclark/regexp2/v2 and gopkg.in/neurosnap/sentences.v1, with github.com/neurosnap/sentences as an indirect requirement. The Punkt sentence segmenter comes from that second dependency. The models are described as loading once per process and being shared, and the README states that Tokenizer, Segmenter, Tagger and Extracter are safe for concurrent use because their models are immutable and per-call scratch space is pooled. The Makefile has a race target that runs go test -race -count=1 ./..., with a comment saying this is what proves the concurrency claim.

Installing jdkato/prose and running a first document

The README gives a single install command, run inside a Go module. The module path carries the v3 major version, so the import path must match it.

go get github.com/jdkato/prose/v3

The quick start builds a Document from a two-sentence string and then prints sentences, tagged tokens and entities. Note that entity labels are not free-form strings: the README's inline comments show PERSON for a person and GPE for a geopolitical entity, which is the standard label set for this kind of recognizer.

package main

import (
    "fmt"
    "log"

    "github.com/jdkato/prose/v3"
)

func main() {
    doc, err := prose.NewDocument("Marie Curie studied in Paris. She won the Nobel Prize twice.")
    if err != nil {
        log.Fatal(err)
    }

    for _, ent := range doc.Entities() {
        fmt.Println(ent.Text, ent.Label)
    }
}

Running that prints each entity and its label. If you only need tags, the README shows importing the tag package directly instead of the root package, which keeps the entity model out of your binary. The stated difference is 4.5 MB, because the tagger model is about 3 MB and the entity recognizer about 4 MB, and each lives in its own package so the linker drops what you do not import.

import "github.com/jdkato/prose/v3/tag"

tagger, err := tag.New()
if err != nil {
    log.Fatal(err)
}

for _, tok := range tagger.Tag([]string{"Kubernetes", "orchestrates", "containers"}) {
    fmt.Println(tok.Text, tok.Tag)
}

One more option matters in production. Work scales with input length, and the README gives roughly 1.4 s for a million words, so NewDocumentContext accepts a context.Context and the example wraps a five-second timeout. The individual tokenizers and taggers take no context, on the stated grounds that they work at microsecond scale.

Where prose stops being the right tool

The English-only restriction is the first hard limit, and it is stated plainly in the README rather than buried. If your input is mixed-language or non-English, no amount of configuration fixes this; the trained models are for English and the tag set is Penn Treebank.

The offset invariant is the second limit, and it cuts the other way from what you might expect. Because prose refuses to normalize, you cannot hand it text straight from a web scrape and expect clean tokens for curly quotes, HTML entities or non-breaking spaces. You must decide what to normalize and do it before the call. Any normalization you perform after tokenization invalidates the offsets you just paid for. Teams that want a library to handle that for them will find this arrangement inconvenient, and the README is upfront that this is a deliberate trade rather than an oversight.

The third limit is that the README publishes no accuracy figures. There are no precision or recall numbers for the tagger or the entity recognizer, no evaluation dataset, and no comparison against other Go NLP libraries. The repository has a testdata directory and a coverage target in the Makefile, but coverage measures how much code the tests exercise, not how well the models perform on your text. You cannot judge tagging quality from the documentation; you have to run it on your own corpus.

The fourth is maintenance signal rather than capability. The last push to the repository was on 2026-07-29, which is recent, but the newest release listed is v1.2.1 from 2020-12-22. The default branch is v3 and go.mod declares go 1.25, so the development line and the tagged releases are far apart. If you pin to a released version rather than the branch, you are pinning to something from 2020.

How prose differs from prose v2 and from calling a Python tagger

The most direct alternative is the project's own earlier line. The releases listed are v1.2.1, v2.0.0 and v1.2.0, all from 2020, while the default branch is v3 and the README documents the v3 import path github.com/jdkato/prose/v3. That means the documentation you read describes code that is not in any tagged release shown here. If you need a version you can pin to a tag, you are choosing between an older API and the branch. The README does not document a migration path from v2 to v3, so the differences are not spelled out anywhere in this material.

The other real alternative is to keep the NLP in Python and call it from Go. spaCy or NLTK would give you a much larger model selection, multilingual support and published evaluation numbers. The difference in approach is architectural, not just quality. A Python service means a second runtime in your deployment, a network hop or an embedded interpreter, and a serialization boundary for every request. prose trades model breadth and documented accuracy for a single static binary with no cgo and no service dependency. For a CLI tool, a sidecar that tags log lines, or a Go service that needs sentence splitting and title casing, that trade is usually worth it. For a batch pipeline where accuracy is the whole point, it usually is not.

There is also a middle option worth naming: if you only need sentence segmentation, the Punkt implementation here derives from gopkg.in/neurosnap/sentences.v1, so that dependency is reachable on its own. Importing prose for segmentation alone pulls in more than you need.

Licence, module requirements and the cost of upgrading

The README states the licence is MIT, and points at the LICENSE file, which the README says also records the upstream work prose derives from. The repository metadata reports the licence as NOASSERTION, which means automated tooling did not classify it from the file. Those two statements are not in conflict, but they do mean a dependency scanner may flag this module and a human will have to open LICENSE to resolve it. That is an administrative cost, not a legal problem, and nothing here should be read as legal advice.

The upgrade cost has two parts. The first is the Go toolchain: go.mod declares go 1.25, so a project on an older toolchain cannot build this module without upgrading Go first. That is a hard gate, not a suggestion. The second is the release gap. With the newest listed release at v1.2.1 from 2020-12-22 and the default branch at v3, moving to the documented API means tracking a branch rather than a tag, or waiting for a v3 release that is not listed here. The README does not document rollback, does not document a deprecation policy, and does not say which v2 APIs were removed.

On the operational side, the Makefile gives the maintenance commands the maintainers use: go test -count=1 ./... for the suite, go test -race -count=1 ./... for the concurrency claim, and golangci-lint run for linting. A team vendoring this module can run the same targets against their fork. The models are the part you cannot easily maintain yourself: retraining them is not covered by the README.

Editorial conclusion

Adopt jdkato/prose if you are writing Go and need English tokenization, Penn Treebank tags, sentence splitting, entity extraction or readability scores without cgo or a service call. Do not adopt it if you need multilingual support, a transformer-based tagger, or a library whose licence file you can classify without opening it. Before you commit, verify three things: that go.mod's go 1.25 directive matches your toolchain, that the LICENSE file's upstream attribution is acceptable to whoever reviews dependencies, and that the tagger's accuracy on your own text is good enough, since the README publishes no evaluation numbers.

Frequently asked questions

Official sources

1. Issues
2. jdkato/prose on GitHub
3. README
4. Releases