pigeon: a PEG parser generator for Go

A grammar file in, a Go parser out

pigeon is described as a command that generates parsers in Go from a PEG grammar, a parsing expression grammar. The grammar and syntax are inspired by the PEG.js project, while the implementation is loosely based on the article on parsing expression grammar support for C# 3.0. The parser it generates reads Unicode text encoded in UTF-8.

The audience is specific: a team that owns a textual format and wants the syntax defined in one place. A grammar written in PEG is a specification you can read, and pigeon turns it into a Go parser with the semantic actions written in Go itself, so there is no second language and no runtime interpreter for the grammar.

The repository structure shows how seriously the grammar-as-source approach is taken. There is a grammar/ directory holding bootstrap.peg and pigeon.peg, an ast/ package for the grammar's own syntax tree, a builder/ package that turns that tree into code, a bootstrap/ directory with several command tools, and then the usual supporting directories of test/, testutils/, misc/ and examples/. The examples directory holds calculator/, indentation/ and json/, which is a good indication of the range the project targets: an expression evaluator, a language defined by indentation, and a structured format that most people would simply hand to encoding/json.

The dependency footprint is small. go.mod declares the module github.com/mna/pigeon, requires Go 1.26.0, and pulls in golang.org/x/tools at v0.51.0 with golang.org/x/mod and golang.org/x/sync as indirect dependencies. That x/tools dependency matters later, because it is the reason for the release policy.

Actions are Go code inside the grammar

The grammar file mixes syntax rules with Go code, and the example in the documentation makes the shape clear. A rule names its captures and then runs an action:

Input <- expr:Expr EOF {
    return expr, nil
}

Expr <- _ first:Term rest:( _ AddOp _ Term )* _ {
    return eval(first, rest), nil
}

The labels such as expr, first and rest bind to sub-results of the rule, and the action block is ordinary Go that returns a value and an error. A helper function defined in the initializer block at the top of the grammar, here eval with an ops map, is available to those actions, which is how a grammar keeps helper code next to the rules that use it.

The whitespace rule and the end-of-input marker show the two idioms a PEG grammar needs:

_ "whitespace" <- [ \n\t\r]*

EOF <- !.

The underscore rule is the conventional name for ignored whitespace, and the negative lookahead in EOF asserts that no input remains. Rules can also choose between alternatives with a slash, as the Factor rule does between a parenthesised expression and an integer.

The payoff is that the semantics live in Go, in the same file, reviewed by the same people. The cost is that a grammar file is not a standalone artefact: it mixes a syntax description with executable code, so the file has to be treated as source, linted and tested like source.

Installing the command and running it

Installation is a single Go tool invocation:

go install github.com/mna/pigeon@latest

That installs the command from the module path, and the tool is invoked with an optional grammar file:

pigeon [options] [PEG_GRAMMAR_FILE]

The default behaviour is worth knowing before you script around it. The input grammar is read from stdin and the generated code is printed to stdout, and you can save the output to a file with the -o flag. So the whole tool can be used in a pipeline without touching the filesystem, and the conventional workflow of committing the generated parser next to the grammar is a choice rather than a requirement.

The generated parser is ordinary Go that you compile into your program. The documentation gives the arithmetic example as the end-to-end check, a grammar that evaluates simple arithmetic and produces this:

18 + 3 - 27 * (-18 / -3)

=> -141

More examples live in the examples subdirectory, and the detailed usage documentation is on the package page at pkg.go.dev, with a separate Pigeon wiki covering both pigeon usage and PEG in general. That split is deliberate and useful, because the PEG concepts are the part that is hard to pick up from a tool's flags.

pigeon generates itself from bootstrap.peg

The most distinctive thing in the repository is the bootstrap/ directory and what the Makefile does with it.

There are two grammars in grammar/: bootstrap.peg and pigeon.peg. The Makefile builds a static code generator from the static_code_generator command, uses a bootstrap-build command to turn bootstrap.peg into bootstrap_pigeon.go, then builds a bootstrap-pigeon binary from that generated code, and finally uses it to regenerate pigeon.go and the pigeon binary itself. The all target chains those steps and then regenerates the test sources from the .peg files in the examples and test directories.

That is a self-hosting parser generator, and it is the reason the project can claim that its grammar is not a hand-maintained artefact that drifts from the implementation. The cost is a build that is more elaborate than a typical go build, and a Makefile with a dozen or so targets and a lot of path variables.

Two smaller files in the tree explain themselves by name. reserved_words.go is the list of Go keywords the generator has to avoid when it emits identifiers, and unicode_classes.go carries the Unicode character class data the generated parsers need for matching ranges outside ASCII. There is also a THIRD-PARTY-NOTICES file, which is where a code generator that incorporates parts of other projects records that obligation.

The release tooling is a .goreleaser.yml, so the published binaries are produced by the same pipeline that builds the project internally.

Version policy follows the Go Security Policy

The release policy section of the documentation is unusually candid, and it explains the version numbers.

From June 2023 the backwards compatibility support for pigeon follows the official Go Security Policy. The reasoning given is a chain of dependency problems. Packages like golang.org/x/tools, which the documentation calls a critical dependency of pigeon, follow the official policy themselves, and while pigeon did not, it was no longer possible to include recent versions of those dependencies, and therefore no longer possible to include critical bugfixes in them.

The second reason is about code quality rather than security. Community practice moved from interface{} to any, and the code generated by pigeon no longer met the expectations of teams following or enforcing that practice.

The conclusion the documentation draws is that following the Go Security Policy has been a smooth experience over recent years, and that updating Go on a regular basis is a reasonable duty to put on users of pigeon.

That has direct consequences for adopters, and they cut both ways. Supported Go versions get dropped on the Go schedule rather than the generator's schedule, so a pigeon upgrade can force a Go upgrade. The offsets are visible in the repository: the release list shows v1.2.0 in September 2023, v1.2.1 in October 2023 and v1.3.0 in September 2024, while the last push was on 2026-10-14 and the repository is not archived. Work continues, but the tagged releases are not frequent, so pinning a version is a real decision. The documentation also records that v1.0.0 is the tagged release of the original implementation and that work has started on v2.0.0 with some planned breaking changes.

Where a generated PEG parser is the wrong choice

There are two cases where pigeon is not the answer, and one of them is in the project's own examples directory.

The first is a format the standard library already handles. The examples include a json/ case, and for JSON the right tool is encoding/json with its struct tags, not a generated PEG parser. Generated code for a format that already has a maintained implementation adds a build step and a hand-maintained grammar for no gain. The same logic applies to formats where a well-maintained third-party parser already exists and the grammar would be a worse copy of theirs.

The second case is when the grammar is not really a grammar. A PEG grammar describes an ordered choice of alternatives with no ambiguity, and it suits formats with a definite structure such as configuration files, small protocol messages, or domain-specific languages. It is the wrong tool for a format whose productions are genuinely ambiguous, or one where parsing depends on context that a context-free rule cannot express, and in those cases you will find yourself rewriting the format to fit the tool rather than fitting the tool to the format.

There is a third, practical concern that the documentation does not raise. The generated parser is code you do not write, and it enters your repository and your build. If your organisation has rules about generated code, vendored code, or the volume of code under review, the size of the output is a factor. The counter is that a PEG grammar is far easier to review than an equivalent hand-written recursive descent parser, and the actions are Go code you wrote, so the review effort moves from a thousand lines of control flow to a page of grammar plus a page of actions. Which trade wins depends on whether your reviewers can read a grammar, and on whether you need to debug by reading code or by reading rules.

Maintainership, licence and the 1.x to 2.x transition

The project has a stable origin. The documentation records that the GitHub user @mna created the package in April 2015 and that @breml is the package's maintainer as of May 2017. That is a decade of continuity for a tool of this kind, and the fact that the release policy is written down at all suggests a project that has thought about its users.

The licence is BSD 3-Clause, stated in the documentation and covered by the LICENSE file. For a code generator that produces output you own and commit to your own repository, that is one of the most convenient licences available, since there is no obligation on the generated code and no requirement to publish anything about your project.

The one thing to plan for is the v2 transition. The documentation says work has started on v2.0.0 with some planned breaking changes, and the security policy alignment already showed what a breaking change looks like in practice here, since moving from interface{} to any changed the shape of the generated code rather than just the internals of the tool. If you adopt pigeon, adopt it on a specific version, keep the grammar and the generated code in the same commit, and expect to regenerate when you upgrade.

For an evaluation, the fastest route to a judgement is the examples directory. Write a grammar for your own format, generate the parser with pigeon -o, and compare the result with what a hand-written parser of the same format would look like in your codebase. The decision usually comes down to whether your format is stable enough to have a grammar, and the answer to that question is usually visible within an hour of writing it.

Editorial conclusion

Adopt pigeon when you own a small or medium textual format and want one grammar file, checked into the repository, to define the syntax with the semantics expressed as ordinary Go code. Skip it if the format is JSON, since the standard library already covers that, if you need left recursion or a grammar the PEG model does not express, or if a generated parser would be harder to review than a hand-written one. Verify first that pigeon generates and runs on your grammar with a small end-to-end example before porting an existing parser, and pin the tool version because the release line has moved between v1.x and the planned v2.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. mna/pigeon on GitHub
4. README
5. Releases