go-enry and language detection as a set of narrowing strategies

A ladder of strategies, not a single heuristic

enry is a programming language detector and a toolbox to ignore binary or vendored files. It started as a port to Go of the original Linguist Ruby library, with an improved 2x performance claim.

The core idea is that detection is a sequence of matching strategies applied progressively to narrow down the possible options, and each strategy varies on the type of input data it needs.

The API is organised around the available input. With only a filename, GetLanguageByExtension uses the extension, which may be ambiguous, and GetLanguageByFilename handles extensionless names such as .gitignore and .bashrc. With text, GetLanguageByShebang reads the first line, GetLanguageByModeline handles a Vim or Emacs modeline such as a set ft=cpp comment at the head or tail, and GetLanguageByClassifier runs a Bayesian classifier trained on all the samples from Linguist.

With a full file, GetLanguagesByContent combines the extension with regexp-based content heuristics, and GetLanguages uses the full set of strategies and is expected to be most accurate.

The order is the design. The cheap signals run first and the expensive classifier runs last, usually as a disambiguator, which is why the classifier needs a candidate list and can accept the keys of data.LanguagesLogProbabilities as a fallback at the price of suboptimal accuracy.

The audience is a Go developer building repository analysis, and the note that filename-only guesses are expected not to be very accurate is the one to keep in mind when choosing which API to call.

The safe flag and the plural API

Every detection call returns a language and a boolean, and the README explains exactly what the boolean means: it is true if there is only one possible language detected.

That is the part of the API worth internalising, because it converts a guess into a decision you can act on. A caller that only wants to colour or highlight files can ignore the flag. A caller building a statistic, a language breakdown on a repository, or a filter that routes files to different linters should check it, because a false flag means the extension matched several languages and the content strategies could not narrow it down.

The plural forms give you the whole candidate set instead of a single answer:

langs := enry.GetLanguages("foo.h",  []byte("<cpp-code>"))
// result: []string{"C", "C++", "Objective-C}

The .h case is the canonical illustration of why the flag exists. A C header could be C, C++ or Objective-C, and the content is what decides.

There are also plural variants that take only an extension or only a filename:

langs := enry.GetLanguagesByExtension("foo.asc", []byte("<content>"), nil)
// result: []string{"AGS Script", "AsciiDoc", "Public Key"}

langs := enry.GetLanguagesByFilename("Gemfile", []byte("<content>"), []string{})
// result: []string{"Ruby"}

The Gemfile example shows the filename strategy resolving a case where the extension is absent entirely.

The file filtering helpers are half the API

Before you can count languages in a repository you have to decide which files to count at all, and enry exposes that as a set of file-level helpers.

The list is IsBinary, IsVendor, IsConfiguration, IsDocumentation, IsDotFile, IsImage, IsTest and IsGenerated. The README frames them as helpers to simplify filtering out files that are less interesting for the purpose of source code analysis.

That is the more useful half for most callers, because a language histogram that includes minified bundles, vendored dependencies, generated code and test fixtures tells you almost nothing about what a repository is written in. The filters are what turn the detector into something you can build statistics on.

The second group is presentation rather than detection. GetColor returns the language colour, which is useful for rendering a breakdown as a graph, and GetLanguageGroup groups similar languages together, with the example given being that Less returns CSS. That second function is a convenience for a treemap or a pie chart, not part of the detection path.

The repository layout reflects the split. The detection core is in enry.go, classifier.go, common.go and utils.go, with the language data under data/ and the regexp heuristics under regex/. There is a shared/ directory holding the native library source, and separate java/, python/ and internal/ trees. A linguist_corpus_test.go at the root is the test that parses the upstream Linguist samples, which is how the port is checked against its source of truth.

Bindings for Java, Python and Rust, and where the CLI lives

The Go library is the primary artifact, and the other language bindings are generated from a C shared library rather than hand-written.

The Java bindings are generated using a C shared library and JNI, and a library is published on Maven as tech.sourced:enry-java for macOS and Linux platforms, with Windows support planned under a linked issue. The Python bindings use a C shared library and cffi, available for Python 3.9 and later, installed with:

pip install enry

Pre-built wheels are available for Linux on x86_64 and for macOS on both Intel and Apple Silicon, and the package is published on PyPI as enry for macOS and Linux. Usage is a two-liner:

import enry

language = enry.get_language("example.go", b"package main")
print(language)  # Output: Go

The Rust bindings are generated using a C static library and live at the go-enry/rs-enry repository.

One thing that trips people up: the CLI binary is not in this repository. The README says it is hosted in a separate repository, go-enry/enry. So this is the library, and the command line tool is somewhere else.

The Makefile shows how the shared library is produced, and it is more involved than a normal Go build. Because the bindings need a C shared object, the build uses buildmode=c-shared with CGO_ENABLED=1, and it detects the target OS to pick the library extension, so on Darwin it produces a dylib and elsewhere an so. The Makefile also clones the upstream Linguist repository to a .linguist directory and runs a code generator, which is how the language data in data/ is produced from the upstream source.

The language database is frozen, and the divergences are documented

The most important fact for anyone adopting this is that the enry library is based on the data from github/linguist version v9.5.0.

That is a specific, frozen snapshot. It means the language list, the extensions, the heuristics and the classifier training data all come from one point in time, and they do not grow when Linguist grows. The Makefile has a code-generate target that can resync from a fresh clone of Linguist, so the data can be refreshed, but the refresh is a deliberate maintainer action rather than something that happens on its own.

The README is also explicit about where the port differs from upstream. Parsing the linguist/samples directory, some enry results differ from Linguist's, and the first named case is the heuristics for the .txt extension in Vim Help File, which could not be parsed because of an unsupported negative lookahead in the RE2 regexp engine.

That detail is worth reading closely, because it explains a whole class of difference. Go's standard regexp engine is RE2, which does not support lookaround, and Linguist's heuristics in YAML use it. The upstream heuristics are therefore not portable verbatim, and the port has to approximate them. The section goes on to list further differences for the .so case, and the README's structure of a diverging item followed by a reason is the pattern to expect: where the port diverges, the reason is usually a regexp feature RE2 lacks.

So the honest framing is that enry is a snapshot with documented approximations, not a live mirror of Linguist.

Editorial conclusion

go-enry is a good fit for a Go service that needs to label files by language, for example to drive syntax highlighting or repository statistics, because the detection is a set of composable strategies rather than a single heuristic and the safe flag distinguishes a confident answer from a guess. It is a poor fit if you need a language database that tracks new languages as they appear, because the data is based on Linguist v9.5.0 and the divergences from upstream are documented rather than silently patched, and it is a poor fit for Java, Python or Rust users who do not need Go, since those bindings live in separate projects. Before adopting, check which language data your codebase needs is present, read the divergences section to see where the port differs from Linguist, and remember the CLI binary is hosted in the separate go-enry/enry repository rather than here.

Frequently asked questions

Official sources

1. go-enry/go-enry on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases