Tokei: counting code across 150+ languages from the command line

What Tokei counts and who needs those numbers

Tokei answers a narrow question: how much code is in this directory, broken down by language? The README states it will "show the number of files, total lines within those files and code, comments, and blanks grouped by language." That is the whole product. It is not a complexity analyser, a coverage tool or a contributor report.

The people who get value from it fall into three groups. Engineers who inherit a repository and want a size estimate before reading anything. Build and CI maintainers who want a threshold check, for example failing a job when the generated JSON crosses a line budget. And anyone writing documentation or a README badge who needs a defensible number rather than a guess.

The output format matters here. The README example shows a table with columns for Files, Lines, Code, Comments and Blanks, plus nested rows: Markdown files containing embedded Rust or JSON are broken out as sub-rows with a (Total) line. That nesting is the part most line counters skip, and it is why the numbers for a polyglot repository look different from a naive `wc -l`.

How Tokei parses files, ignores paths and parallelises work

Tokei does not count lines with a regular expression over raw text. The README claims it "correctly handles multi line comments, nested comments, and not counting comments that are in strings." That requires a per-language lexer, and the repository supports the claim structurally: languages.json sits at the top level, build.rs generates code from it at build time, and the Cargo.toml build-dependencies include tera, ignore, serde_json and json5. So the language table is data, compiled into the binary, not a hardcoded match statement.

File discovery is delegated to the ignore crate, which is the same library behind ripgrep. That is why Tokei respects .gitignore and .ignore files automatically, and why the --exclude flag is documented as having "the same semantics as .gitignore". A .tokeignore file in the tree adds project-specific exclusions using the same gitignore syntax.

Parallelism comes from rayon and crossbeam-channel, both listed as dependencies. The practical consequence is that Tokei is bound by filesystem traversal and by how many cores you give it, not by parsing speed on a single file. Files are also read through encoding_rs_io, which means non-UTF-8 sources are transcoded rather than skipped.

The release profile sets lto = "thin" and panic = "abort". Thin LTO keeps build times reasonable while still inlining across crates; panic = "abort" removes unwinding tables and makes the binary smaller. Neither choice affects counting accuracy, but both explain why the distributed binary is small and why a panic in a parser terminates the process instead of being caught.

Installing Tokei and running a first count

The README lists package managers for Unix, macOS and Windows. On macOS the Homebrew formula is the shortest path:

brew install tokei

After that, `tokei --version` should print the installed version. On Windows the README gives winget and scoop:

winget install XAMPPRocky.tokei

If no package manager fits, the README points at prebuilt binaries in the releases section, or a source build via cargo:

cargo install --git https://github.com/XAMPPRocky/tokei.git tokei

The README notes that building from source "requires the latest stable Rust compiler", and Cargo.toml pins rust-version = "1.71", so anything older than that will refuse to build.

A first real run is one argument. The README's basic example reports on ./foo and all subfolders:

tokei ./foo

You should see the bordered table from the README, with one row per detected language and a Total row at the bottom. To get per-file rows instead of per-language totals, add --files:

tokei ./foo --files

To sort by a column rather than alphabetically, pass --sort with one of blanks, code, comments or lines:

tokei ./foo --sort code

For machine consumption, the README states Tokei can output CBOR, JSON and YAML. Those formats are behind cargo features: Cargo.toml defines cbor, yaml, and an all feature that enables both, with cli as the default. A binary installed without those features will not accept the corresponding output flags.

Where Tokei gets the count wrong or stops being the right tool

The accuracy claim is bounded by the language definitions, not by the parser engine. languages.json is generated and maintained by hand, so a language whose comment syntax is unusual, or a file extension shared by two languages, can be classified incorrectly. When a number looks wrong, the first thing to check is whether your extension is in that file and which language it maps to.

Nested output is a second source of confusion. In the README example, Markdown rows appear twice: once as a top-level language and again as a sub-row under Rust, with a (Total) line that sums both. If you parse the JSON without understanding that nesting, you will double count.

Tokei also has no memory. There is no database, no history, and no diff between runs. To compare this week against last week you must store the JSON or YAML output yourself and diff it. The README does mention reusing output, but only in the sense that a previous run's statistics can be combined with another set, not that Tokei tracks time.

Finally, Tokei counts lines, not effort. A 40,000-line generated file and a 40,000-line hand-written module are the same number. Teams that use line counts as a productivity metric will get exactly the misleading figure they deserve, and Tokei offers no mechanism to prevent that.

Tokei versus cloc, scc and other line counters

The obvious comparison is cloc, the long-standing Perl tool that Tokei's own keywords list as a tag. cloc runs on a Perl interpreter and ships a large set of language definitions in the same spirit. The difference in approach is the runtime and the concurrency model: Tokei is a compiled Rust binary that fans out across files with rayon, while cloc is interpreted and largely sequential. On a large monorepo that difference is visible in wall-clock time, which is the comparison the README points readers toward in the 11.0.0 release notes.

The second comparison is scc, another Go-based counter that also parallelises and also emits structured output. Both tools answer the same question, so the deciding factors are language coverage, output schema and whether the binary is already in your package manager. Tokei's edge is the library API: Cargo.toml exposes the crate with a cli feature, so the same counting engine can be embedded in a Rust program without shelling out.

A third option is simply `git ls-files | xargs wc -l`. It is always available, needs no install, and is wrong in a specific way: it counts comments and blanks as code and has no notion of nested languages. That is fine for a rough size estimate and useless for a report you intend to publish.

Maintenance, licensing and the cost of upgrading

The repository is not archived, and the last push was on 2026-09-06, the same day v15.0.0 was tagged. The release cadence visible in the data is roughly annual for major versions: v13.0.0 in November 2025, v14.0.0 in December 2025, v15.0.0 in September 2026. Major-version bumps in a CLI of this kind usually mean flag or output changes, so pinning the version in CI is worth the effort.

Upgrade cost is low for the binary itself, since it has no runtime dependencies and no daemon. The real cost sits in two places. First, output schema: if you parse JSON or YAML into a dashboard, a major version can change field names and break the parser. Second, language definitions: a new release may reclassify extensions, which silently shifts your totals without any change on your side.

Licensing is dual, MIT OR Apache-2.0 per Cargo.toml, with LICENCE-MIT and LICENCE-APACHE both present at the repository root. That is the standard Rust ecosystem arrangement and is permissive for both source and binary distribution. The repository metadata reports the licence as NOASSERTION, which reflects a limitation of the metadata rather than the project: the crate manifest is explicit. If your legal review requires a single identifier, point it at Cargo.toml and the two LICENCE files rather than at the repository badge.

Editorial conclusion

Adopt Tokei if you want a single binary that walks a tree, respects .gitignore and .tokeignore, and emits JSON, YAML or CBOR for a dashboard. Do not adopt it if you need per-author attribution, historical trends or a hosted service; Tokei has none of those. Before rolling it into a pipeline, verify which of the 150+ language definitions in languages.json match your extensions, and check that the required Rust version (rust-version = "1.71" in Cargo.toml) is available if you build from source rather than using a package manager.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. XAMPPRocky/tokei on GitHub