# js-tokens: a tokenizer that never fails on invalid JavaScript

> js-tokens is a MIT-licensed JavaScript lexer built on regular expressions that never throws and never stops, producing a stream of token objects for any input including broken code, and it passes all but three of the test262 parser tests. It suits a tool that has to make sense of source it did not write, and its regex versus division heuristic is the one place it can be wrong.

**lydell/js-tokens** — Tiny JavaScript tokenizer.

- Repository: https://github.com/lydell/js-tokens
- Stars: 555 · Forks: 39
- Language: JavaScript
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/lydell-js-tokens

## The promise, and what it means in practice

js-tokens describes itself as the tiny, regex powered, lenient, almost spec-compliant JavaScript tokenizer that never fails.

The usage is one function:

```js
import jsTokens from "js-tokens";

const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';

Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)
```

The contract has three parts and they are all unusual for a lexer. For the empty string the function yields nothing, which can be turned into an empty list. For any other input it always yields something, even for invalid JavaScript, and it never throws. And concatenating the token values reproduces the input exactly.

That last property is the one that makes the library useful for the job it is actually good at. A tokenizer that always terminates and always round-trips can be run over anything, including a file that is half-written, and the result is still a faithful account of the characters. A conforming parser would refuse.

The package exports a generator function, jsTokens, that turns a string of JavaScript code into token objects, and it accepts a single options object with one documented option, jsx, a boolean defaulting to false, which enables JSX support.

The compliance claim is specific rather than vague. The package is very close to being fully spec compliant, and it passes all but three of the test262-parser-tests, with the exceptions listed in a known-errors section. That is a measurable claim against the standard test suite, and it is the right way to state a lexer's quality.

The project is MIT licensed, the primary language is JavaScript, the default branch is main, and the last push to the repository was on 2026-06-05. There are no GitHub releases, and the package is published to npm as js-tokens.

## The token vocabulary, and how invalid input is represented

The token type is a discriminated union, and the type list is short enough to read in one go.

```ts
export default function jsTokens(input: string): Iterable<Token>;

type Token =
  | { type: "StringLiteral"; value: string; closed: boolean }
  | { type: "NoSubstitutionTemplate"; value: string; closed: boolean }
  | { type: "TemplateHead"; value: string }
  | { type: "TemplateMiddle"; value: string }
  | { type: "TemplateTail"; value: string; closed: boolean }
  | { type: "RegularExpressionLiteral"; value: string; closed: boolean }
  | { type: "MultiLineComment"; value: string; closed: boolean }
  | { type: "SingleLineComment"; value: string }
  | { type: "HashbangComment"; value: string }
  | { type: "IdentifierName"; value: string }
  | { type: "PrivateIdentifier"; value: string }
  | { type: "NumericLiteral"; value: string }
  | { type: "Punctuator"; value: string }
  | { type: "WhiteSpace"; value: string }
  | { type: "LineTerminatorSequence"; value: string }
  | { type: "Invalid"; value: string };
```

The design decision visible here is the closed flag. Strings, templates, regular expressions and multi-line comments all carry a closed boolean, and it is false when the terminator is missing. That is how the tokenizer represents a truncated file without inventing structure: the token is still a string literal, it just did not end properly.

The Invalid type is the escape hatch for everything the tokenizer cannot classify, and it is the token type that makes the never-throw promise work in practice.

The lexical rules are the specification's rules with two documented shortcuts. A string literal whose closing quote is missing has closed set to false, and since JavaScript strings cannot contain unescaped newlines, an unclosed string simply ends at the end of the line. Escape sequences are supported but may be invalid, so a string containing an invalid escape is still matched as a StringLiteral.

Templates follow the same pattern. A template without interpolations is matched as is, and one with interpolations is matched as several tokens, a TemplateHead, then a TemplateMiddle for each interpolated section, and a TemplateTail. The TemplateMiddle is optional and the TemplateTail can be unclosed, and templates can contain unescaped newlines so an unclosed template runs to the end of the input.

The identifier handling is where the tolerance shows most clearly. IdentifierName covers keywords, reserved words, null, true and false, variable names and property names, and the examples include the full Unicode range, from the Greek pi to the Kannada letter, and an escaped form that is written out in Unicode escapes.

## The one heuristic, and where it can be wrong

There is exactly one place where js-tokens has to guess, and the documentation is honest about it.

Differentiating between regex and division in JavaScript is really tricky, and js-tokens looks at the previous token to tell them apart. As long as the previous tokens are valid it should do the right thing. For invalid code, js-tokens might be confused and start matching division as regex or vice versa.

That is the whole story of the difficulty. In JavaScript, a slash can begin a regular expression literal or be a division operator, and the grammar cannot decide without context, because the rule depends on what the parser expects next. A tokenizer that only sees the character stream has to guess from the previous token, and the guess is reliable in valid code and unreliable in broken code.

The consequence is worth stating precisely. If your use case is to tokenize arbitrary input including invalid input, and you rely on the token types to decide what the text is, then a single misclassified slash propagates: a division operator read as the start of a regex swallows text until the next slash, and the rest of the file is one token.

The rest of the regex handling is faithful to the specification, including its oddities. Regex literals may contain invalid regex syntax and are still matched as regex literals, and the documentation quotes the specification on why, noting that the productions are used by the input element scanner to find the end of the regular expression literal, and that the source text is subsequently parsed again using the more stringent ECMAScript regular expression grammar. If the closing slash is missing the token has closed set to false, and since JavaScript regex literals cannot contain newlines, not even escaped ones, an unclosed regex ends at the end of the line. The flags are IdentifierPart characters according to the specification, so unknown and repeated flags are errors at a later stage rather than at tokenization time.

## How it is built and tested, and the unusual source language

The repository is small, and one detail stands out immediately: the implementation is written in CoffeeScript.

The tree contains index.coffee as the implementation, index.d.ts as the hand-written type definition, index.test-d.ts for the type tests, a build.js that compiles the source, a benchmark.js, a package.json that is marked private, and a package-real.json that appears to hold the published package metadata. There is a test/ directory, a vitest.config.js, an eslint.config.js, a .prettierrc.json, a CHANGELOG.md and a LICENSE file.

The CoffeeScript choice is interesting. A regular expression driven lexer is a piece of code whose behaviour is much easier to specify in examples than in a type system, and CoffeeScript's terse syntax and its regular expression literals keep such a file readable. The cost is a build step, and the build script exists for that reason.

The test script shows the whole verification chain in one line: it runs eslint with the report-unused-disable-directives flag, checks formatting with prettier, runs the build, runs tsd for the type tests, runs the benchmark, and then runs vitest.

The dev dependencies explain the rest. The test262-parser-tests package at version 0.0.5 is the conformance suite referenced in the documentation, and its presence as a dependency is what makes the claim of passing all but three of the parser tests checkable rather than aspirational. The @babel/parser package is there for comparison, most likely as a reference implementation in the tests. tsd checks the type definitions against the test file, vitest 4 with the coverage-v8 provider runs the unit tests, and everything.js is the assert-and-compare helper. Prettier and ESLint with the typescript-eslint utilities handle style, and globals provides the environment definitions.

Running a benchmark as part of the test script is a small choice with a large effect on a library whose selling point is that it is tiny, because a regression in throughput is caught by the same command that catches a broken test.

## Where js-tokens is the wrong tool

The first limit is that a tokenizer is not a parser. A token stream tells you what the characters are, not what the program means, so any question that requires structure, such as what is the value of this expression or which function does this call reach, needs a real parser built on top. The library is a first stage, and using it as if it were the whole job means writing the rest yourself.

The second is the regex and division heuristic, which is the one documented failure mode. On valid input it is reliable, and on invalid input it can swallow text. If your application is specifically about broken or partial JavaScript, for example a live editor that must re-lex a file after every keystroke while it is mid-edit, then the failure case is the normal case, and the parts of a file that follow a misclassified slash will be reported as one large token.

The third is that JSX support is off by default and is a single boolean. The library is a JavaScript tokenizer, and JSX is a syntax extension whose handling depends on where the parser is in the file, and a lexer that turns it on with a flag is making a simplification that a project using JSX heavily will want to understand.

The fourth is the CommonJS entry point requirement. The documentation notes that the require form requires Node.js 20.19.0 or later, which is a recent enough floor that it excludes some long-lived server environments, and it is a floor that is easy to overlook because the package itself has no engine constraint in the published manifest to warn you.

And the fifth is the maintenance profile. The project has no GitHub releases, the last push was on 2026-06-05, and the implementation is a CoffeeScript file that has to be compiled before it can be published. A library this small is unlikely to need changes often, and the token grammar it implements is a specification, so a frozen version is not a liability. But a consumer pinning a version should be aware that the package is a fork's worth of decisions away from the specification, and the three known test262 exceptions are the place to look first.

## Conclusion

js-tokens is worth using when you need to look at JavaScript source you did not generate, because it never throws and never stops, so a truncated or corrupted file still yields a token stream rather than an exception, because the token values concatenate back to the exact input, and because the near-spec compliance means a well-formed file is tokenized the way the standard says it should be. It is the wrong tool if you need a parser, since a token stream is not an AST, and it is the wrong tool if you need to reject invalid code. Before you depend on it, check three things: that you can live with the one documented heuristic, because differentiating between a regular expression and division requires looking at the previous token, and on invalid code the tokenizer might match division as regex or the other way around; that you know the JSX support is opt-in, since the jsx option defaults to false and enabling it changes how the input is scanned; and that you check the Node.js requirement for the CommonJS entry point, because the documentation says the require form requires Node.js 20.19.0 or later, which is a floor that will rule out some older server runtimes.

## FAQ

### How do I install and use js-tokens?

Run npm install js-tokens, then import it with import jsTokens from "js-tokens", or use const jsTokens = require("js-tokens") which requires Node.js 20.19.0 or later. Call jsTokens(string, options?) to get an iterable of token objects, with a single documented option, jsx, a boolean defaulting to false that enables JSX support.

### What happens when js-tokens is given invalid JavaScript?

It never throws and never stops. For any non-empty input it always yields something, even for invalid JavaScript, and concatenating the token values reproduces the input exactly. Tokens that cannot be classified come back as type Invalid, and tokens whose terminator is missing, such as an unclosed string or comment, are returned with closed set to false.

### How does js-tokens tell a regular expression apart from a division operator?

By looking at the previous token. The documentation says that as long as the previous tokens are valid it should do the right thing, and that for invalid code js-tokens might be confused and start matching division as regex or vice versa. That is the one documented heuristic in an otherwise specification-faithful lexer.

## Sources

- [Issues](https://github.com/lydell/js-tokens/issues)
- [License: MIT](https://github.com/lydell/js-tokens/blob/main/LICENSE)
- [lydell/js-tokens on GitHub](https://github.com/lydell/js-tokens)
- [README](https://github.com/lydell/js-tokens/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/lydell-js-tokens