# vscode-textmate: running TextMate grammars outside the editor

> Microsoft's vscode-textmate is the grammar engine behind Visual Studio Code's syntax highlighting, packaged as a standalone library that tokenizes text using TextMate grammars and the oniguruma regex dialect. It is the right tool for editors, documentation tools and code highlighters that need scope-accurate tokens rather than colour output.

**microsoft/vscode-textmate** — A library that helps tokenize text using Text Mate grammars.

- Repository: https://github.com/microsoft/vscode-textmate
- Stars: 679 · Forks: 136
- Language: TypeScript
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/microsoft-vscode-textmate

## The grammar engine that Visual Studio Code itself uses

The README is short, and the first sentence carries the important claim: this is an interpreter for grammar files as defined by TextMate, and it is the library used in VS Code. That is a meaningful distinction from the many small syntax highlighting packages on npm. A TextMate grammar is a hierarchical rule set where each pattern assigns a scope name, and scopes nest. A regex highlighter returns a colour. A TextMate engine returns the scope stack for every token, and everything built on top, from colouring to semantic tokens to folding to bracket matching, derives from that structure.

So the audience is anyone who needs scope-accurate tokens rather than colours. If you are writing a documentation generator that wants to mark up code with the same scopes an editor would, a browser-based code viewer, a chat interface that highlights pasted code, or a review tool, this library gives you the same results the editor produces rather than an approximation.

The implementation is TypeScript, published to npm as vscode-textmate, with the repository owned by Microsoft Corporation. The licence is MIT, with the licence file named LICENSE.md at the root and a ThirdPartyNotices.txt alongside it, plus a SECURITY.md for reporting vulnerabilities.

The project is not archived, and the last push was on 2026-10-19, so the engine is still being worked on even though it is old enough to have a stable major line.

## Oniguruma is a required dependency, and it is a WebAssembly binary

The one hard constraint in this library is the regex engine. TextMate grammars use the oniguruma dialect, not JavaScript regular expressions, so the library cannot simply use the platform's regex. The engine is provided separately by vscode-oniguruma, which compiles oniguruma to WebAssembly and loads it in the host runtime.

The README's usage example shows the whole shape of the setup. The oniguruma binary is read from the release directory of the installed package, loaded as WebAssembly, and then wrapped in an object exposing two factory methods, createOnigScanner for patterns and createOnigString for the subject text. That object is what you hand to the registry as onigLib, and the library will not tokenize without it.

```js
const oniguruma = require('vscode-oniguruma');

const wasmBin = fs.readFileSync(path.join(__dirname, './node_modules/vscode-oniguruma/release/onig.wasm')).buffer;
const vscodeOnigurumaLib = oniguruma.loadWASM(wasmBin).then(() => {
    return {
        createOnigScanner(patterns) { return new oniguruma.OnigScanner(patterns); },
        createOnigString(s) { return new oniguruma.OnigString(s); }
    };
});
```

Two consequences follow. First, this is not a zero-dependency library, and in a browser the WebAssembly file has to be served to the client. Second, because the grammars assume oniguruma semantics, porting them to a JavaScript regex engine changes their behaviour, which is why the library takes this route rather than translating patterns.

The package metadata confirms oniguruma sits in devDependencies at ^1.5.1 rather than in dependencies, which is consistent with the README treating the oniguruma instance as something the consumer provides.

## The registry loads grammars lazily by scope name

The central abstraction is a Registry, constructed with an onigLib promise and a loadGrammar callback. The callback receives a scope name and returns either a promise of a raw grammar or null, and the registry takes care of pulling in whatever else the grammar depends on.

The README's example makes the behaviour explicit in a comment: it loads the JavaScript grammar and any other grammars included by it asynchronously. The output in that example is instructive because it shows the failure path, with a line reading Unknown scope name: source.js.regexp. The JavaScript grammar includes a regexp grammar by reference, and since the example's loadGrammar only knows about source.js, the include cannot be resolved. The engine continues, and the tokens for the function declaration still come out correctly, but the scope stacks are thinner than a full installation would give you.

That is the practical constraint on the loadGrammar callback: it is a resolver over a dependency graph of scopes, and a callback that only handles one scope name will produce degraded scope information for included grammars. Anyone porting grammars from an existing TextMate setup will need to answer for every scope those grammars reference.

The library also supports loading grammar files from JSON or PLIST format, and the example parses a PLIST file with parseRawGrammar, which is the format the textmate bundles in the wild use.

## Tokenizing is line by line, and the rule stack carries context

Tokenization happens one line at a time, and the state between lines is explicit. The example keeps a variable called ruleStack, initialised to vsctm.INITIAL, and after each line assigns the ruleStack that came back in the result. The grammar state machine therefore has to be carried by the caller, line by line, exactly as the editor does it.

This is the design detail that matters for correctness. A TextMate grammar can have rules that span lines, for example a block comment or a here-document, and a naive line-by-line tokenizer that resets state between lines would break them. Passing the returned ruleStack into the next call is what makes multi-line constructs work, and forgetting to do so is the most common mistake when writing an integration.

Each call returns a tokens array plus the ruleStack, and every token carries a startIndex, an endIndex and a scopes array. The scopes are the stack, outermost scope first, which is the shape you need for anything from colour selection to semantic token mapping. The example output shows a function keyword with scopes source.js, meta.function.js, storage.type.function.js, and a parameter with source.js, meta.function.js, variable.parameter.function.js.

Note that the scope stack is what makes the output useful. A highlighter that returns a single scope per token has already thrown away the information that a grammar author encoded.

## Cross grammar injections are the missing feature

The README states one limitation plainly: cross grammar injections are currently not supported.

This is worth understanding properly, because injections are how TextMate grammars share rules. An injection grammar declares that it applies inside another grammar's scope, which is how languages such as JavaScript get embedded highlighting for template literals, or how HTML gets embedded script and style highlighting, without either grammar knowing about the other. VS Code's own extension ecosystem relies on the mechanism heavily through injection grammars.

So if your grammars come from a set that includes injection grammars, those rules are simply not applied. The engine will tokenize, and the embedded-language highlighting will be missing, with no error. The example in the README is a version of this problem: the JavaScript grammar references a regexp grammar it cannot resolve, and the result is thinner scope information rather than a failure.

That constraint shapes who should use the library. If your grammar set is a small number of self-contained grammars, the absence of injections is a footnote. If you are porting a full TextMate bundle, or a VS Code extension's grammar contribution, the absence is a functional gap that you will not close without preprocessing the grammars yourself.

The alternative approach in the same ecosystem is to depend on a library that handles injections and grammar dependency resolution directly, at the cost of a heavier dependency and less control over the tokenization loop. The trade is between a small, explicit engine where you resolve grammars yourself and a larger one that does the resolution for you.

## Tooling for grammar authors, and the inspect command

The README points grammar authors at two external tools. vscode-tmgrammar-test is for writing unit tests against a grammar, and textmate-validate is for linting the regexes of a grammar using this library's own regex engine. The second is the more interesting one, since linting against the actual oniguruma implementation catches the patterns that look fine as JavaScript regexes and behave differently under oniguruma.

For debugging, the package scripts include an inspect command that takes a grammar path and a file path, invoked as `npm run inspect -- PATH_TO_GRAMMAR PATH_TO_FILE`.

Under the hood the script runs node out/tests/inspect.js, so it expects a compiled build in the out directory. This is the tool to reach for when a grammar produces unexpected scopes, since it isolates the tokenizer from your application code.

The API documentation is the source file itself. The README says to see the main.ts file for the API doc, and the repository root carries an api-extractor.json configuration, which is the tool Microsoft uses to roll up a public API surface from TypeScript declarations. The generated type definitions ship in the package as types/vscode-textmate.d.ts, so consumers get types without needing the source.

The test suite runs on vitest, and there is a test-cases directory at the root alongside a benchmark directory with its own script, node benchmark/benchmark.js.

## Version 9.3.2, an active repository, and a stable major line

The release history shows a mature package with a steady patch cadence. v9.3.0 was published on 2025-12-02, v9.3.1 on 2026-01-06, and v9.3.2 on 2026-01-22. The last push to the repository was on 2026-10-19, and the repository is not archived, so work continues on main even between tags.

That gap between the newest tag and the newest commit is worth understanding. The project publishes on a schedule that a consumer pinning to a version will not see, and the published package is 9.3.2 while main has moved on. If you are evaluating the library for a long-lived product, pin the version and read the changelog between 9.3.2 and main before assuming the fixes are in the published package.

The publishing pipeline is visible in the scripts. A prepublishOnly step runs the TypeScript compiler, then webpack with progress output, then api-extractor, so the published bundle is built and its public API is rolled up. The version script compiles and tests before the version bump, and postversion pushes the commit and the tags, so a release is always tested first.

The package ships only what consumers need. The files list includes the release JavaScript, excludes the release tests, and includes the type declaration file, which keeps the published tarball small. The MIT licence is permissive enough for commercial use without conditions, and ThirdPartyNotices.txt records the licensing of what is bundled.

The one operational note is the overrides block, which pins serialize-javascript to at least 7.0.5 and a specific minimatch version for one of its ranges. Those are the kind of transitive pins that indicate a dependency advisory was addressed, and they are worth watching in future releases.

## Conclusion

vscode-textmate fits teams building an editor, documentation site, chat interface or code review tool that needs the same scope-accurate tokenization Visual Studio Code uses, particularly when existing grammars are already in the TextMate format. It is the wrong choice for a small highlighter that only needs a regex, and for projects that depend on cross grammar injections, which the README states are not currently supported. Before adopting it, check whether your grammars rely on injections, confirm the oniguruma WebAssembly dependency is acceptable in your runtime, and read src/main.ts, which the README names as the API documentation.

## FAQ

### How do I install vscode-textmate?

The README gives the install command as npm install vscode-textmate, and the package is published under that name on npm.

### Does vscode-textmate need oniguruma to tokenize text?

Yes. TextMate grammars use the oniguruma dialect, so the library needs the vscode-oniguruma WebAssembly build loaded and passed in as the onigLib option of the registry.

### Does vscode-textmate support cross grammar injections?

No, the README states that cross grammar injections are currently not supported.

### How do I debug a TextMate grammar with vscode-textmate?

The package scripts include an inspect command, npm run inspect -- PATH_TO_GRAMMAR PATH_TO_FILE, which runs the inspect script against a compiled build.

## Sources

- [Issues](https://github.com/microsoft/vscode-textmate/issues)
- [License: MIT](https://github.com/microsoft/vscode-textmate/blob/main/LICENSE)
- [microsoft/vscode-textmate on GitHub](https://github.com/microsoft/vscode-textmate)
- [README](https://github.com/microsoft/vscode-textmate/blob/main/README.md)
- [Releases](https://github.com/microsoft/vscode-textmate/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/microsoft-vscode-textmate