quanteda is an R package where a corpus is a first-class object

A corpus, and everything hangs off it

quanteda is an R package for managing and analyzing text, created and maintained by Kenneth Benoit and Kohei Watanabe, and the project site is quanteda.io.

The framing in the repository is a corpus-first design, and the family of packages makes that concrete. The core quanteda package contains all of the core natural language processing and textual data management functions. Everything else was split out of it. quanteda.textmodels holds the text models and their supporting functions, the textmodel_*() family, split with the v2 release. quanteda.textstats holds statistics for textual data, the textstat_*() functions, split with v3. quanteda.textplots holds the plots, the textplot_*() functions, also split with v3.

The README is direct about the pattern: the trend of splitting quanteda into modular packages was completed with the release of v3. So what is left in the core package is the part every other package depends on, which is the handling of documents and their features.

Two further packages are in development and available from the project's GitHub pages for now. quanteda.sentiment provides functions and lexicons for sentiment analysis using dictionaries, and quanteda.tidy provides extensions for manipulating document variables in core quanteda objects using tidyverse functions. That last one is worth noting for anyone who arrives from the tidyverse, because it is the bridge between quanteda's document-feature model and the data frame world.

The result is a package family with a clear division of labour, and a migration story: the split happened across two major versions, and code written against quanteda 1.x or 2.x will need to know which package a function moved to.

Version 4 changed the tokens and the tokeniser

The 4.0 release is described as a major release that improves functionality and performance and further improves function consistency by removing previously deprecated functions.

The headline technical change is a new external pointer tokens object, and the README links to an article about it, an article showing performance benchmarks for the new external pointer tokens objects along with some of the tokeniser improvements, and the v4 changelog in NEWS.md as a full listing of the changes, improvements and deprecations.

The tokeniser rewrite is the other half. The README says version 4 includes significant new tokeniser rules that make the default tokeniser smarter, with new Unicode and ICU-compliant rules enabling it to work more consistently with even more languages. ICU compliance is the substantive part for a text analysis package, because tokenisation is where a corpus is either comparable across languages or quietly not.

The removal of previously deprecated functions is the part that affects an existing user most, and it is the reason the changelog is linked so prominently. A project that upgrades from 3.x to 4.x may find that a function it was still calling has gone, with a replacement in a different package. The README's own framing, further improves function consistency, is the maintainers' justification, and it is a reasonable one, but it is a breaking change and should be planned for rather than discovered.

The current CRAN version is 4.5.0, released on 2026-08-05, with 4.3.0 in May 2025 and 4.2.0 in January 2025. The development version badge reads 4.5.1, and the last push to the repository was on 2026-10-15.

Installing on Linux needs TBB before anything else

The normal installation path is CRAN, either through an R GUI or from a console:

install.packages("quanteda")

There is a Linux caveat that catches people out, and the README flags it as new in quanteda v4.0. Because all installations on Linux are compiled, Linux users need the Intel oneAPI Threading Building Blocks for parallel computing installed first for the installation to work. The commands are given per distribution family:

# Fedora, CentOS, RHEL
sudo yum install tbb-devel

# Debian and Ubuntu
sudo apt install libtbb-dev

The reason is the next section rather than a packaging decision. quanteda takes advantage of parallel computing through the TBB library to speed up computations, and TBB is a native dependency rather than something the package can vendor.

The macOS and Windows route has the same underlying requirement, expressed differently, because compiling from source is needed for the development version there, and that pulls in C++ and Fortran compilers. On macOS the README asks for the Xcode command line tools, then TBB and pkg-config from Homebrew, and then gfortran:

xcode-select --install

brew install tbb pkg-config

On Windows the answer is RTools, which the README notes includes the TBB libraries. So a Windows user with a binary install from CRAN is fine, and a Windows user building the development version needs the toolchain.

The parallel guide is a setup document, not a footnote

Enabling parallelisation has its own section in the README, and it is worth reading as a setup guide rather than a performance note, because it changes what you have to install on every platform.

The TBB library is the mechanism: quanteda takes advantage of parallel computing through the Threading Building Blocks library to speed up computations, and the guide provides step-by-step instructions for setting up your system on Windows, macOS and Linux. The macOS path in that guide is the longer one, with four numbered steps, installing the Xcode command line tools, then Homebrew if it is not already there, then TBB and pkg-config, then gfortran through brew install gcc. The Homebrew bootstrap line in the guide is the standard one:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

What this section reveals is that parallel computation is not a runtime switch but a build-time dependency chain. TBB is linked in, and a machine without it either fails to install the package or loses the parallel path. A team provisioning analysis servers needs to bake TBB into the image, not discover it on the first large corpus.

The guide also points to more detail in the quanteda documentation on parallelisation, which is the right place to go if you are tuning this, since the README gives you the install steps and the documentation covers the configuration.

Where quanteda stops, and what to use instead

The honest limit of quanteda is that its unit of work is a corpus with a fixed vocabulary, and that assumption is what makes the matrix statistics fast and what makes it a poor fit for some problems.

A document-feature matrix is a good representation when the same vocabulary applies to every document and you want to count, weight, regress and model over the result. It is a bad representation when the text is a stream, when the vocabulary has to grow as you read, or when the analysis is a deep learning model over raw sequences. quanteda is a tool for the first case. Someone whose pipeline ends in a transformer over raw text will find quanteda useful for exploration and corpus construction, and awkward for the modelling step.

There is a second limit, and it is the one that catches newcomers. The package is a large surface with a long history, and the 4.0 release had to remove previously deprecated functions to make the naming consistent. A codebase written in the quanteda 1.x or 2.x idiom will not simply work on 4.x, and the split into four packages means a function you used may now live somewhere else entirely. The answer is to read the changelog before upgrading and to use the cheatsheet linked from the README, which is a PDF held in the tests directory of the repository and lists the function names by task.

The third limit is operational rather than conceptual. This is a package that compiles C++ and Fortran, links TBB, and needs a compiler toolchain on the platforms where binaries are not used. A team with a locked-down environment, or one that wants a container with no build step, has to arrange all of that. A pure-R alternative with no native dependencies would install more easily, and for many text analysis tasks the modelling you need is in a different package anyway.

A GPL-3.0 package with an academic funding lineage

The licence is GPL-3.0, with the full text in the LICENSE file at the top of the repository. For an R package used in research this is a common choice and it carries the usual implications for anyone redistributing it, particularly the network clause for code that is not distributed.

The funding history is documented in the README, which is unusual and interesting. Creation was funded by the European Research Council grant ERC-2011-StG 283794-QUANTESS, and continued development is supported by the Quanteda Initiative CIC, a community interest company hosted in the same GitHub organisation as the package.

That structure explains the shape of the project better than the feature list does. A package that has been developed for more than a decade on grant money and then sustained by a community interest company has every incentive to make the core stable, publish papers about the design decisions, and move peripheral functionality into separately funded packages. The v4 articles on the external pointer tokens object and on performance benchmarks are the kind of artefact a project produces when it wants its design decisions to be citable rather than merely documented.

The support channels are also worth noting for a package at this stage. The documentation is at quanteda.io, there is a quick start guide, the cheatsheet PDF, a StackOverflow channel for questions tagged quanteda, and the repository carries a Zenodo DOI badge and a Journal of Open Source Software DOI, which is the standard way an academic package gets a citable archival record.

The repository layout is a conventional R package with the pieces you would expect: R for the code, src for the compiled sources, man for the documentation, tests, vignettes, a DESCRIPTION file and a NAMESPACE, a pkgdown directory for the site, and a configure and configure.ac at the root for the platform-specific build steps that TBB and the compilers require.

Editorial conclusion

quanteda suits a researcher who works in R, has a corpus rather than a handful of documents, and wants document-feature matrices, models and statistics from one coherent object model. It is the wrong tool for a project that treats text as a preprocessing step for a machine learning model in another language, since the corpus and its vocabulary are the product here, and it is the wrong tool for anyone who cannot install a compiler, because the package compiles C++ and Fortran and needs TBB on Linux. Before you adopt it, read the v4 changelog in NEWS.md, because version 4 removed previously deprecated functions and changed the tokeniser, and check the cheatsheet in the tests directory for the function naming you will actually use.

Frequently asked questions

Official sources

1. License: GPL-3.0
2. Project website
3. quanteda/quanteda on GitHub
4. README
5. Releases