Scattertext: Finding and Plotting the Terms That Separate Two Corpora

The problem Scattertext addresses: two corpora, one readable plot

Comparing two sets of documents is easy to state and awkward to do well. You can rank terms by frequency difference, but a ranked list hides how a term is distributed, and it gives no sense of which terms are common on both sides versus which are lopsided. Scattertext's answer is positional. Each term becomes a point whose x and y coordinates are its usage rank in one category and in the other, so the diagonal is the space of terms used equally by both sides, and the off-diagonal corners are where the distinguishing vocabulary lives. The README frames the project as "a tool for finding distinguishing terms in corpora and displaying them in an interactive HTML scatter plot," with points selectively labeled so labels do not collide with each other or with points. The audience is computational social scientists, stylometrists and anyone doing exploratory text analysis who wants the term-level picture rather than a single accuracy number. The 2012 American political conventions example in the README, where the 2,000 most party-associated unigrams are plotted by dense rank of usage among Democratic and Republican speakers, is a fair picture of the intended use: two labeled groups, a vocabulary worth inspecting, and a reader who wants to click around.

How a corpus becomes a plot: CorpusFromParsedDocuments and the compactor

The pipeline in the README's opening example is short. You take a DataFrame, add a parsed column by applying a tokenizer such as st.whitespace_nlp_with_sentences to the text column, then call st.CorpusFromParsedDocuments(df, category_col='party', parsed_col='parse').build(). From there the example calls get_unigram_corpus() and then compact(st.AssociationCompactor(2000)), which trims the vocabulary to the 2,000 terms that matter for the association being studied. That compaction step is the part worth understanding before you trust a plot. Without it, a large corpus produces a scatter plot with tens of thousands of overlapping points, and the selective labeling cannot rescue readability. The category argument to produce_scattertext_explorer takes one value ('democrat') and the other side is named through not_category_name='Republican', so the function is built around a binary contrast rather than an arbitrary number of groups. The axes are controlled by transform=st.Scalers.dense_rank, which converts raw frequency into a rank-based scale. Dense rank is a deliberate choice: raw frequency would put almost every term in the bottom corner and a handful of function words at the top, whereas rank spreads the vocabulary out across the plot. That is also why the README has a dedicated section on understanding the scaled F-score, because the default scoring and the axis scaling are two separate decisions that both shape what you see.

Getting it running, and what the install actually pulls in

Installation is a single command per the README: pip install scattertext, on Python 3.11 or higher. The README also states that Scattertext should mostly work with Python 2.7, but may not, which is a hedge rather than a support statement. The optional dependencies are where the real setup cost sits. The README recommends jieba, spacy, empath, astropy, flashtext, gensim and umap-learn in order to take full advantage of the library, and those packages enable specific features rather than general operation: jieba for Chinese segmentation, empath for topic and category visualization, gensim and umap-learn for the word-embedding projection plots, astropy somewhere in the plotting stack. If you skip spaCy, the README gives the escape hatch plainly: substitute nlp = spacy.load('en') lines with nlp = scattertext.WhitespaceNLP.whitespace_nlp, and it warns that this is not compatible with word_similarity_explorer and that tokenization and sentence boundary detection become low-performance regular expressions. There is a demo_without_spacy.py referenced as an example. The output side is a string of HTML that you write to a file yourself, as in open('./demo_compact.html', 'w').write(html). Nothing is rendered server-side, so the artifact is portable, but the README notes the HTML outputs look best in Chrome and Safari.

The visualization options are the product, and they are numerous

The table of contents reads like a menu of scoring and rendering choices, and that breadth is the honest description of the library's scope. Beyond the default scaled F-score, the README documents Cohen's d, Hedge's g, Cliff's delta and bi-normal separation as alternative term-scoring methods, each with its own section. There are document-based scatter plots, phrase associations rather than single terms, color gradients that explain what a score means, and a gradient labeling scheme in the example that names the two ends 'More Republican' and 'More Democratic' with 'Metric: Dense Rank Difference' in the middle. The advanced uses section covers custom term positions, emoji analysis, SentencePiece tokens, scikit-learn classification weights, lexicalized semiotic squares, topic models, T-SNE-style embedding projections, SVD on arbitrary embeddings, matplotlib export, and a same-scale-for-both-axes option. Two of those deserve emphasis for anyone deciding whether this fits. The semiotic squares section means the library is not limited to a single left-right contrast. The matplotlib export matters because it is the escape route for people who need a static figure. There is also a position-select-plot process described as its own section, which is the library's framing of how you move from term positions to a chosen plot, and a note on chart layout that acknowledges the layout is a design problem the project has opinions about.

Where the approach breaks down

The binary framing is the first constraint. produce_scattertext_explorer takes one category and one not-category, so a three-way comparison has to be decomposed into pairwise runs, and the pairwise runs are not independent of each other. The second constraint is the rank transform itself. Dense rank is what makes the plot legible, but it also means the axes are not frequencies, and a reader who interprets distance on the plot as a frequency gap will be wrong. The README's own decision to include a section on understanding the scaled F-score suggests the default score is not self-explanatory either. Third, the browser dependency is real: the output is HTML that the README says looks best in Chrome and Safari, so a PDF-bound workflow needs the matplotlib export path instead. Fourth, the spaCy-free fallback is a genuine downgrade rather than a neutral alternative, since the README describes the tokenizer and sentence boundary detection as low-performance regular expressions and states it is incompatible with word_similarity_explorer. For morphologically rich languages or text where sentence boundaries matter to your analysis, that fallback is the wrong tool. Finally, the release list shows 0.0.2.4.4 dated 2017-03-13 while the README header reads Scattertext 0.2.3, and the last push is 2026-07-04. The versioning is not obviously linear from the material available, so treat the README's version string as documentation of a state, not as a guarantee about what pip resolves.

Alternatives and the difference in approach

The most direct alternative in the same ecosystem is a term-frequency ranking done by hand in pandas, or a log-odds ratio with an informative prior, which gives you a sorted table of distinguishing terms and nothing else. The difference is not accuracy, it is the artifact. A ranked table forces the reader to trust the ranking; Scattertext's plot lets the reader see that a term is high on one axis and moderate on the other, and the selective labeling keeps the crowded regions readable. For embedding-based exploration, the README's own T-SNE and SVD sections point at the other family of tools, where terms are placed by vector similarity rather than by category frequency. Those answer a different question: they cluster terms that behave similarly, while Scattertext positions terms by how differently the two categories use them. If your goal is to explain a classifier, the scikit-learn weights section is the relevant path, and it is a distinct workflow from the default scaled F-score. If your goal is a static figure for a paper, matplotlib export is the alternative to the HTML explorer, and it trades interactivity for reproducibility in a build pipeline. None of these is a drop-in replacement, and the choice should follow from whether your reader needs to interrogate the plot or just cite it.

Licence, maintenance and what to check before you commit

Scattertext is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant, but it also carries notice and attribution obligations, and the README asks that you cite Jason S. Kessler, Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ, ACL System Demonstrations, 2017, with a BibTeX entry provided. That citation request is separate from the licence and is a scholarly norm rather than a legal term. On maintenance, the repository is not archived, the primary language is Python, and the last push is 2026-07-04, so the project is being touched. The release list, however, shows only 0.0.2.4.4 from 2017, which means the release channel and the repository activity are not obviously in step. The practical implication is that you should pin a version explicitly in your requirements file rather than rely on an unpinned pip install scattertext resolving to something the README describes, and you should confirm which of the optional dependencies your chosen version actually needs. The dependency surface is the other cost to budget for: spaCy, gensim, umap-learn, empath, astropy, flashtext and jieba are a heavy install, and each one has its own release cadence and its own breakage risk. If your environment is locked down, check that the packages your features require are available before you build the corpus, because the fallback tokenizer changes the analysis rather than just the convenience.

Editorial conclusion

Adopt Scattertext if you have a two-category corpus already sitting in a DataFrame and you need an interactive artifact you can hand to a collaborator who does not write Python, since the output is a self-contained HTML file. Do not adopt it if you need a static, publication-quality figure produced inside a matplotlib pipeline, or if your categories are not meaningfully comparable in size and vocabulary, because the scaled F-score and the dense-rank axes both assume that comparison makes sense. Before committing, verify the tokenization path: run the pip install scattertext command, decide whether you will install spaCy or fall back to scattertext.WhitespaceNLP.whitespace_nlp, and check that the fallback's regular-expression tokenizer and sentence boundary detection are good enough for your language. The library also carries a 2017 ACL System Demonstrations citation and a 0.0.2.4.4 release from 2017 on the release list even though the README header reads 0.2.3, so pin the version you install and read the version's own notes rather than assuming the README describes what pip will fetch.

Official sources

1. Issues
2. JasonKessler/scattertext on GitHub
3. License: Apache-2.0
4. README
5. Releases