shorttext: an intermediate-representation toolkit for classifying very short documents
Various Algorithms for Short Text Mining
At a glance
- What is it?
- PyShortTextCategorization ships the shorttext package, which turns short strings into topic-model or word-embedding vectors before handing them to a classifier. It is a research-oriented library with a broad algorithm menu and a narrow set of guarantees.
- Who is it for?
- Adopt shorttext if your inputs are genuinely short (keywords, titles, subject headings) and you want topic-model or embedding representations feeding scikit-learn classifiers without writing the vectorization layer yourself. Do not adopt it if your texts are long documents, or if you need a maintained API contract: the release history shows a steady stream of major version bumps, so pin an exact version and read the changelog before upgrading.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The sparsity problem shorttext was built around
A tweet, a subject heading, a product title: these carry too few tokens for a bag-of-words matrix to be anything but nearly empty. The README states the motivation directly, that due to the sparseness of words and the lack of information carried in the short texts themselves, an intermediate representation is needed before any classification algorithm sees them. That sentence is the whole design premise. shorttext is not a classifier library that happens to accept text. It is a representation library that sits between raw short strings and whatever model you already trust.
The audience follows from that. If you are doing supervised categorization of short strings and you have already decided that TF-IDF plus logistic regression is too thin a signal, this package offers the next step up: latent topic distributions and dense embedding vectors, produced through a consistent interface. If your texts are paragraphs or full documents, the premise does not apply to you and the extra machinery is overhead.
What the representation pipeline actually contains
The README lists three families of intermediate representation. First, gensim topic models: LDA, LSI and Random Projections, plus an autoencoder. Second, pre-trained word-embedding support. Third, character-level sequence-to-sequence learning. Around those sit the parts that make the representations usable: text preprocessing, cosine distance classification, maximum entropy classification, and neural network classifiers including ConvNet and C-LSTM.
The integration point worth noting is that topic model representations are supported for supervised learning using scikit-learn. That means the topic vector becomes a feature matrix you can pass into a scikit-learn estimator, rather than a separate modelling universe. The package also ships phrase-difference metrics, including a soft Jaccard score built on Damerau-Levenshtein distance and Word Mover's distance, plus spell correction. Those metrics matter for short text specifically, because a one-character edit in a five-word string changes a large fraction of the content, and exact-match similarity collapses under that.
What the README does not give is the call signature of any of these classes, the shape of the intermediate matrices, or guidance on choosing between LDA and an autoencoder for a given corpus. That detail lives in the linked tutorial and FAQ pages, which are outside the material available here. Treat the README as an inventory, not a manual.
Getting it installed and what the install pulls in
The installation instructions are two lines. From a console, pip install shorttext. If you want the development version from the default branch, pip install git+https://github.com/stephenhky/PyShortTextCategorization@master. The README points to an installation guide for more detail, and the package is published on PyPI under the name shorttext with the project homepage at https://pypi.org/project/shorttext/.
The support matrix is stated plainly: the package runs on Python 3.11, 3.12 and 3.13. Note what is absent. There is no mention of 3.9 or 3.10 support in the README, so if you are pinned to an older interpreter you should check the PyPI metadata before assuming compatibility. The dependency surface is also not enumerated in the README, but the feature list names gensim and scikit-learn as load-bearing, so the install will drag in a scientific Python stack. On a constrained container image that is a real cost, and it is the kind of thing to measure before committing.
The README shows the install command with a leading >>> prompt, which is a documentation style rather than a shell transcript. Run it without the prompt.
Version churn is the maintenance cost to budget for
The release log in the README runs from 1.1.0 in March 2019 to 4.0.4 in July 2026, and the cadence is high: 4.0.0 in April 2026, 4.0.1 in May, 4.0.2 in May, 4.0.3 in May, 4.0.4 in July. Three major version increments landed in 2025 and 2026 (3.0.0 in August 2025, 4.0.0 in April 2026), with 2.0.0 in July 2024 before that. For a library that wraps gensim topic models and scikit-learn estimators, major bumps usually mean the wrapper API moved, not just the internals.
That has a concrete operational consequence. Pin an exact version in your requirements file and read the changelog before moving. A floating dependency on shorttext will hand you a breaking change on an ordinary rebuild. The upside of the cadence is that the project is active: the last push recorded is September 2026 and the repository is not archived, so issues and pull requests are being seen. The downside is that any tutorial or blog post older than the current major version may not match the installed API.
The licence is MIT. That is permissive and short, and it is the kind of licence most teams can adopt without a review cycle, but the usual caveat applies: read the LICENSE file in the repository and get your own legal read if the package ends up inside a distributed product. Nothing here is legal advice.
Where shorttext is the wrong tool
The name is the constraint. This is a short text package, and the README's own justification is that short texts lack the information needed for direct classification. Feed it long documents and you get topic models applied to a corpus that did not need them, plus the dependency weight of gensim and the neural components for no gain.
There is a second, less obvious failure mode. The package offers a wide menu: LDA, LSI, Random Projections, an autoencoder, pre-trained embeddings, ConvNet, C-LSTM, maximum entropy, cosine distance, soft Jaccard, Word Mover's distance, seq2seq, spell correction. Breadth like that is a research-library signature. It means each path is implemented and documented to the depth its author needed, not to the depth a production team needs. The README gives no benchmarks, no accuracy figures, and no guidance on which combination works for which corpus size. If your decision process requires an expected-accuracy number before you commit, this package will not supply it, and you should treat the algorithm list as a set of things to evaluate yourself rather than a set of recommendations.
The bundled example data (subject keywords and NIH RePORT) is the intended starting point. If neither resembles your data, you are on your own for validation.
How this differs from reaching for a transformer encoder
The obvious alternative for short text classification today is a pre-trained transformer encoder fine-tuned on your labels. The difference in approach is not just accuracy, it is where the work happens. A transformer brings its own representation: you feed raw strings and the model's pretraining supplies the semantics. shorttext makes you build the representation explicitly, choosing LDA or LSI or an autoencoder, then hands that matrix to scikit-learn. You keep control of the intermediate object and you can inspect it. A topic distribution over twenty topics is readable in a way that a 768-dimensional contextual embedding is not.
That control is the reason to pick shorttext over a transformer when your corpus is small. Fine-tuning an encoder on a few thousand short labelled strings is a good way to overfit, and the README's emphasis on topic model representation supported for supervised learning using scikit-learn points at exactly the regime where a linear model over a low-dimensional topic vector is competitive. The trade-off is that you now own the vectorization step, including its hyperparameters, and the package does not tell you how to set them.
Against plain TF-IDF plus a linear classifier, the difference is narrower. shorttext is the layer you add when TF-IDF has plateaued and you want to test whether a denser representation helps. It is not a replacement for TF-IDF on corpora where TF-IDF already works.
Who should install it, and what to check before you do
The fit is a team with a short-text categorization problem, an existing scikit-learn workflow, and an appetite for experimenting with representations rather than buying a hosted model. The example data and the tutorial give you a path to a first result. The MIT licence keeps the paperwork light. The active release cadence means bugs get attention.
The misfit is a team that needs a stable API across a multi-year product, or that has long documents, or that cannot absorb a gensim and scikit-learn dependency tree. For those cases the package's own premise argues against it.
Before committing, check three things in order. Confirm your interpreter is 3.11, 3.12 or 3.13 as the README states, since older versions are not listed. Run pip install shorttext in a clean environment and record what it resolves to, because the README does not enumerate dependencies. Then open the tutorial and FAQ pages linked from the README and confirm the class names and call patterns match the version you pinned, since the major version history suggests they have moved more than once.
Editorial conclusion
Adopt shorttext if your inputs are genuinely short (keywords, titles, subject headings) and you want topic-model or embedding representations feeding scikit-learn classifiers without writing the vectorization layer yourself. Do not adopt it if your texts are long documents, or if you need a maintained API contract: the release history shows a steady stream of major version bumps, so pin an exact version and read the changelog before upgrading. Verify first that your Python version is 3.11, 3.12 or 3.13, that pip install shorttext resolves cleanly in your environment, and that the example datasets bundled with the package cover a shape close to your own data.
Community notes