adaptive-classifier: Continual Text Classification Without Retraining

The label set that keeps growing

Most text classifiers assume a closed world. You collect labelled data, fine-tune a transformer, and freeze the label set. The moment a new category appears, the model is wrong by construction, and the fix is a full retrain plus a redeploy. For support tickets, moderation queues, or router decisions, that cycle is the bottleneck.

adaptive-classifier targets that gap. It is a PyTorch library that wraps a HuggingFace transformer and keeps two things side by side: a prototype memory for similarity search and a trainable classification head. The README describes the combination as hybrid predictions, taking both prototype similarity and neural network output into account. New classes enter through the prototype side, which is why the project can advertise dynamic classes at runtime without retraining.

The audience is narrower than the tagline suggests. This is for engineers who already run a transformer encoder and need the label set to move. It is not a no-code classifier and it is not a replacement for a fine-tuned model on a stable taxonomy. The package classifiers in setup.py mark it Development Status 4 - Beta, which is the honest reading of a 0.2.0 release.

Prototype memory, an adaptive head, and what pooling does to both

The architecture is a two-path predictor over one embedding. Every text is encoded by a HuggingFace transformer into a single vector. That vector is stored in a FAISS index as a class prototype, and it is also fed to a trainable neural layer that produces logits. Prediction combines the two signals.

The prototype path is what makes runtime class addition cheap: a new class is a new set of vectors, not a new set of weights. The neural path is what makes the system more than nearest-neighbour search. The README names Elastic Weight Consolidation as the protection on the adaptive neural layer, which is the standard answer to catastrophic forgetting when you keep training on new examples.

The README devotes a section to pooling, and it is the most interesting design note in the repository. Before 0.2.0 the library always used the CLS token. That is wrong for most encoders: all-MiniLM-L6-v2 publishes pooling_mode_mean_tokens, and a masked language model never trains CLS as a sentence representation. The README reports a semantic-similarity check where CLS scored unrelated sentence pairs at 0.53 cosine similarity while mean pooling scored them at -0.03. Since the prototype path is cosine nearest-neighbour search over those vectors, a bad pooling choice degrades the whole memory, not just one score. From 0.2.0 the default is auto, reading the model's own sentence-transformers pooling configuration. If you are upgrading from 0.1.x, this is the change that matters.

Installing adaptive-classifier and adding your first class

The README gives a single pip command. It installs ONNX Runtime as a dependency, which the project says gives 2-4x faster CPU inference out of the box.

pip install adaptive-classifier

For working on the library itself, the README documents an editable install from a clone, with pytest, pytest-cov and pytest-randomly as the optional test dependencies. The pyproject.toml testpaths entry is tests, and markers include integration and slow, so you can deselect the integration suite with -m "not integration".

git clone https://github.com/codelion/adaptive-classifier.git
cd adaptive-classifier
pip install -e .
pip install pytest pytest-cov pytest-randomly

The repository ships three example scripts under examples/: basic_usage.py, advanced_usage.py, and multilabel_usage.py. The README also links Colab notebooks for support ticket classification, query classification, multilingual sentiment as an ensemble, product category classification with batch processing, and multi-label classification. The basic example is described as demonstrating continuous learning with a cat-or-dog task.

A first real use follows the shape the README describes: instantiate AdaptiveClassifier over a HuggingFace model, add labelled examples, then predict. The exact constructor arguments are not reproduced in the README excerpt, so read examples/basic_usage.py before you write your own. What the README does commit to is the workflow: add examples, add classes at runtime, predict. Models move between a training job and a service through the Hub, which the README lists as a HuggingFace integration for pushing and pulling models directly.

Where the design gives way

The benchmark tables in the README deserve a careful reading rather than a summary. The strategic classification numbers come from adversarial examples in the AI-Secure/adv_glue dataset, and they are small: clean accuracy 80.00% for the regular classifier against 82.22% for the strategic one, adversarial accuracy 60.00% against 82.22%. A 2.22 point clean gain on a small evaluation is not a reason to switch modes on its own.

The hallucination detection table is the more sobering one. Evaluated on RAGTruth, the overall precision is 40.89% with recall 80.68% and F1 51.54%. Summarization recall is 96.91% but precision is 22.18%. If you read that as a general-purpose hallucination detector, you will be disappointed; it flags a great deal that is not a hallucination. The data-to-text row, at 65.00% precision and 100.0% recall, is the only configuration in the table that looks usable as a filter.

The LLM router numbers are similarly scoped: 500 queries from arena-hard-auto-v0.1, cost savings moving from 25.60% to 32.40%. That is a routing experiment, not a property of the library.

Two structural limits follow. First, the whole system rests on embedding quality, so a domain where your encoder was never trained will give you a prototype memory that separates nothing, and the pooling fix in 0.2.0 only corrects a mismatch between the library and the model's own configuration, not a mismatch between the model and your text. Second, the README does not document rollback. Zero-downtime updates are advertised, but what happens when a newly added class turns out to be wrong is not described in the repository documentation.

Against a fine-tuned classifier, and against a vector database

The obvious alternative is the ordinary path: take a HuggingFace transformer, fine-tune it with Trainer or your own loop, and serve the result. The difference is where the label set lives. In a fine-tuned classifier, classes are baked into the output layer, so adding one means touching the weights and revalidating the model. In adaptive-classifier, classes live in the prototype memory, so adding one is an index operation. The trade is that you now run two prediction paths and have to reason about how they combine, and you inherit a FAISS index as a piece of state you must persist and version.

The second alternative is to skip the classifier entirely and use a vector database with k-nearest-neighbour search over embeddings. That gets you dynamic classes for free. What it does not get you is the adaptive neural layer or the EWC protection, so you have no mechanism for improving the decision boundary as labelled examples accumulate. adaptive-classifier is essentially that vector-search idea plus a trainable head on top, which is also why its failure modes are the union of both.

Maintenance, licence, and what upgrading from 0.1.x costs

The repository is not archived, and the last push was on 2026-09-17, the same day v0.2.0 was tagged. The prior release, v0.1.2, is dated 2025-10-07, and v0.1.1 is from the same day. That is a long quiet stretch followed by a release that changed the pooling default, so the practical upgrade question is whether your encoder was relying on CLS pooling. If it was, 0.2.0 will change your prototype vectors and therefore your predictions. Nothing in the README describes a migration flag or a way to pin the old behaviour.

The dependency list is not small: torch, transformers, faiss-cpu, scikit-learn, huggingface_hub, and optimum[onnxruntime]. The comments in requirements.txt note that faiss-gpu is the GPU variant, so a GPU setup means substituting that package rather than installing both. Python 3.8 is the floor according to setup.py.

The licence is Apache-2.0, which permits commercial use and modification with the usual notice and patent terms. Nothing here is legal advice; if you ship a model trained with this library, check how the licence of your base encoder interacts with your distribution plan, since that is a separate question from this library's licence.

Editorial conclusion

Adopt adaptive-classifier if your label set changes faster than you can retrain, or if you want a prototype memory you can inspect and extend with add_class. Do not adopt it as a drop-in replacement for a fine-tuned classifier on a fixed label set, and do not expect the published hallucination-detection numbers to hold for your data. Before committing, run the examples/basic_usage.py flow on your own encoder, check what pooling mode the library selects for it, and read the strategic classification section of the docs to see whether the game-theoretic mode applies to your threat model.

Frequently asked questions

Official sources

1. codelion/adaptive-classifier on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases