LLMLingua: Prompt Compression Before the Model Sees the Tokens
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
At a glance
- What is it?
- Microsoft's LLMLingua family removes non-essential tokens from prompts using a small language model as the compressor. It targets token limits and prompt-based pricing, and the repository documents three distinct compression methods with different speed and domain trade-offs.
- Who is it for?
- Adopt LLMLingua when your prompts are long, your billing is per token, and you can tolerate a small compressor model in the request path. Do not adopt it when the prompt is short, when every token is legally or operationally load-bearing, or when you cannot re-evaluate task accuracy after compression.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The token-limit and per-token billing problem LLMLingua addresses
The README opens with three concrete complaints: hitting the token limit when asking ChatGPT to summarize long texts, the model forgetting earlier instructions after long context, and the cost of running GPT-3.5 or GPT-4 experiments at scale. These are the problems the project claims to solve. The stated mechanism is a compact, well-trained language model (the README names GPT2-small and LLaMA-7B as examples) that identifies and removes non-essential tokens in prompts. The headline figure is up to 20x compression with minimal performance loss. The audience is anyone paying per prompt token or fighting a context window: RAG pipelines, long-document summarization, chain-of-thought prompting, meeting transcripts and code contexts, all of which appear in the examples directory. The repository is not a hosted service. It is a Python library plus a set of notebooks, and it sits between your prompt construction code and whatever LLM you call.
Three compressors, three different mechanisms
The README describes three methods rather than one. LLMLingua, from the EMNLP 2023 paper, uses a small language model to score and drop non-essential tokens; it is the original approach and the slowest of the three. LongLLMLingua, from ACL 2024, targets the lost-in-the-middle behaviour of long-context models and the README states it improves RAG performance by up to 21.4 percent while using one quarter of the tokens. LLMLingua-2 is the odd one out architecturally: it is trained by data distillation from GPT-4 for token classification using a BERT-level encoder, is described as task-agnostic, and the README claims it surpasses LLMLingua on out-of-domain data with 3x to 6x faster performance. That speed difference matters because compression itself costs latency. A GPT2-small scoring pass over a long prompt is not free, and LLMLingua-2's encoder-based classifier is a different cost profile. The repository also lists SecurityLingua, a safety guardrail model that uses security-aware prompt compression to surface malicious intent behind jailbreak attempts, with the README claiming 100x lower token cost than state-of-the-art LLM guardrail approaches. SecurityLingua is a separate model and a separate paper (CoLM 2025), not a drop-in mode of the compression API.
Getting it running: install, compress, call your model
The README's install path is the standard Python one: pip install llmlingua. The documented usage pattern is to construct a PromptCompressor, pass it a prompt string, and receive a compressed prompt that you then send to your target LLM. The release notes show the API surface growing across three versions: v0.2.0 added support for a customized compression spec, v0.2.1 added LLMLingua-2, and v0.2.2 added compress_json. That last one matters for structured prompts, because compressing JSON by treating it as plain text risks breaking keys and quotes. The examples directory contains notebooks for LLMLingua-2, RAG, online meetings, chain-of-thought, code, and a LlamaIndex RAG example, plus a Retrieval.ipynb script tied to the project's blog post on RAG cost savings. The project has also been integrated into Prompt flow, LangChain and LlamaIndex, so if you already build on those frameworks you may not need to call the compressor directly. What the README does not show in the supplied material is a full configuration reference for the compression spec keys; you will need to read the source or the notebooks to find the exact parameter names and defaults.
Where compression is the wrong tool
Token removal is irreversible from the downstream model's point of view. If your prompt contains an account number, a legal clause, a function signature or a dosage, a compressor that scores tokens by learned importance has no way to know that this particular token is load-bearing. The README's own framing is about non-essential tokens, which assumes a distinction your data may not support. The second limitation is evaluation cost. The 20x figure and the 21.4 percent RAG improvement are paper results on the datasets those papers used; applying the library to your own prompts means re-measuring task accuracy at your chosen compression ratio, and the repository does not ship an evaluation harness for arbitrary tasks. Third, the compressor is itself a model. LLMLingua's GPT2-small or LLaMA-7B scorer has to load into memory and run, which is a poor trade for short prompts where the overhead exceeds the tokens saved. Fourth, latency: the README's 3x to 6x speed claim for LLMLingua-2 is relative to LLMLingua, not relative to not compressing at all. If your prompts are already near the model's sweet spot, compression adds a hop without removing a bottleneck.
Alternatives and the actual difference in approach
The nearest alternative is retrieval-side reduction: instead of compressing the prompt after you build it, retrieve fewer or shorter passages so the prompt never grows. LLMLingua's own RAG example and its LlamaIndex integration sit in this space, and the difference is where the decision is made. Retrieval filtering drops whole documents using relevance scores; LLMLingua drops tokens inside the documents you already retrieved, which preserves fragments of many passages rather than all of a few. That is a real trade-off: token-level compression can keep a sentence that matters from a document that scored poorly, but it can also mangle a passage that scored well. A second alternative is summarization by the target LLM itself, which costs a generation call and can hallucinate; LLMLingua's compressor is a separate small model, so it cannot invent content, only delete it. A third is simply choosing a model with a larger context window. That solves the limit but not the per-token bill, and the README's motivation explicitly includes cost, so the two approaches address different halves of the problem.
Maintenance, releases and licence
The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are included. That is a permissive licence and does not impose copyleft obligations on your application. It is not legal advice; read the LICENSE file in the repository before relying on it. On maintenance, the last push date in the supplied metadata is 2026-04-08, but the most recent release listed is v0.2.2 from 2024-04-09. That gap is worth noting: the release cadence visible in the material slowed after the LLMLingua-2 work, while the README's news section continues to point at related Microsoft research (SCBench, RetrievalAttention, MInference). Those are separate projects, not updates to this library. The practical upgrade cost is low if you pin a version, because the API is small: a compressor object, a compress call, and a compression spec. The risk is that the compression spec keys added in v0.2.0 are the part most likely to change, so pinning llmlingua in your requirements file and reading the changelog before bumping is the cheap precaution. The dependency weight is the other cost: the compressor pulls in a transformer model, so your install and cold-start profile will look more like a model-serving service than a pure text utility.
Who should adopt it and what to verify first
The strongest fit is a team with long, repetitive prompts and a per-token bill: RAG over many documents, meeting transcripts, or long chain-of-thought traces where much of the text is scaffolding rather than content. The weakest fit is a team with short, dense prompts where every token was placed deliberately. Between those, the decision hinges on measurement. The README gives paper-level numbers (up to 20x compression, 21.4 percent RAG improvement, 3x to 6x faster for LLMLingua-2) and those are the right starting hypotheses, not your results. The concrete first step is to take one representative prompt from your workload, run it through the compressor at a modest ratio, and diff the compressed text against the original by hand before you trust any automated accuracy metric. If the diff removes something you would have kept, the compression spec needs tuning, and v0.2.0 exists precisely because that spec is configurable. If the diff looks clean, then measure end-to-end latency with the compressor in the path, because LLMLingua-2's speed advantage is only meaningful if the compressor is not the new bottleneck.
Editorial conclusion
Adopt LLMLingua when your prompts are long, your billing is per token, and you can tolerate a small compressor model in the request path. Do not adopt it when the prompt is short, when every token is legally or operationally load-bearing, or when you cannot re-evaluate task accuracy after compression. Before committing, verify the compression ratio your own data tolerates, check whether LLMLingua-2's BERT-level encoder fits your latency budget better than the GPT2-small approach, and confirm the MIT licence text in the repository matches your redistribution plans.
Community notes