LLM2Vec: Turning Decoder-Only LLMs into Text Encoders

The gap LLM2Vec fills between generation and retrieval

Decoder-only LLMs generate text well but do not naturally produce a single fixed vector for a sentence. Their attention is causal, so each token only sees the tokens before it, and the last hidden state is shaped for next-token prediction rather than for similarity comparison. The usual workaround is to run a separate encoder such as a BERT derivative alongside the LLM, which means two models, two tokenizers and two sets of weights in memory. LLM2Vec targets that duplication. The README describes it as a recipe to convert decoder-only LLMs into text encoders in three steps, and the repository ships both the library and the transformed checkpoints. The intended user is someone who already has a Llama-3, Mistral, Llama-2 or Sheared-Llama checkpoint and wants embeddings without adding a second model family to the stack.

Bidirectional attention, MNTP and contrastive learning in sequence

The three steps are ordered and each one changes the model in a specific way. First, bidirectional attention is enabled, which removes the causal mask so every token can attend to every other token. Second, masked next token prediction trains the model to recover masked spans, which the README presents as the adaptation stage that teaches the model to use the newly available context. Third, unsupervised contrastive learning with a SimCSE objective on a wiki corpus pulls semantically related sentences together. A fourth, optional stage uses supervised contrastive learning on public E5 data and is distributed as separate LoRA weights. The library itself is a wrapper over HuggingFace models: the README states that LLM2Vec supports enabling bidirectionality, sequence encoding and pooling operations, and that from_pretrained accepts a base model identifier plus an optional PEFT model identifier. That means the base checkpoint and the trained LoRA adapter are loaded separately, so the same base model can serve generation while the adapter supplies the encoder behaviour.

Installing llm2vec and loading a checkpoint

Installation is two commands, and the second one is the one that tends to fail. The README gives:

pip install llm2vec pip install flash-attn --no-build-isolation

Cloning the repository and running pip install -e . is offered as the alternative for the latest version. Loading a model takes a base path and a PEFT path. The README example uses McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp as the base and McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse as the adapter, with device_map set to cuda when available and torch_dtype set to bfloat16. Swapping peft_model_name_or_path to the -supervised suffix loads the contrastively trained adapter instead. Two keyword arguments matter at load time: pooling_mode, which defaults to mean, and max_length, which defaults to 512. The README does not list the full set of accepted pooling values, so anyone who needs cls or last-token pooling should check the source rather than assume a string.

The encode API and the instruction asymmetry

Encoding accepts either a flat list of strings or a list of [instruction, text] pairs, and the README is explicit that instructions are supplied for both sides in symmetric tasks and only for queries in asymmetric ones. The retrieval example builds queries as [instruction, question] pairs and documents as plain strings, then normalises both with torch.nn.functional.normalize and multiplies them with torch.mm to get a cosine similarity matrix. The README prints a two-by-two matrix for its sample query and document set. Note what this implies: the instruction prefix is part of the input text, so it consumes tokens against the 512 default max_length, and a query with a long instruction has less room for the query itself. The examples directory is where the README points for classification, clustering and sentence similarity, so the retrieval path shown here is one case rather than the whole surface.

Where LLM2Vec is the wrong tool

The obvious cost is size. The published checkpoints are built on 7B and 8B base models, with Sheared-Llama-1.3B as the smallest option in the model table. Serving an 8B encoder means roughly an order of magnitude more memory and latency per embedding than a 110M parameter BERT-style encoder, and the README's own loading example uses bfloat16 on CUDA, which points at a GPU requirement rather than a CPU one. The flash-attn dependency is a second constraint: it is installed with --no-build-isolation, which means the build sees your existing PyTorch and CUDA rather than a clean environment, and that build fails on unsupported GPU architectures. There is also a training cost. The recipe has three to four stages, and the README describes the corpus for the unsupervised stage as wiki and the supervised stage as public E5 data. Reproducing those stages is not a single fine-tuning run. Finally, the library wraps HuggingFace loading, so anything the base model cannot do, such as a non-standard attention implementation, is inherited as a limitation rather than solved.

LLM2Vec against a sentence-transformers encoder

The natural comparison is sentence-transformers, which loads a BERT-style encoder and returns embeddings from a single forward pass with no adapter, no flash-attn build and no bidirectional conversion. The difference in approach is architectural rather than a matter of tuning. A BERT-style encoder is bidirectional from pretraining, so its pooled output was always meant to summarise a sequence; LLM2Vec takes a model trained for left-to-right generation and retrofits bidirectionality through the three stages. That retrofit is what lets one set of base weights serve both chat and retrieval, which sentence-transformers cannot offer because its encoders cannot generate. The trade is that you pay for an 8B encoder to get that reuse, and if you never need the generative side, the retrofit buys you nothing over a purpose-built encoder.

Version history and what upgrading costs

The releases listed are 0.2.0, 0.2.2 and 0.2.3, with 0.2.3 dated 2025-01-24. The README updates mention added support for Gemma and Qwen-2, support for recent transformer versions covering Llama 3.1 and 3.2, and an mteb_eval_custom.py script for evaluating any LLM2Vec model. Those entries are dated 03/10 and 04/07 without a year, so the exact release each landed in cannot be confirmed from the material. The practical upgrade cost sits in the transformer dependency rather than in the llm2vec API: the README ties new model support to newer transformer versions, so a pinned older transformers install will not load Gemma or Qwen-2 checkpoints. On licensing, the repository is MIT, which covers the code. The base checkpoints are separate artefacts under their own terms (Llama and Mistral weights are not MIT), and the LoRA adapters are published on HuggingFace under whatever licence those model cards carry. That distinction matters if you plan to redistribute a fine-tuned encoder, and it is worth reading the model card rather than assuming the repository licence travels with the weights.

Editorial conclusion

Adopt LLM2Vec if you already run a Llama-3 or Mistral checkpoint and want embeddings from the same weights, accepting an 8B parameter encoder and a flash-attn build step. Do not adopt it if you need a small CPU-friendly encoder or a model you can train from scratch on modest hardware; a BERT-style encoder such as sentence-transformers remains the cheaper default there. Before committing, verify that your GPU supports the flash-attn build, check which pooling mode the checkpoint expects, and confirm whether you need the supervised or unsupervised LoRA variant for your task.

Official sources

1. License: MIT
2. McGill-NLP/llm2vec on GitHub
3. Project website
4. README
5. Releases