# OpenVINO GenAI: generative model pipelines on the OpenVINO Runtime

> OpenVINO GenAI is a library of generative AI pipelines, optimized execution methods and samples that sit on top of the OpenVINO Runtime, with light C++, Python and Node.js APIs for text, image, video, speech and retrieval work. Its selling point is a self-contained stack that runs on CPU, GPU and NPU without pulling in a separate inference framework.

**openvinotoolkit/openvino.genai** — Run Generative AI models with simple C++/Python API and using OpenVINO Runtime

- Repository: https://github.com/openvinotoolkit/openvino.genai
- Website: https://openvinotoolkit.github.io/openvino.genai/
- Stars: 603 · Forks: 486
- Language: C++
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/openvinotoolkit-openvino-genai

## Pipelines, not a framework

OpenVINO GenAI is a library of the most popular generative AI model pipelines, optimized execution methods and samples that run on top of the OpenVINO Runtime. The framing is deliberate: this is not a competing inference stack, it is a layer of ready made pipelines that sit on the runtime OpenVINO already provides.

Two properties follow from that. The first is that it is friendly to PC and laptop execution and optimized for resource consumption, with the readme showing text generation with a LLaMa 3.2 model running on an Intel ARC770 dGPU. The second is that it requires no external dependencies to run generative models, because it already includes the core functionality, with tokenization handled by the openvino-tokenizers library rather than by a separate package you have to find and configure.

The APIs are described as very lightweight, and there are three of them: C++, Python and Node.js, the last with a published npm package. The project ships a documentation site at openvinotoolkit.github.io/openvino.genai, and the licence is Apache 2.0 with the licence file at the root of the repository.

The repository structure is that of a serious C++ project: a CMakeLists.txt at the root, `cmake`, `src`, `tests`, `tools`, `thirdparty`, `samples` and a `site` directory holding the documentation, plus a Jenkinsfile for CI, a `bandit.yml` for the Python security linter, a `.clang-format` and a pre-commit configuration.

## The quick start is install, export, run

The quick start in the readme has three steps, and they are worth reading closely because the middle one is where the real work is.

First, install from PyPI:

```sh
    pip install openvino-genai
```

Second, obtain the model, for example by exporting it to the OpenVINO IR format from Hugging Face:

```sh
    optimum-cli export openvino --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --trust-remote-code TinyLlama_1_1b_v1_ov
```

That command uses the `optimum-cli` export path, requests int4 weights, and passes `--trust-remote-code`, which is what allows the export to run the model's own remote code rather than only the standard configuration. The model preparation guide in the documentation covers this in more detail, and the trailing `TinyLlama_1_1b_v1_ov` is the output directory.

Third, run inference:

```python
    import openvino_genai as ov_genai

    pipe = ov_genai.LLMPipeline("TinyLlama_1_1b_v1_ov", "CPU")  # Use CPU or GPU as devices without any other code change
    print(pipe.generate("What is OpenVINO?", max_new_tokens=100))
```

The pipeline constructor takes the model directory and a device name, and the comment in the sample makes the point that switching between CPU and GPU needs no other code change. Generation is a single call with a prompt and a token limit.

The Python package metadata shows what this costs you in dependencies: a single runtime requirement, `openvino_tokenizers`, and Python 3.11 or newer. The classifiers mark the project as Production/Stable, and it is supported on Unix, Linux, Windows and macOS.

## Eight scenarios behind the same pipeline style

The scenario list is broader than the name suggests, because the library covers the whole generation stack rather than just chat.

Text generation with large language models is the headline case, with local Llama, Phi, Qwen and other models. Visual processing with vision language models handles images and video with LLaVa, MiniCPM-V and others. Image generation and video generation both go through Diffusers, with Stable Diffusion and Flux for images and LTX for video. Speech recognition uses Whisper to turn speech into text, and speech generation uses SpeechT5 to turn text into speech.

The last two are the retrieval side. Semantic search using text embedding computes embeddings for documents and queries to enable efficient retrieval in RAG workflows, and text rerank for retrieval augmented generation analyses the relevance and accuracy of documents and queries for the same workflows. That pairing is the one that matters for anyone building a RAG system, because embedding and reranking are separate stages with separate models, and having them behind the same API removes a second runtime to manage.

LoRA adapters are supported for the text, image and video scenarios. The library can load multiple adapters per model, select the active adapters for every generation, and mix multiple adapters with coefficients through alpha blending. Adapter swapping per request is what makes it practical to serve several fine-tunes of one base model from a single loaded set of weights.

## Speculative decoding, sparse attention and continuous batching

The optimization methods are the part that separates this from a thin wrapper, and the readme describes them without marketing language.

Speculative decoding employs two models of different sizes and uses the large model to periodically correct the results of the small model. The intuition is that the small model produces candidate tokens quickly and the large model verifies batches of them, which is cheaper than generating every token with the large model. The catch is that the correction step is what makes it correct, and the small model has to be right often enough for the savings to survive.

A KVCache token eviction algorithm reduces the size of the KVCache by pruning the tokens that have the least impact. That matters for long contexts, where the cache is the memory bottleneck rather than the weights.

Sparse attention accelerates prefill by attending only to the most important regions of the attention matrix, and the library currently supports two modes, Tri-shape and XAttention. Prefill is the phase where a prompt is processed, and it is the part that scales badly with context length, so attention sparsity there is where the compute is saved.

On top of that, the library implements a continuous batching approach for using OpenVINO within LLM serving, with prefix caching that caches fragments of previous generation requests and the corresponding KVCache entries internally and reuses them on a repeated query. The readme notes that the continuous batching functionality is used within OpenVINO Model Server to serve LLMs, and that the library could be used in other LLM serving frameworks.

## The device story is CPU, GPU and NPU

All the scenarios run on top of the OpenVINO Runtime, which supports inference on CPU, GPU and NPU. The readme points at the OpenVINO system requirements page for the platform support matrix, which is where the real constraint on deployment lives.

That distinction matters. OpenVINO GenAI is not a runtime in the way the name of the underlying project might suggest; it inherits the hardware coverage of the runtime and adds the pipelines on top. On an Intel CPU or an integrated GPU, that means the path of least resistance. On an NPU, it depends entirely on whether the OpenVINO release you pair with this library supports the part, and the readme does not promise otherwise.

The packaging reflects the same split. The Python package requires Python 3.11 or newer and the classifiers list 3.11, 3.12 and 3.13, with C++ and C also listed as the languages of the project. There is an npm package for the Node.js bindings, and the documentation has a page for it. The build is a CMake build, and the Python wheel is produced through py-build-cmake, with a minimum CMake version of 3.26 and a dedicated stub target and wheel component in the configuration.

So the deployment question is not which model you want but which devices the runtime supports, and the answer to that lives in the OpenVINO release notes rather than in this repository.

## Versioning, licences and the samples tree

The releases follow the OpenVINO calendar style: 2026.3.0.0 on 2026-08-05, 2026.4.0.0 on 2026-09-17 and 2026.4.1.0 on 2026-10-01, and the last push was on 2026-10-20. The version in the Python package metadata is 2026.5.0.0, so the packaging is already ahead of the newest tagged release, which is normal for a repository that builds a release from the branch rather than from a tag.

The licence is Apache 2.0, declared in the package metadata with the `LICENSE` file, and the repository also carries a `SECURITY.md` for disclosure. A `third-party-programs.txt` file records the third party components, and the `thirdparty` directory holds the vendored sources, which is what a project that wants to build without fetching the world looks like.

The samples directory is organised by language, with `c`, `cpp`, `js` and `python` subdirectories, a `CMakeLists.txt` and separate requirements files for export and deployment. The presence of an export requirements file next to a deployment one is a small but useful hint: the sample workflow is a two stage thing, where the export tooling runs once to produce the IR and the deployment side runs afterwards.

For learning the library, the readme also points at two blog posts, one on building an OpenVINO GenAI app in C++ and one on running Llama 3.2 locally with OpenVINO, plus the supported models page and the model preparation guide.

## Conclusion

OpenVINO GenAI suits a team already committed to OpenVINO that wants one runtime for both classic inference and generative workloads, and that values a small dependency footprint over the flexibility of a heavier framework. It is a poor fit if you are on a hardware stack OpenVINO does not cover, since the platform support matrix is the limit on where this can run, and a poor fit if you need a model that is not in the supported list. Before you adopt it, read the supported models page for your architecture, check the system requirements for your CPU, GPU or NPU, and read the model preparation guide, because the export step in the quick start is where most of the work in a first run happens.

## FAQ

### How do I install and run a model with OpenVINO GenAI?

Install the package with `pip install openvino-genai`, export a model to the OpenVINO IR format, for example with the optimum-cli export path, then build a pipeline with `ov_genai.LLMPipeline` pointing at the exported directory and a device such as CPU or GPU, and call `generate` with a prompt.

### Does OpenVINO GenAI need separate packages for tokenization?

No. The library requires no external dependencies to run generative models because it already includes the core functionality, including tokenization, which is handled through the openvino-tokenizers library. The Python package lists that library as its only runtime requirement.

### Which hardware can OpenVINO GenAI run on?

All the scenarios run on top of the OpenVINO Runtime, which supports inference on CPU, GPU and NPU. The readme points to the OpenVINO system requirements page for the platform support matrix, so the device coverage comes from the runtime release you pair with the library.

### What optimization techniques does OpenVINO GenAI provide?

Speculative decoding with two models of different sizes where the large model corrects the small one, a KVCache token eviction algorithm that prunes low impact tokens, sparse attention for prefill in two modes called Tri-shape and XAttention, and a continuous batching approach with prefix caching for LLM serving.

### Can I use LoRA adapters with OpenVINO GenAI?

Yes. The library supports LoRA adapters for the text, image and video scenarios, with the ability to load multiple adapters per model, select the active adapters for every generation, and mix several adapters with coefficients through alpha blending.

## Sources

- [License: Apache-2.0](https://github.com/openvinotoolkit/openvino.genai/blob/master/LICENSE)
- [openvinotoolkit/openvino.genai on GitHub](https://github.com/openvinotoolkit/openvino.genai)
- [Project website](https://openvinotoolkit.github.io/openvino.genai/)
- [README](https://github.com/openvinotoolkit/openvino.genai/blob/master/README.md)
- [Releases](https://github.com/openvinotoolkit/openvino.genai/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openvinotoolkit-openvino-genai