vllm-project

vllm-project

vllm

★ 92,677

A high-throughput and memory-efficient inference and serving engine for LLMs.

vllm-project

vllm-omni

★ 7,085

A framework for efficient model inference with omni-modality models.

vllm-project

semantic-router

★ 5,980

A programmable Mixture-of-Models router for heterogeneous LLM inference

vllm-project

aibrix

★ 5,113

Cost-efficient and pluggable Infrastructure components for GenAI inference.

vllm-project

llm-compressor

★ 3,827

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

vllm-project

vllm-ascend

★ 2,911

Community maintained hardware plugin for vLLM on Ascend

vllm-project

production-stack

★ 2,631

vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

vllm-project

vllm-metal

★ 1,787

Community maintained hardware plugin for vLLM on Apple Silicon

vllm-project

guidellm

★ 1,635

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

vllm-project

speculators

★ 858

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

vllm-project

vime

★ 479

An LLM post-training framework with vLLM for RL Scaling