vllm-project

vllm-project

vllm

★ 91,926

适用于法学硕士的高吞吐量和内存高效的推理和服务引擎。

vllm-project

vllm-omni

★ 6,826

使用全模态模型进行高效模型推理的框架。

vllm-project

semantic-router

★ 5,847

A programmable Mixture-of-Models router for heterogeneous LLM inference

vllm-project

aibrix

★ 5,090

用于 GenAI 推理的经济高效且可插拔的基础设施组件。

vllm-project

llm-compressor

★ 3,789

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

vllm-project

vllm-ascend

★ 2,836

Community maintained hardware plugin for vLLM on Ascend

vllm-project

vllm-metal

★ 1,739

Community maintained hardware plugin for vLLM on Apple Silicon

vllm-project

guidellm

★ 1,609

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

vllm-project

speculators

★ 835

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

vllm-project

vime

★ 467

An LLM post-training framework with vLLM for RL Scaling