vllm-project

vllm-project

vllm

★ 91,844

適用於法學碩士的高吞吐量和記憶體高效的推理和服務引擎。

vllm-project

vllm-omni

★ 6,792

使用全模態模型進行高效模型推理的架構。

vllm-project

semantic-router

★ 5,826

A programmable Mixture-of-Models router for heterogeneous LLM inference

vllm-project

aibrix

★ 5,090

用於 GenAI 推理的經濟高效且可插拔的基礎設施組件。

vllm-project

llm-compressor

★ 3,785

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

vllm-project

vllm-ascend

★ 2,829

Community maintained hardware plugin for vLLM on Ascend

vllm-project

vllm-metal

★ 1,739

Community maintained hardware plugin for vLLM on Apple Silicon

vllm-project

guidellm

★ 1,609

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

vllm-project

speculators

★ 833

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

vllm-project

vime

★ 467

An LLM post-training framework with vLLM for RL Scaling