vLLM Jev turns Jev decision checkpoints into an HTTP service on vLLM

A server for question, candidate answers, label and probability

vLLM Jev serves compatible Jev-style checkpoints through vLLM. The contract is narrow and worth stating precisely: you give it a question and candidate answers, and it returns a label and a probability for each candidate. That is a different shape of request from the chat completions most inference servers expose, and it is the reason this package exists as a vLLM plugin rather than as a small wrapper around an HTTP endpoint.

Three decision types are supported over HTTP: Choice, Noul for yes or no questions, and Score for ordered levels. The multimodal side covers Valen, jev, RSI-Jev v4.0-VL, JevAny Pointer and Clef, which accept text and images, with Valen and Clef also accepting short videos. Platform availability and request formats are model-specific, which is the first thing to check before planning a deployment.

On Linux with an NVIDIA GPU the runtime is native vLLM, so scheduling, batching, compilation, KV cache and metrics come from that stack rather than being reimplemented. Apple Silicon is described as a preview: supported text models run through MLX or PyTorch MPS, and Valen multimodal decisions run through MLX.

Automatic setup is a stated goal: you start a supported Hugging Face model ID with one command and the tool downloads and prepares the model.

Registered as two vLLM plugins, not a wrapper around one

The packaging tells you how the integration actually works. The project metadata registers two entry points. The first is a general plugin, vllm_jev_model pointing at vllm_jev:register, which is how the Jev model family becomes available inside vLLM. The second is an endpoint plugin, vllm_jev_endpoint pointing at vllm_jev.endpoint:JevEndpointPlugin, which is how the HTTP surface is added.

That design has a consequence for operators. The decision endpoint is not a separate process talking to vLLM over a socket; it is part of the vLLM server process. Anything that reconfigures a vLLM deployment, such as changing the port or the GPU memory utilisation, changes where the decision API lives as well, and you have to keep the two in sync.

The dependency pins are strict. vllm is pinned to exactly 0.29.0, and transformers is bounded differently per platform: greater than or equal to 5.10 and below 5.11 away from darwin, and greater than or equal to 5.14 and below 6 on darwin. mlx-vlm is pinned to 0.6.17 and only installed on darwin, and laya is pinned to 0.3.20. Everything else is looser: peft, safetensors, filelock and huggingface_hub. Python 3.12 or newer is required.

The project version is 0.3.7, the build backend is setuptools with a requirement of setuptools 77 or newer, and the package data is restricted to vllm_jev. Linting is configured with ruff at a line length of 88.

Installing on Linux and starting the first server

On Linux with an NVIDIA GPU, installation from the repository directory is a single uv command:

uv pip install .

For development work, the project documentation points to an editable installation instead. Once installed, you start a server by handing it a supported Hugging Face model ID:

vllm-jev serve ZefanCai/Open-Jev-2B

The command downloads and prepares the model, then starts native vLLM at http://127.0.0.1:8795. Later runs reuse the checkpoint that was already prepared, so the first invocation is the slow one.

GPU, port and serving options are exposed through standard vllm serve flags. The documented example pins a device and overrides the memory fraction and port:

CUDA_VISIBLE_DEVICES=0 vllm-jev serve ZefanCai/Open-Jev-2B \
  --gpu-memory-utilization 0.90 --port 8795

Requests then go to /v1/systemone for Choice, Noul and Score decisions. The endpoint name is unusual enough to be worth noting, since it does not follow the usual chat or completions convention.

The console scripts are two, not one. vllm-jev maps to the main CLI entry point, and vllm-jev-radixflow maps to a separate launcher for the experimental scheduler, so the feature can be started on its own.

The Apple Silicon path is a separate setup script and a separate runtime

On an Apple Silicon Mac running macOS 15 or newer, setup is done once from the repository directory and then the server is started:

source scripts/install_mac.sh
vllm-jev serve Valen-Team/Valen-Preview-0923

The command downloads and prepares the model on first use. To serve a different model, replace the model ID with another supported one from the table.

The split at the dependency level explains why the two platforms behave differently. On darwin the package installs mlx-vlm at a fixed version and a different transformers range, and the README states that Apple Silicon uses MLX, or PyTorch MPS for Laya. On Linux the same checkpoints run through native vLLM.

Model support is not uniform across platforms. Open-Jev-2B is listed for Linux and Mac, Open-Jev-9B for Linux only, and both laya and laya-multilingual for Linux and Mac. The remaining entries in the supported model table were truncated in the source available here, so the table itself is the authority on which checkpoint runs where.

One adapter breaks the pattern. Drex v2.0 has experimental Linux support for text and JSON Choice, Noul and Score, with official BF16 weights that have passed native dual-A800 functional stress tests, and prefix caching and RadixFlow are disabled for it.

RadixFlow defers a pending branch instead of stopping the scan

RadixFlow is the scheduler change the project is currently pushing, and it is experimental and disabled by default. It is enabled with --radixflow true on supported configurations.

The reasoning starts from prefix reuse. Jev scores candidate branches through prefill, so a common prefix across candidates can be reused rather than recomputed. An earlier pull request in the project explored shared-prefix reuse and cache on or off behaviour, and the observation that followed was about what happens after a cache hit.

The scheduling gap is described precisely: a matching prefix can still have hybrid state being produced in the current round, and stopping the scan at that pending branch leaves capacity that independent ready branches could have used. RadixFlow therefore defers the pending branch, continues selecting ready work, and rechecks the deferred branch after the current work completes.

The documentation says the engineering integration recovered executable capacity in the tested hybrid models, and that separate documents cover the configuration and report worker rounds, speed and token reuse. That framing is careful, and worth preserving: the claim is about recovered capacity in hybrid models, not a blanket speedup, and the feature is off unless you opt in.

The reported numbers come from the project, and their scope is narrow

The demo section of the project documentation contains the only quantitative comparisons available, and they are stated with their conditions attached.

For video understanding, the same Valen checkpoint answers 24 structured questions about one video using eight sampled frames and one A800 per runtime. In that live run the PyTorch reference path took 5.31 s while vLLM Jev took 1.22 s, described as 4.3 times faster, with all 24 selected answers agreeing and model loading excluded. For text, the Open-Jev-2B author HTTP server and vLLM Jev received the same 40 text-only Choice requests at once on one A800 each. On Apple Silicon, Valen 2B answers six GameQA Maze image questions locally on an Apple M5 using MLX, with the display following live HTTP responses.

Read those conditions carefully. They describe specific checkpoints, one GPU, one workload shape, and model loading excluded from the timing. The video comparison excludes load time and uses a single A800 per runtime, so it says nothing about throughput on smaller hardware or about cost per request at scale. The Apple Silicon demo is explicitly a preview.

The honest limitation here is that the comparison set is the PyTorch reference path, not another serving stack. A team already running a well tuned PyTorch pipeline may see less difference than the 4.3 times figure suggests, and anyone on a machine without an A800 is outside the conditions of the measurement entirely.

No tagged releases, Apache-2.0, and a hard Python floor

The repository publishes no GitHub releases, so there is no tag to pin and no changelog to read before upgrading. The last push was on 2026-10-19, which is recent, but the version in the packaging metadata is 0.3.7 and the upgrade path is to reinstall from the repository. The repository tree includes a rust/ directory alongside the Python package, plus scripts/, tests/, docs/ and a licenses/ directory, with THIRD_PARTY_NOTICES.md and license-files pointing at LICENSE, that notices file and the licence texts under licenses/.

The licensing setup is unusually explicit for a package of this size. The project licence is Apache-2.0, and the licence files are declared in the package metadata so that a built wheel carries them. Third party notices are tracked in their own file with per-dependency texts in licenses/. That is what you want to see when a package bundles components with different licences, and it means you can check attribution obligations from the installed artifact rather than from a source checkout.

The upgrade cost, then, is dependency risk rather than migration work. vllm is pinned to a single version, transformers is pinned to a narrow range per platform, and mlx-vlm and laya are pinned exactly, so a fresh install on a different platform resolves to a different transformers range than the one a Linux install uses.

Editorial conclusion

vLLM Jev is the right choice for teams already running Jev decision checkpoints who need one HTTP endpoint for Choice, Noul and Score requests with real batching and metrics, and it is the wrong choice for a single model on a laptop without an NVIDIA GPU, since the Apple Silicon path is a preview and the package requires Python 3.12 or newer with vllm pinned to 0.29.0. Before adopting it, verify your checkpoint appears in the supported model table and leave --radixflow false until you have read docs/radixflow.md, since the prefix reuse scheduler is experimental and prefix caching is disabled outright for the Drex v2.0 adapter.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. mode-io/vllm-jev on GitHub
4. README