Vidur: Simulating LLM Inference Without Booking a GPU

The question Vidur answers: which deployment shape, before you rent the GPUs

Sizing an inference deployment normally means renting the hardware, loading the model, replaying a trace, and measuring. That loop is expensive when the decision space includes tensor parallel size, pipeline stages, replica count, scheduler choice and batch caps. Vidur replaces the measurement loop with a discrete-event simulation. The README states the tool can study system performance under different workloads and configurations, do capacity planning, and test research ideas such as new scheduling algorithms or speculative decoding, and it frames the payoff directly: all of this without access to GPUs except for a quick initial profiling phase. The intended user is an infrastructure or serving engineer who already knows what TTFT and TPOT mean and wants a ranked shortlist of configurations before spending on a cluster. It is not a serving engine. Nothing in the material suggests Vidur executes a model or serves traffic.

Where the numbers come from: profiling, then a random forest predictor

The architecture splits into two halves. The simulator itself is a discrete-event model of a replica: the CLI exposes replica_config_device, replica_config_model_name, replica_config_tensor_parallel_size and replica_config_num_pipeline_stages, plus a cluster_config_num_replicas count. The other half is the execution time predictor, and this is the part that determines whether the output means anything. The README points to docs/profiling.md for instructions on adding a new model to existing or new SKUs, which implies the shipped accuracy rests on measurement data collected per model and per device. The predictor configuration keys are visible in the example command: random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size, prediction_max_batch_size and prediction_max_tokens_per_request. Note the spelling of random_forrest in the config key. It is misspelled in the project itself, so copy it exactly. Those three keys bound the range over which the predictor is valid, which means a simulation run outside those bounds is extrapolation, not interpolation. The README does not describe the predictor's error characteristics, so treat the output as a ranking signal rather than a forecast.

Supported models are a short list, and the gaps are real

The compatibility table covers Meta-Llama-3-8B and 70B, Llama-2-7b-hf and 70b-hf, CodeLlama-34b-Instruct-hf, internlm-20b and Qwen-72B, across four device configurations: A100 80GB DGX, H100 DGX, a 4xA100 80GB pairwise NVLink node and an 8xA40 pairwise NVLink node. The table is not fully populated. Llama-3-8B and Llama-3-70B are marked as supported on A100 DGX and on the 4xA100 pairwise node, but not on H100 DGX, and not on the 8xA40 node. Everything else is marked supported everywhere listed. Context length is a second constraint: all models support 4k except Llama3-8B and Llama3-70B, which reach 16k only if you pass the three predictor bounds above. Parallelism has its own arithmetic. The PP dimension must divide the number of layers, TP1, TP2, TP4 and TP8 are available on DGX nodes with 8 NVLink-connected GPUs, and TP4 on the 4x pairwise node is explicitly noted as less performant than TP4 on DGX because only two pairs are NVLink-connected. If your model is not in that table, you are in the profiling path, not the simulation path.

Getting a run: mamba, venv, or conda, then one long command

Three setup paths are documented. With mamba: mamba env create -p ./env -f ./environment.yml followed by mamba env update -f environment-dev.yml. With venv: install Python 3.10, run python3.10 -m venv .venv, activate it, then python -m pip install -r requirements.txt. Conda works the same way as mamba but the README labels it the least recommended option. Wandb is optional and on by default in spirit: you either run wandb login --host https://<your-org>.wandb.io, or opt out with export WANDB_MODE=disabled, or blank out wandb_project and wandb_group in vidur/config/default.yml and drop the corresponding CLI flags. The bare entry point is python -m vidur.main, and python -m vidur.main -h lists every parameter. The documented full example sets the device to a100, the model to meta-llama/Meta-Llama-3-8B, one replica, TP1 and one pipeline stage, a synthetic request generator with 512 requests, a trace length generator reading ./data/processed_traces/splitwise_conv.csv with max_tokens 16384, a Poisson interval generator at 6.45 QPS, and the sarathi scheduler with batch_size_cap and chunk_size both at 512. Note that the length generator is named trace while the interval generator is poisson: request sizes come from a file, arrival times from a distribution.

Outputs: wandb, a timestamped directory, and a Chrome trace

Metrics are logged to wandb and copied to simulator_output/<TIMESTAMP>. The README points to docs/metrics.md for a description of every logged metric, which is the file to read before interpreting anything. Vidur also exports Chrome traces of each simulation into the simulator_output directory, loadable at chrome://tracing/ or edge://tracing/. The Chrome trace is the more useful artifact for debugging an unexpected result, because it shows where simulated time went rather than only the aggregate. Two operational notes follow from this layout. First, runs are not self-contained unless you disable wandb or accept the dependency. Second, the timestamped directory means repeated runs accumulate; there is no documented cleanup or comparison command, so you are diffing directories yourself. The README does not state how the trace is named or how large it gets for a long simulation.

The canary branch is where prefix caching and routing live

The README is unusually direct about an unfinished area. It says work is underway on prefix caching, different routing policies and reducing the simulator's memory requirements, and that there are sharp edges still being resolved. Those features are on the canary branch, not main. This matters for evaluation: if prefix caching is central to your workload, the main branch will not model it, and the canary branch is described by its own maintainers as having unresolved sharp edges. There is a second maintenance signal worth naming. The repository has no retrieved releases, so there is no versioned artifact to pin against; you are tracking a branch. The README also notes that most contributions require a CLA. That is a contribution-process detail, not a licensing one, but it shapes who can send patches upstream.

When simulation is the wrong instrument

The failure mode is straightforward. Vidur's accuracy is bounded by the profiling data behind the random forest predictor and by the three max_* config keys that define its valid range. If your target model is absent from the supported table, or you need a context length beyond what the shipped predictor was trained on, the simulation is extrapolating and you should not present its numbers as a capacity plan. The same caution applies to the workload model: the example combines a trace-derived length distribution with a Poisson arrival process. Real serving traffic is frequently burstier than Poisson, and the README does not document an alternative interval generator, so a bursty production trace may be represented poorly. As an alternative approach, consider replaying against a real serving stack on a single GPU with a subset of traffic. That gives you measured latency including kernel-level effects, memory fragmentation and preemption behaviour that a discrete-event model approximates. The trade-off is the opposite of Vidur's: you get ground truth for one configuration, and you cannot cheaply sweep a dozen tensor-parallel and pipeline-parallel layouts. Vidur wins on breadth of configurations per unit of hardware; direct measurement wins on fidelity for the single configuration you actually deploy.

Editorial conclusion

Adopt Vidur if you are choosing between tensor-parallel and pipeline-parallel layouts, or between scheduler variants, and you can afford one profiling pass on the real hardware. Skip it if you need absolute latency numbers for a model or device combination that is not in the supported table, or if your workload's arrival process cannot be approximated by the built-in generators. Before trusting any output, run the supported-model check against your exact checkpoint and context length, then confirm that the random forest predictor was trained on the same SKU you intend to deploy.

Official sources

1. Issues
2. License: MIT
3. microsoft/vidur on GitHub
4. README