vs-mlrt: choosing an ML runtime for VapourSynth filters

The gap vs-mlrt fills between models and VapourSynth

VapourSynth is a frame server: filters are plugins that take a clip and return a clip. Neural network filters do not fit that shape on their own, because a model file has to be loaded by an inference runtime, and each runtime speaks a different API. vs-mlrt is the adapter layer. It ships VapourSynth plugins that wrap inference runtimes, so a script can call a filter and get frames back without the script author touching ONNX Runtime, TensorRT or ncnn directly.

The target user is someone building a video processing script who wants a specific model applied to a clip. The README names the models the project bundles support for: waifu2x, DPIR, RealESRGANv2 and v3, Real-CUGAN, RIFE, SCUNet and ArtCNN. That list matters more than the backend list for most people, because the model is what changes the picture and the backend is what changes the speed.

The project also ships scripts/vsmlrt.py, described as a Python wrapper for all bundled models with a unified interface for selecting backends. That wrapper is the reason a script can be written once and pointed at a different runtime later, which is the practical benefit of the whole arrangement.

Six plugins, one interface, and why the backend choice is the real decision

The repository is organised as one directory per runtime: vsov, vsort, vstrt, vsmigx and vsncnn, alongside common/ and scripts/. Each directory is a separate plugin with its own dependencies, and the README maps them to hardware.

vsov is the OpenVINO-based runtime, aimed at x86 CPUs and Intel GPUs. The README states Intel GPU support covers Gen 8 and later on Broadwell and newer, plus the Arc series. vsort is the ONNX Runtime plugin, providing CPU and CUDA GPU paths. vsort also covers Apple SoC through a CoreML backend, which is the only route in this project to Apple hardware. vstrt is the TensorRT plugin for NVIDIA GPUs, with vstrt_rtx as a TensorRT-RTX variant. vsmigx is the MIGraphX plugin for AMD GPUs. vsncnn is the Vulkan plugin, and it is the only one that spans vendors.

The README is direct about the trade-off in vstrt: TensorRT benchmarks to find the optimal kernel for your specific GPU, so an engine must be built from the ONNX network on the machine that will run the filter. The README calls this extra step harder than the other runtimes, and also says the resulting performance is typically much better than the CUDA backend of vsort. That is a real deployment cost, not a footnote: a prebuilt engine is tied to the machine it was built on.

vsncnn inverts the trade-off. Because it targets Vulkan, the README says it works on any GPU exposing a Vulkan interface, listing NVIDIA, AMD, and Intel integrated and discrete parts. It also notes a significantly smaller footprint than the other GPU runtimes, contrasting with the CUDA backends of vsort and vstrt, which the README says require more than 1GB of CUDA libraries. The README then states the main drawback plainly: it is slower.

Installing a vs-mlrt plugin and running a first filter

The installation instruction is the same for every runtime in the README: download the latest release and extract it into your VapourSynth plugins directory. There is no package manager step documented, so the release archive is the distribution channel. The README does not state the exact subdirectory name or whether the archive unpacks into a versioned folder, so check the extracted tree before assuming VapourSynth will find the plugin.

Once the plugin is in place, the README does not give a command to confirm it loaded, so that check has to come from your own VapourSynth setup. For actual filter calls, the README points to the wiki for supported models and usage information, and to scripts/vsmlrt.py for the unified interface. The wrapper is the piece that makes the backend selectable, so the practical first step is to read that script and the wiki page for the model you want rather than guessing at filter arguments. The README does not reproduce the filter signatures, so any argument list has to come from the wiki.

The one file the README does point at directly is the wrapper, and it is worth opening before anything else:

scripts/vsmlrt.py

Where vs-mlrt is the wrong tool

The clearest limitation is that this is a VapourSynth plugin collection, not a standalone application. If you do not already have a VapourSynth script pipeline, installing vs-mlrt gives you nothing to run. There is no command line entry point documented in the README and no GUI. The homepage field is empty, so there is no separate product site to fall back on.

The second limitation is vendor lock-in at the plugin level. The README maps runtimes to hardware, and the mapping is not interchangeable: vstrt and vstrt_rtx are NVIDIA only, vsmigx is AMD only, vsov is x86 CPU and Intel GPU. A build of one plugin will not help on another vendor's card. Only vsncnn crosses vendors, and the README concedes it is the slow option. If your priority is maximum throughput on an NVIDIA card, you accept the engine-building step; if you want a single binary that works everywhere, you accept Vulkan performance.

Third, vstrt's engine build is a deployment problem, not just a setup annoyance. The README states the engine is built on the machine you are going to use the vstrt filter on. That means a model prepared on one workstation does not transfer as a ready artifact to another without repeating the step.

Finally, the project does not document rollback. The README says to download the latest release and extract it; it does not describe how to revert to an earlier version, whether plugins are versioned side by side, or what happens when a new release changes a filter signature. Recent releases are labelled test builds, so anyone tracking the newest archive is tracking something the project itself labels as a test.

How vs-mlrt compares with running the model outside VapourSynth

The obvious alternative is to run the model in its own framework and feed it frames as image files, rather than inside the frame server. A PyTorch or ONNX Runtime script that reads PNGs, runs the network and writes PNGs back is the pattern most people start with. The difference in approach is where the frames live. In that pattern every frame round-trips through the filesystem and the model runs as a separate process; in vs-mlrt the inference runs inside the filter graph, so frames pass between filters in memory and the model is one node in a chain.

That difference decides which tool fits. If you are processing a handful of stills, the external script is simpler: no VapourSynth install, no plugin directory, no backend selection, and you can use whatever framework version you like. If you are processing video, the external approach means writing the frame extraction and reassembly yourself, and you lose the ability to interleave the model with other VapourSynth filters such as source decoders or resizers. vs-mlrt exists precisely so that the model sits between those filters.

Within the project the same logic repeats at a smaller scale. vsncnn against vsort-cuda is the choice between one plugin that runs on any Vulkan GPU with a small footprint and one that runs only on NVIDIA hardware with a large CUDA dependency but better speed. The README states the trade-off in both directions, which is unusual and useful.

Maintenance, licensing and what a version bump costs

The repository is not archived, and the last push was on 2026-08-06. The most recent release in the list is v16.2.test1, dated 2026-08-05, and the two before it, v16.1.test1 and v16.test1, are also labelled test. That naming is the maintenance signal worth reading: the project is publishing test builds rather than stable ones, so anyone pinning to a release is pinning to something the project has not labelled as final.

The licence is GPL-3.0. For a VapourSynth plugin that a user installs and runs locally, the practical effect is that the plugin is free software and the source is available. The licence matters more if you intend to redistribute the binaries as part of a larger product, because GPL-3.0 carries obligations that a permissive licence does not. This is not legal advice; if redistribution is on the table, read the LICENSE file at the repository root and get proper advice.

Upgrade cost is dominated by the runtimes, not by vs-mlrt itself. vstrt requires rebuilding the engine on the target machine after a model or runtime change, which the README describes as an extra step relative to the other runtimes. The CUDA-backed plugins pull in CUDA libraries the README sizes at over 1GB, so a CUDA or TensorRT upgrade is a large download. vsncnn avoids both problems at the cost of speed. If you want upgrades to be cheap, that is the argument for Vulkan or for the OpenVINO CPU path.

Editorial conclusion

Adopt vs-mlrt if you already run VapourSynth and want waifu2x, Real-ESRGAN, Real-CUGAN, DPIR, RIFE or SCUNet inside a filter graph rather than as separate tools. Do not adopt it if you want a standalone upscaler with a GUI, or if you are not prepared to match a plugin to your GPU vendor: a CUDA-only build is useless on an AMD card, and the Vulkan plugin trades speed for portability. Before committing, verify that the release archive for your chosen runtime contains the plugin and its dependencies for your operating system, and confirm which models that runtime ships support, because the README points to the wiki for the model list rather than listing them itself.

Frequently asked questions

Official sources

1. AmusementClub/vs-mlrt on GitHub
2. Issues
3. License: GPL-3.0
4. README
5. Releases