vs-mlrt: One VapourSynth Interface, Six ML Backends
Efficient CPU/GPU ML Runtimes for VapourSynth (with built-in support for waifu2x, DPIR, RealESRGANv2/v3, Real-CUGAN, RIFE, SCUNet, ArtCNN and more!)
At a glance
- What is it?
- vs-mlrt bundles OpenVINO, ONNX Runtime, TensorRT, MIGraphX, ncnn and CoreML runtimes behind a single VapourSynth filter API, plus a Python wrapper that selects the backend for you. The trade-off is that you install the runtime that matches your hardware, and TensorRT and MIGraphX add an engine-building step before the first frame.
- Who is it for?
- Adopt vs-mlrt if you already script VapourSynth and want waifu2x, DPIR, RealESRGAN, Real-CUGAN, RIFE, SCUNet or ArtCNN applied inside an existing filter graph rather than in a separate GUI tool. Skip it if you do not run VapourSynth, or if you want a single binary that works on any machine without matching a runtime to the GPU.
- Can I use it commercially?
- Yes, with conditions. GPL-3.0 is a copyleft licence: if you distribute software that includes it, you must release that software's source code under the same licence. Running it internally without distributing it does not trigger that obligation.
- Is it still maintained?
- Yes. The repository last received commits 41 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: model runtimes multiply faster than your filter graph
VapourSynth filters are loaded as plugins, and each machine learning filter traditionally arrived as its own plugin with its own inference library baked in. That means a script calling waifu2x, then DPIR, then RIFE could pull in three separate builds, each with a different notion of which GPU it talks to and how a model file is specified. vs-mlrt collapses that by shipping several runtimes that expose a common filter interface, so a script can swap the backend without rewriting the filter calls. The README describes the project as providing "VapourSynth ML filter runtimes for a variety of platforms", and the platform list is the point: x86 CPU, Intel GPU, Nvidia GPU, AMD GPU and Apple SoC each get at least one supported path. The audience is people who already write .vpy scripts and want the model step to sit inside the same graph as resize, denoise and interpolation, not in a separate application that writes intermediate files. The named model coverage is waifu2x, DPIR, RealESRGANv2/v3, Real-CUGAN, RIFE, SCUNet and ArtCNN, with the README pointing to the wiki for the full supported list and usage details.
Six runtimes, one filter surface: how the plugin set is arranged
The repository is organised as one directory per runtime, and the naming is consistent enough to read as a family. vsov wraps OpenVINO for pure CPU and Intel GPU inference. vsort wraps ONNX Runtime for CPU and CUDA. vstrt wraps TensorRT, with a separate vstrt_rtx variant for RTX cards. vsmigx wraps MIGraphX for AMD GPUs. vsncnn wraps ncnn over Vulkan. vsort-coreml covers Apple SoC through ONNX Runtime. Each is installed the same way: download the release archive and extract it into the VapourSynth plugins directory, as the README states for vsov, vsort, vstrt, vsmigx and vsncnn in turn. Above that sits scripts/vsmlrt.py, a Python wrapper described as providing a unified interface to select different backends and covering all bundled models. That wrapper is where the abstraction actually lives. The plugins are separate binaries with separate dependencies; the wrapper is what lets one script target several of them. The README does not spell out the wrapper's function signatures, so the wiki is the place to look for the exact model-to-backend mapping.
Where each backend actually runs, and the one that runs everywhere
The hardware coverage is uneven by design, and the README is direct about it. OpenVINO targets x86 CPUs and Intel GPUs, with Intel GPU support documented as Gen 8+ on Broadwell+ and the Arc series. ONNX Runtime covers CPU and CUDA, and separately CoreML on Apple silicon. TensorRT is Nvidia only. MIGraphX is AMD only. ncnn over Vulkan is the outlier: because Vulkan is device-independent, the README says vsncnn supports all GPUs that provide a Vulkan interface, naming Nvidia, AMD, and Intel integrated and discrete parts. It also states the trade-off plainly, that the main drawback is that it is slower, and that its footprint is significantly smaller because the CUDA backends for vsort and vstrt require more than 1GB of CUDA libraries. That single sentence is the most useful deployment fact in the README. If disk or distribution size matters, Vulkan is the path. If throughput matters and you have the CUDA libraries anyway, it is not.
TensorRT and MIGraphX build an engine before the first frame
Most of the runtimes load a model and run it. TensorRT does not. The README explains that it uses benchmarking to find the optimal kernel for your specific GPU, so there is an extra step to build an engine from the ONNX network on the machine where the filter will run, and it acknowledges that this makes deploying models harder than the other runtimes. The same benchmarking approach is attributed to MIGraphX. Two consequences follow. First, a TensorRT engine is not portable between machines in the way an ONNX file is; the build is tied to the GPU it ran on. Second, any workflow that provisions a fresh machine or container needs that build step in the provisioning path, or a prebuilt engine cache carried alongside. The README does not describe where engines are cached or how to invalidate them, so that is a question for the vstrt directory documentation. TensorRT-RTX is offered as a middle ground: a specialisation for RTX GPUs that the README says compiles engines faster with comparable performance.
Getting it running: extract to plugins, then pick a backend
Installation for every runtime follows one instruction, repeated per section: download the latest release and extract it into your VapourSynth plugins directory. There is no package manager step and no build from source described in the README. After that, the entry point is scripts/vsmlrt.py, which the README describes as a Python wrapper for all bundled models with a unified interface for selecting backends. The repository's release history shows test builds rather than stable tags: v16.2.test1, v16.1.test1 and v16.test1, dated June through August 2026. Anyone pinning a version should read that naming as a signal about how the maintainers label builds. The README does not list the wrapper's function names or the backend selector strings, so a script cannot be written from this document alone; the wiki is referenced twice as the place for supported models and usage information. The per-runtime directories (vsov, vsort, vstrt, vsmigx, vsncnn) are the next stop for anything runtime specific.
The wrong tool: when a separate inference application fits better
vs-mlrt is a plugin collection for VapourSynth. If your pipeline is built on FFmpeg filter graphs, or on a standalone upscaling or frame interpolation application with a GUI, nothing here plugs into it. The plugins load into VapourSynth and the wrapper is a Python module for VapourSynth scripts, so the surrounding tool has to be VapourSynth. A second boundary is hardware. There is no single artifact that runs everywhere: vsov-gpu is documented for Intel Gen 8+ on Broadwell+ and Arc, vstrt is Nvidia only, vsmigx is AMD only, and vsort-coreml is Apple SoC. If you need one build that runs unchanged across a mixed fleet, vsncnn over Vulkan is the only candidate the README presents as device-independent, and it is also the one the README calls slower. A third boundary is the engine step already described: on TensorRT and MIGraphX, a disposable or rapidly recreated environment costs you a rebuild each time unless you carry the engine with the image.
Alternatives and the real difference in approach
The most direct alternative is to use the upstream inference tools the models ship with, running them as separate processes that read and write image sequences, and then feeding the result back into VapourSynth with a source filter. That approach keeps model execution out of the filter graph entirely. The difference is where state lives: a separate process reloads the model per invocation unless you build a server around it, while a vs-mlrt plugin holds the runtime inside the VapourSynth process for the duration of the script. For a single clip that is a real difference in overhead; for a batch of short clips it may not be. The second alternative is to pick one runtime directly rather than through this project: use ONNX Runtime's own tooling, or TensorRT's, and wire the results in yourself. That buys you the vendor's current documentation and release cadence, at the cost of writing the VapourSynth-side glue that vs-mlrt already provides through scripts/vsmlrt.py. The third is the narrow case: if you only ever run one model on one GPU, the value of a unified multi-backend interface is close to zero, and a single-purpose plugin or tool is less to install.
Licence, maintenance, and what the release cadence implies
The repository is GPL-3.0. That matters if you distribute scripts or a packaged build that includes these plugins, because the licence travels with the binaries rather than only with the source. This is not legal advice; read the licence text and, if you ship a product around it, get proper review. On maintenance, the last push recorded is 2026-08-06 and the three most recent releases are all labelled test builds, v16 through v16.2, spread across roughly two months. That cadence suggests active work on the version 16 line rather than a frozen release. The practical cost of upgrading is backend churn: a version bump can move which runtime version is bundled, and for vstrt and vsmigx a bundled runtime change may invalidate previously built engines, which means a rebuild on each machine. The README does not document an upgrade procedure or a compatibility matrix between plugin versions and engine caches, so treat that as unverified and check the per-runtime directories before replacing an installation that is working. The other recurring cost is disk: the README notes that the CUDA backends for vsort and vstrt require more than 1GB of CUDA libraries, against a significantly smaller footprint for the Vulkan path.
Editorial conclusion
Adopt vs-mlrt if you already script VapourSynth and want waifu2x, DPIR, RealESRGAN, Real-CUGAN, RIFE, SCUNet or ArtCNN applied inside an existing filter graph rather than in a separate GUI tool. Skip it if you do not run VapourSynth, or if you want a single binary that works on any machine without matching a runtime to the GPU. Before committing, check the wiki for the model you intend to use, confirm your GPU generation falls inside the documented range (Intel Gen 8+ on Broadwell+ or Arc for vsov-gpu), and on Nvidia decide between vsort-cuda and vstrt by whether you can accept a per-machine engine build step. Then verify the GPL-3.0 terms against how you distribute your scripts.
Community notes