QVAC SDK: running local AI from a TypeScript or Python codebase

What problem QVAC solves, and for whom

Most AI features in an application end up as an HTTP call to somebody else's server. That brings three costs: data leaves the device, the feature stops working offline, and the bill scales with usage. QVAC's answer is to move the model onto the device and keep the API surface small. The README describes it as an SDK for building local-first AI applications in JavaScript/TypeScript and Python, plus an HTTP server that exposes an OpenAI-compatible API so existing tools can point at your machine instead of a hosted provider.

The audience is therefore specific. It is a developer building a desktop, mobile or on-premises application who has already decided that inference must happen locally, and who wants one interface for several workload types rather than stitching together a llama.cpp binding, a Whisper binding and a diffusion pipeline. The repository topics list llama, whisper, stable-diffusion and openai-compatible, which matches the README's claim of a single SDK covering LLMs, speech, vision, image and video generation. The project is written in TypeScript, so the JavaScript path is the native one; Python is a client over the same worker.

It is not a hosted service and it is not a model. QVAC ships the runtime and the plumbing; the weights come from elsewhere, and the quickstart downloads them on first run. That distinction matters when you plan storage and first-launch behaviour.

How QVAC is put together: SDK, worker, and an OpenAI-compatible server

The shape visible in the README is a layered one. At the bottom is a worker process. The Python section states that the PyPI package is self-contained and bundles the QVAC worker and the Bare runtime, with no Node.js required. That tells you the Python client is not a reimplementation of inference; it is a client that talks to the same worker, which is why the Python example constructs a Client and then passes client.transport into every call. The transport object is the channel to the worker.

Above that sits the SDK. Its surface in the JavaScript quickstart is four functions: loadModel, completion, unloadModel, and a model constant such as LLAMA_3_2_1B_INST_Q4_0. loadModel returns a modelId, and every later call takes that id, so model lifetime is explicit and manual. There is no garbage collection of loaded models. The completion call takes a history array in chat format and returns an object with a tokenStream that you iterate with for await. Streaming is a first-class option, not a wrapper.

The third layer is the HTTP server, installed as the @qvac/cli package, which the README says also pulls in @qvac/sdk as a transitive dependency. That server is what makes the OpenAI-compatible claim concrete: the README names OpenCode and OpenClaw as tools you can connect. Configuration is read from a file pointed at by QVAC_CONFIG_PATH, and the README's example config enables console logging with loggerConsoleOutput and sets loggerLevel to info. Those two keys are the only configuration surface the quickstart shows, and the README does not document the rest of the schema.

Installing QVAC and running a first completion in JavaScript

The JavaScript path is the shortest. Create an examples directory, make it an ES module package, and install the SDK from npm.

mkdir qvac-examples
cd qvac-examples
npm init -y && npm pkg set type=module
npm i @qvac/sdk

Next, create qvac.config.json in the same directory. The README uses it to turn on client and server logging during the run, which is useful while you are still working out whether the worker started.

{
  "loggerConsoleOutput": true,
  "loggerLevel": "info"
}

Now the script itself. It loads a quantised Llama 3.2 1B model, prints download progress to stderr, streams the answer to stdout, and unloads the model. Write this as quickstart.js.

import { loadModel, LLAMA_3_2_1B_INST_Q4_0, completion, unloadModel } from '@qvac/sdk';

const modelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  onProgress: (p) => process.stderr.write(`Downloading ${p.percentage.toFixed(0)}%\n`),
});
const result = completion({
  modelId,
  history: [{ role: 'user', content: 'Explain quantum computing in one sentence' }],
  stream: true,
});
for await (const token of result.tokenStream) process.stdout.write(token);
await unloadModel({ modelId });

Run it with the config path set, and the README says you will see the model download first, then streamed tokens in the terminal.

QVAC_CONFIG_PATH=./qvac.config.json node quickstart.js

The Python route is shorter in code but longer in setup, because the install command is not a plain pip install. The README instructs you to point pip at the GitHub release asset page for a specific tag.

pip install tetherto-qvac-sdk \
  -f https://github.com/tetherto/qvac/releases/expanded_assets/sdk-v<version>

The placeholder is deliberate: you substitute a release such as sdk-v0.17.0, which is the example the README gives. After that, python quickstart.py behaves the same way as the JavaScript version, with the model downloading and tokens streaming.

The install path is the weakest part of the documentation

The Python install is the clearest friction point. A self-contained package that bundles a worker and a runtime is a reasonable design, but the documented way to get it is to resolve wheels from an expanded_assets URL for a named release tag. That means the version is not discovered by pip's resolver; you name it. It also means the -f flag is doing the work that an index normally does, so a typo in the tag produces a resolution failure rather than a helpful message. The README gives sdk-v0.17.0 as its example while the release list shows sdk-v0.19.1 as the most recent SDK release, so the example is behind the releases. Nothing in the README says whether the package is published to PyPI in a way that a plain pip install would find, and no upgrade command is documented.

The model download is the second sharp edge. The quickstart downloads weights on first run, and the README's progress callback exists precisely because that takes time. There is no documented way to pre-seed the model cache or to point loadModel at a local GGUF file, even though the repository topics include GGUF. For a mobile app, that first-run download is a product decision, not an implementation detail.

Third, the .env.example in the repository root lists GH_TOKEN, HF_TOKEN and NPM_TOKEN, with HF_TOKEN described as required for model licence verification. That is a build and release concern rather than a runtime one, but it signals that model licensing is handled inside the project's tooling, and it is worth knowing before you assume any GGUF you find will load.

QVAC against calling a hosted API, and against a single-purpose runtime

The obvious alternative is a hosted inference API. The difference is not speed, it is where the boundary sits. With a hosted API you send the prompt and the data it contains to a third party, you need a key, and the feature degrades to an error when the network does. With QVAC the README's stated position is that no cloud or third-party APIs are required, so the prompt and the model both stay on the machine. The trade is that you now own the download, the disk footprint, the memory pressure and the model version. Hosted providers absorb all four.

The second alternative is a single-purpose local runtime, for example a llama.cpp binding for text and a separate Whisper binding for speech. That approach gives you a smaller dependency per task and lets you pick the best implementation for each. QVAC's counter-argument is one interface and one worker for text, speech, vision and generation, which is less glue code and one process to manage. The cost is that you inherit QVAC's model catalogue and its release cadence for every capability, including the ones you do not use. If your application only ever needs text completion, a narrower binding is the smaller commitment.

A third comparison the README invites is peer-to-peer model distribution. The project describes fetching models directly between peers, in the style of BitTorrent or IPFS. That is a different answer to the download problem than a CDN, and it is the part of the design with the least detail in the README.

Maintenance, licensing and what an upgrade actually costs

The repository is not archived, and the last push was on 2026-09-15. The published releases show sdk-v0.18.2 on 2026-08-26, sdk-v0.19.0 on 2026-09-07 and sdk-v0.19.1 on 2026-09-11, so the SDK is being cut at a pace of roughly one release every week or two. That cadence is a real cost if you pin versions: you will be deciding whether to move every few weeks, and no long-term support line is published.

The licence is Apache-2.0, which is a permissive licence with an explicit patent grant and a requirement to preserve notices. That covers the SDK code. It does not cover the model weights, and the two are separate questions. The presence of HF_TOKEN in .env.example for model licence verification indicates the project treats weight licensing as something to check rather than something it can grant. If you plan to ship a model inside a commercial product, read the licence attached to that specific model; the Apache-2.0 header on this repository says nothing about it. Nothing here is legal advice.

The upgrade surface itself is small on the JavaScript side, because the documented API is four functions and a config object. The Python side is where upgrades bite, since the install command embeds a version tag. Moving from one release to the next means editing that tag and re-resolving from the release assets, which is a manual step in any CI pipeline that builds the environment.

Editorial conclusion

Adopt QVAC if you are building a JavaScript or TypeScript application that must run inference without a cloud call, or if you want an OpenAI-compatible endpoint on your own machine for tools that already speak that API. Do not adopt it if your stack is Python-first and you want a normal PyPI install, because the documented command resolves wheels from a GitHub release asset page and pins an exact release. Before writing production code, verify two things: that a pinned sdk-v<version> release actually publishes the wheel for your platform, and that the model you intend to ship is covered by the licence check the SDK performs at load time.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. tetherto/qvac on GitHub