jarvis: a browser voice assistant whose brain is a coding agent

What it is

JARVIS is a browser voice assistant with a holographic interface. You say a wake word, it wakes, listens, and does real things through your tools: it searches the web, generates images, drives your phone, reads your mail. The face is a web page, built with a component library, a bundler, a 3D library and custom shading code. The brain is the coding agent, run headless as a library.

The subscription story is the first thing the readme leads with, and it is the practical reason to try the project. The only subscription needed is the coding agent one. There is no API key, no separate account with another provider, and no separate cloud bill, because the brain runs on the login you already have for the agent, and the model itself runs on that provider's servers. So even a low-end machine only has to draw the interface.

And then there is the optional extra: a key for a speech service, which gives the assistant a much better voice and sharper hearing. Without it, the assistant speaks and listens through the browser's own speech facilities, and everything still works.

That is a well-chosen framing, because the alternative design, a key for the language model and a key for speech, doubles both the setup and the bill. This one has a floor and an upgrade, and the floor needs nothing but a browser and the agent you already have.

The requirements section states the floor in one line: the agent subscription, plus two free things every computer can have, which are a JavaScript runtime and Chrome. And then it breaks it down with the kind of detail that usually only appears after somebody has been confused. The agent has to be installed and logged in, and the readme gives both the package manager command and the platform installer. The runtime has a version floor. The browser has to be one of the two named ones, in a real window, and the reason is given: preview panes, including the one inside editors and inside the agent's own interface, block microphone access, so the page loads and looks right and never hears you. The 3D interface also needs the graphics support those browsers provide.

And there is a setup script that checks all of this for you, in plain language, which is the right way to handle a set of requirements that are not really requirements but a set of conditions that have to be true.

npm install
npm start          # runs the brain and the face together

That is the whole installation: two commands, and then the page asks for the microphone and waits for the wake word.

Two processes, and why the bridge exists

The architecture diagram in the readme is the clearest part of the document, because it shows that the system is two processes with a specific division of labour.

The browser is the face and the voice: the wake word, a local voice activity detector, speech to text, the 3D interface, text to speech, and a heads-up display. The bridge is the brain and the hands: a Node process, the agent SDK, and a permission gate that decides whether a tool may run.

They talk over a WebSocket, plus a few ordinary HTTP endpoints, on a local port. Everything you see and hear happens in the browser; the bridge is a single Node process that runs the agent SDK, which spawns the real command line agent as a child process. The readme's phrasing is that the brain literally is the agent, running headless.

And then the question the readme answers, which is why a bridge at all. A browser tab cannot spawn the local servers that speak the standard tool protocol, the ones for the phone, the browser automation, the speech service, the image services, the search services. The bridge can, because it is a Node process on your machine.

That single sentence is the whole justification for the architecture, and it is worth pausing on, because it is the kind of thing that could have been solved differently. A browser extension with native messaging could do it. A local server is simpler, and simpler is what a project you are going to run on your own laptop wants.

The second reason the bridge exists is authentication. Because it uses the agent SDK, it authenticates off the login you already have for the agent, so there is no key to configure and the usage is billed to the same account. And the model and its effort are configuration: the readme names the defaults, and says they can be overridden with two environment variables, and that the bridge prints its choice on startup so you can see what it is actually running.

Printing the choice on startup is a small courtesy with real value. A voice assistant that is doing the wrong thing is hard to debug, and the first question is always which model it thinks it is using.

The voice pipeline, and how it fails gracefully

The voice section is designed, the readme says, so that nothing silently dies and so that interrupting it feels natural. The three stages each have a specific approach to failure, and the whole design is about not being the assistant that just stopped responding.

Detection is local. An energy-based voice activity detector in its own module decides when you are speaking. It is instant, it cannot quietly fail, and it is what makes interrupting the assistant work: speak while it is talking and it stops.

That last point is the interesting one. Voice activity detection is normally done in the browser's own speech recognition, which means the assistant is deaf while it speaks, because the microphone is not being read. Doing it locally with a simple energy threshold means the microphone is always being read, so the assistant can notice that you have started talking and stop. It is a crude algorithm, and it is the right one, because the property that matters is that it always runs.

Transcription has two tiers, chosen automatically at boot. The browser asks the bridge for its health and picks the best available: if a key for the speech service is present, that service's transcription is used through a bridge endpoint; if nothing is configured, the browser's own speech recognition is used, guarded by a heartbeat so that it recovers when the browser throttles it.

That heartbeat detail is a real-world lesson. Browsers throttle background speech recognition, and without a watchdog the feature dies silently after the assistant has been running for a while, with no error anywhere. A heartbeat that notices the silence and restarts the recogniser is the difference between a feature that works for a demo and one that works for an afternoon.

Speaking follows the same pattern. The cloud voice is used when a key is present, and the browser's own synthesis otherwise. If a cloud call fails, it falls back to the browser voice; and if the operating system's voice is itself broken, it latches over to the cloud voice, meaning it stays there rather than retrying a path it knows is failing.

And the capability detection lives in one module, which probes the bridge's health endpoint once at start-up and reports which of the two engines are available. One probe, one decision, at boot. That is the cheapest possible place to make the choice, and it is why the readme can say that the assistant works with no keys and upgrades by itself when a key appears, with no flag to set.

The tools, and what the bridge will and will not see

Beyond answering questions, the assistant reaches every tool server in your agent configuration, and it can drive its own interface.

The tool servers are handed to the agent SDK explicitly, from a local configuration file, and the readme lists the categories with the example names: web and search, with several providers; images and video, again several; voice; your phone; and the browser, through the browser automation server. Then a few example phrases: ask what is happening in the field this week, generate an image of a particular suit, take a screenshot of the phone, open the notifications on a code hosting site.

The last one is the reason the local configuration file matters, and the readme is explicit about the limitation in a note about account connectors. Servers added through the web account of the agent are not stored on disk, so the bridge cannot see them. It works from the servers in the local configuration file, of which there are about fourteen, and not from the ones attached to the account.

That is a real and interesting boundary. The tool ecosystem of a coding agent has two halves, the ones you installed locally and the ones you connected through the web interface, and only the first half is available to a local process. The readme states it plainly rather than letting somebody discover it when a tool they expected is missing.

Then there is the interface control, which is the part that makes the assistant feel like a character rather than a chatbot. The bridge exposes tools for the interface itself: one to set the theme, with the accent colour, the background and the colours per phase; one to control the reactor, with its colour, scale, intensity, spin and style, where the style is one of a ring, a sphere or a wire, plus its visibility; and one to put images into orbit around the reactor.

So the assistant can change how it looks depending on what it is doing, and it can put the image it just generated into the scene around itself. That is a small thing with a large effect on the impression the thing makes, and it is implemented the only sensible way: as tools on the same protocol as everything else, so the model decides to use them the same way it decides to search the web.

The permission gate, and the two audiences for configuration

The environment example file is the most instructive document in the repository, and it is where the project's safety model is written down.

The headline is that everything in it is optional in the default mode. The assistant runs with an empty file, because the bridge authenticates off the agent login and the voice falls back to the browser's own speech. The one line worth adding is the speech service key, and the file says never to commit a real key.

And then the section that matters most: a variable that allows effectful tools. The comment lists what that means: send, tap, delete, install, pay. And the file says it is off by default, so the assistant can look at anything and change nothing, and that the script for enabling writes sets it for you, and tells you to read the gate in the bridge source first.

That is a well-designed safety story. The default is read-only, so an assistant that mishears you and decides to do something cannot, and the tools that can act are named explicitly in the documentation rather than hidden. And the permission to enable them is a deliberate act: either an environment variable or a separate script whose name says what it does.

The second thing the environment file explains is the two audiences. The bridge reads its variables from the shell that starts it, while the front end reads the ones prefixed for the bundler from a local file. And the rule that falls out of that: only the prefixed values ever reach the browser. That is the standard arrangement for a bundler, but stating it explicitly in the file is worth doing, because the alternative, a single environment file read by both, would put a credential in the browser's hands.

The file also shows the shape of the configuration as a whole. The port the bridge listens on, with a note to change the front end's value to match if you change it. The model and the effort, with their defaults. And the write gate, with its default of off. Each is a comment saying what it does, what it defaults to, and what happens if you get it wrong, which is exactly what a configuration file should contain when the person reading it is a user rather than a maintainer.

The repository and the scripts

The top level of the repository is a modern front end project with a small server beside it.

There is the source directory for the browser application, the bridge directory with its server, a scripts directory, the public directory for static assets, the entry page, a bundler configuration, and the type configurations. The type configuration comes in four files, one for the application, one for the node side of the build, and and one for the base settings, which is the arrangement a bundler-based project needs when the same code base is compiled both for the browser and for the server.

The lint configuration is for a fast linter written in the same style as the bundler ecosystem, and the linter script in the manifest is a single word, which is the sign of a tool that is expected to be run constantly.

The manifest's scripts are the operational surface of the project, and there are eight of them, and two of them are about the same thing in different modes. The start script runs a small Node program that launches the brain and the face together, which is the one-command path. The dev script runs the bundler alone, and the bridge script runs the server alone, which is the two-terminal path for anybody who wants to see the logs of each.

And then the one I would point at. There is a second bridge script whose only difference is that it sets the write gate to enabled before starting the server. Which means the safe mode is the default command, the unsafe mode is a different command, and the difference is visible in the command name rather than hidden in a configuration file. That is a good design decision: the user who has never read the documentation runs the safe one, and the user who wants the unsafe one has to type a different command whose name says what it does.

The build script type checks the whole project and then builds the front end, the setup script is the one that checks the requirements in plain language, and the package manifest declares the runtime version floor.

The dependencies tell the rest of the story. There is the agent SDK, the 3D library and its React bindings, a post-processing library, an animation library, a state store, a schema validator, a sanitiser, a WebSocket server, a wake word detection library with its voice processing companion, a speech synthesis library, and a media pipe vision library. So the wake word is a dedicated commercial engine, the 3D interface is a proper scene graph rather than a canvas, and the state is kept in a small store rather than in a framework's own machinery.

Which is a fairly complete picture of a project that is one weekend of somebody's work and more careful than most.

Editorial conclusion

JARVIS fits a developer who wants to drive their own tools by voice and who already pays for a coding agent subscription, and who is comfortable running a local server that can spawn their tool servers. It does not fit somebody who wants a general assistant with no setup, and it is not a product with a support team behind it. Verify first which of your tool servers are on disk, because the bridge only sees the ones in a local configuration file and not the ones attached to a web account. Check that the read-only mode is what you run at first, because the write gate is a single environment variable away and the tools behind it include sending, installing and paying, and that the browser requirement is a real one, since an embedded preview pane will look perfectly alive and simply never hear you.

Frequently asked questions

Official sources

1. adewaskar/jarvis on GitHub
2. Issues
3. License: MIT
4. README