Fuji-Web: a browser sidepanel agent that drives the page you are already on

The problem Fuji-Web targets: a task on a page you are already looking at

Most browser automation starts from a script. You write a selector, run it headless, and hope the site has not changed. Fuji-Web starts from the opposite end. The README describes an agent that "understands the user's intent, navigates websites autonomously, and executes tasks on the user's behalf while explaining each action step." The intended user is someone sitting in front of a page who wants a one-off action performed there and then, without writing Playwright code or opening a terminal. The unit of work is a sentence typed into a sidepanel, not a committed script. That framing has consequences for who benefits. A developer automating a nightly scrape gains little. A person who repeatedly fills the same awkward form, or who wants a page summarised and acted on, is the target. The README's own phrasing, "get tasks done online with a single command," sets the same expectation: interactive, single-session, human-in-the-loop.

How the annotation and prompting pipeline works

The README credits Microsoft's UFO paper as the inspiration for "Fuji-Web's image annotation method," and that is the clearest architectural signal available. Rather than relying on DOM selectors alone, Fuji appears to annotate interactive elements in a rendered view of the page so the model can refer to them by label. The extension then sends the task plus page content to the model API. The README is explicit about the data path: "all prompts (text and image) are sent directly to the API of your selection," and the API key "will be stored in your browser, and will not be uploaded to a third party." So the flow is: sidepanel captures the task, the extension captures the current page state, the model returns an action, the extension executes it, and the loop repeats with a stated explanation of each step. What the README does not document is the retry policy, the step ceiling, or how it recovers when the model picks an element that no longer exists. Those are the questions that decide whether an agent is usable on a real site, and the repository does not answer them in the material available here.

Installing from a release versus building from source

There are two paths, and they differ in trust. The release path is manual and unpacked: download fuji-extension.zip from the releases page, unzip it, open chrome://extensions/, toggle Developer mode, click Load unpacked extension, and select the unzipped folder. The README warns that "you may need to refresh the page for the extension to work." The source path assumes Node.js (development was done on Node v20), then npm install -g pnpm, pnpm install, and either pnpm dev for the development server or pnpm build for a production bundle. When loading a self-built copy you must point Chrome at the dist folder, not the repository root. The latest listed release is v2.2.1 from 2024-09-20, while the last push to main is dated 2026-08-30, so the release artefacts and the current source are not the same thing. If you need behaviour that landed after September 2024, the release zip will not contain it and you have to build.

The API key model and what it means for cost and privacy

Fuji-Web ships no backend of its own for inference. You paste an OpenAI or Anthropic key into the sidepanel, and the README states it is stored in your browser. That design removes a subscription and a vendor account, but it moves three problems onto you. First, cost: every step in a multi-step task is a model call, and image prompts are typically the expensive kind, so a long task on a visually complex page can consume a meaningful amount of credit. The README gives no token estimates or per-task cost figures, so you should measure on your own workload before committing. Second, key hygiene: a key sitting in extension storage is reachable by anything with access to that extension's context, and the README does not describe encryption at rest. Third, data exposure: the README says prompts go directly to your chosen API and that Fuji-Web "does not attempt to collect any information from you," but the page content itself is the payload, so the privacy boundary is the model provider's policy, not Fuji's.

Single-tab scope and the workflows the roadmap has not shipped

The roadmap is the most useful part of the README because it is a list of what does not exist. It names an API for integration with Puppeteer, Playwright and Selenium; support for complex and cross-tab workflows; more browsing behaviours such as selecting from a dropdown and extracting content from an entire page; saving workflows; sharing workflows and instructions; and a shared knowledge base. Each of those is a current gap. Practically, that means Fuji-Web is a poor fit for anything spanning multiple tabs or windows, for scheduled or headless runs, and for repeatable automation you want to hand to a colleague. The roadmap item about extracting content from an entire page is worth reading twice: if whole-page extraction is listed as future work, the present extraction behaviour is narrower than a reader might assume. This is the clearest case where Fuji-Web is the wrong tool. If your task is "run this flow every morning across three sites," a scripted tool is the right answer and Fuji-Web is not.

Fuji-Web against scripted automation such as Playwright

Playwright and similar frameworks take a declarative approach: you write the steps, they run deterministically, and failures are reproducible because the failing selector is in your source. Fuji-Web takes an inferential approach: the model decides the next action from the page state, which tolerates layout changes a hard-coded selector would not survive, at the cost of non-determinism and per-step API calls. Neither is strictly better. The trade is predictability and cost against adaptability and setup effort. Fuji-Web's own roadmap points at the convergence: an API for Puppeteer, Playwright and Selenium would let the agent be invoked from inside a scripted harness, which is the combination most teams actually want. Until that ships, choosing Fuji-Web means accepting an interactive, one-tab, model-driven loop, and choosing Playwright means accepting that you maintain selectors. The README also credits TaxyAI's browser extension as the inspiration for the sidepanel concept and notes that some UI code was reused, so if the sidepanel pattern is what appeals to you, that project is the direct ancestor.

Maintenance surface, licence and upgrade cost

Fuji-Web is Apache-2.0, which permits commercial and closed-source reuse provided you keep the licence and notice files and state significant changes; this is a description of the licence text, not legal advice, so read the LICENSE file and the NOTICE requirements before redistributing a modified build. The practical maintenance burden comes from two directions. The extension is built on Jonghakseo/chrome-extension-boilerplate-react-vite, so Manifest and Vite toolchain changes upstream eventually reach you. And the agent depends on model APIs whose request shapes and model names change on the provider's schedule, not yours. Because the latest release predates the latest commits, upgrading means either waiting for a new release zip or building from main yourself and re-loading the unpacked extension. There is no published release cadence in the material here, and the gap between v2.2.1 and the most recent push is wide enough that you should treat the release channel as slow-moving.

Editorial conclusion

Adopt Fuji-Web if you want a self-hosted, single-tab agent that explains each action and you are willing to fund it with your own OpenAI or Anthropic key. Do not adopt it if you need multi-tab workflows, saved or shared automations, or headless execution in CI; the roadmap lists all of those as unbuilt. Before trusting it, verify the extension's permissions in chrome://extensions/ and confirm that a task on a page with sensitive session data does not send more of that page to the model than you expect.

Official sources

1. Issues
2. License: Apache-2.0
3. normal-computing/fuji-web on GitHub
4. README
5. Releases