# FluidUse: on-device form filling through the macOS Accessibility API

> FluidUse reads a form in a running Mac app through the Accessibility API, asks a small Core ML model what belongs in each field and types the answer in, with about 1 ms per decision on the Neural Engine. It also ships laya, a typed decision model that answers choice, score and probability questions with no generated tokens.

**FluidInference/FluidUse** — Local computer use on Apple silicon: on-device typed decisions with laya and form filling with CUA-S1-FORMS, driven through the macOS Accessibility API

- Repository: https://github.com/FluidInference/FluidUse
- Website: https://huggingface.co/FluidInference
- Stars: 525 · Forks: 40
- Language: Swift
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/fluidinference-fluiduse

## Local computer use driven through the macOS Accessibility API

FluidUse is described as local computer use on Apple silicon, and the mechanism is specific enough to be interesting. It reads a form in a running Mac app or browser through the Accessibility API, asks a small on-device model what belongs in each field, and types the answer into the real app. The README states the performance claim plainly: about 1 ms per decision on the Neural Engine, and nothing leaves the machine. The two claims fit together, because a model small enough to run a thousand times a second on the Neural Engine is also small enough to keep entirely local. The primary language is Swift, the licence is Apache-2.0, and the releases are v0.1.0, v0.2.0 and v0.3.0 in September 2026 with the last push on 2026-10-14, so this is a young project with a fast start.

## The first model is a 706K parameter form specialist

The form filling path starts from CUA-S1-FORMS, a 706K-parameter form specialist from Cua, converted to Core ML and served by FluidInference. The size is the point. This is not a general-purpose model that has been shrunk; it is a model built for one job, filling in form fields, and it is small enough to answer in about a millisecond. The README also points to Documentation/DecisionModelSupport.md for the current Core ML model matrix and handoff notes, which is the document to read before choosing which model to run. The project publishes its models on Hugging Face under the FluidInference organisation, which is the same organisation behind the laya models discussed later.

## The API is snapshot, render options, score, then type

The usage example in the README is the whole flow. The package is added as a Swift Package Manager dependency:

```swift
.package(url: "https://github.com/FluidInference/FluidUse.git", from: "0.3.0")
```

Then the code loads a model, builds a driver, takes a snapshot, renders the options, scores each actionable field and acts on the result:

```swift
import FluidAudio
import FluidUse

let model = try await CuaS1FormsManager.load()
let driver = AccessibilityFormDriver(application: safari)   // any NSRunningApplication
let page = try await driver.snapshot()                       // fields, labels, values
let options = FormSchema.renderOptions(entities: profile)    // "fill Email: …", check, click, skip

for field in page.elements where field.isActionable {
    let context = FormSchema.renderContext(formTitle: page.title, element: field)
    let decision = try await model.score(context: context, options: options)
    // decode with FormSchema.decode, then driver.type / driver.click
}
```

The steps are worth naming separately because they map to distinct concerns. `CuaS1FormsManager.load()` brings the model up. `AccessibilityFormDriver(application: safari)` takes any `NSRunningApplication`, so the target app is not special-cased. `driver.snapshot()` returns the page as fields, labels and values. `FormSchema.renderOptions(entities: profile)` turns a profile into the option set, and the comment spells out the shape: fill Email, check, click, skip. `FormSchema.renderContext(formTitle:element:)` builds the context for one field, and `model.score(context:options:)` returns the decision, which is decoded with `FormSchema.decode` and then executed with `driver.type` or `driver.click`.

The example passes `entities: profile`, and the README explains where a profile comes from. `DocumentEntities` turns a PDF or text file of `Label: value` lines into the profile, which is a pleasantly ordinary data format rather than a model-specific one. There is also an optional `PredeterminedAnswer` sheet covering question-style fields the model does not decide, which is a sensible division of labour: the model handles the fields where inference is useful, and the operator answers the rest by hand. The same driver pattern applies to web content, since `WebFormDriver` does the same for an embedded `WKWebView`. That symmetry is what makes the tool usable against both native apps and browsers without a second integration.

## laya is a typed decision model, not a generator

The second model is laya, from Convai Innovations and Apache-2.0 licensed, described as an open Jev-style decision model. The mechanism is a 322M mmBERT encoder plus a decision head that answers typed questions about a text state with calibrated probabilities in one pass, with no generated tokens. The three question types are `choice`, `score` and `noul`, and the example shows all three asked about the same customer message:

```swift
let laya = try await LayaManager.load()  // downloads the 128 + 512 buckets and tokenizer.json
let answers = try await laya.answer(
    state: "Customer: I was charged twice for order #4471 and nobody replies. Refund me today.",
    questions: [
        .choice("What does the customer want?", options: ["refund", "order status", "technical help"]),
        .score("How frustrated is the customer?", levels: ["calm", "annoyed", "angry"]),
        .noul("Is the customer likely to churn?"),
    ])
print(answers[0].selectedLabel, answers[1].expectedScore!, answers[2].noul!)
```

The `score` question returns an `expectedScore`, and the `noul` question returns a probability, so a caller can branch on a numeric value rather than parse text. That is the practical difference from a text-generating model. Nothing has to be parsed, the answers are typed, and the whole thing is one forward pass. The README's benchmark table lists the accuracy figures per suite and a median of 5.2 ms per question against 61.6 ms for the PyTorch reference on the same Mac's CPU, with the full tables in Benchmarks.md.

## Bucket selection and weight precision are configuration, not magic

The Core ML runtime detail is in `LayaManager.Configuration`, and it explains how the latency claim holds. The configuration picks which buckets to load, what compute units each uses, and the weight precision. The buckets come from the laya-coreml package on Hugging Face and cover 128, 256, 512 and 1024 tokens with 32 options. For compute units, 128 goes to CPU plus ANE while longer inputs use all units, and the weight precision is either `fp16` or `e8`, the latter with an int8 embedding table at 30 percent less weight and the same accuracy. A prompt then runs on the smallest loaded bucket that fits, and the largest one truncates the state on the right in the same way as laya's own `max_len`. The tokenizer is a Swift port of the mmBERT and Gemma byte-fallback BPE that matches the Hugging Face `tokenizers` behaviour on the conversion fixtures.

## Command line entry points for answering, and for playing games

The README documents a set of command line invocations, and they are worth reading because they show how the model is used away from a user interface. A question can be asked directly:

```bash
swift run -c release FluidUseLaya answer --state "…" --type choice \
    --instructions "What does the customer want?" --options "refund|order status|technical help"
swift run -c release FluidUseLaya tetris --shortlist --describe graded --pieces 200  # headless Tetris, P(clean) per landing
swift run -c release FluidUseLaya 2048 --precision lut8 --games 10
swift run -c release FluidUseLaya benchmark --suites <mobius>/benchmark/suites.jsonl --reference <mobius>/benchmark/reference-rows.jsonl
```

There are also game harnesses in the same tool, a headless Tetris mode with `--shortlist` and `--describe graded` that reports a cleanliness probability per landing, and a 2048 mode with `--precision lut8` and a game count, plus a `benchmark` subcommand that takes a suite file and a reference rows file. The reasoning is that a decision model is a general mechanism, and if it can pick a form field it can pick a Tetris landing. The README reports a ten-seed capped run averaging 3,667 pieces for the GLiClass policy against 2,874 for the corrected heuristic control, and the SwiftUI demos in Sources/ put the same comparison in a window next to a terminal.

## GLiNER 2.5 adds on-device classification, with a size warning

The third model family is GLiNER 2.5, run on device by `GLiNER2Manager` using the published base or multilingual classification head. The example loads a variant and asks a labelled question:

```swift
let classifier = try await GLiNER2Manager.load(variant: .base)
let answer = try await classifier.classify(
    text: "The rocket launched successfully.",
    task: "topic",
    labels: ["science", "sports", "politics"])
print(answer.selectedLabel, answer.probabilities)
```

The manager downloads pinned, SHA-256 checked Core ML assets, the source config files and the matching Unigram tokenizer on first use, from either the base or the multilingual package. The constraints are stated: both packages use an L128 bucket, support up to eight labels, and quantize only the embedding table to eight bits. The sizes are the part to notice, with the base package at 291 MB and the multilingual at 385 MB, and inputs exceeding 128 tokens raise an error. The README also draws a clear boundary, saying these packages contain the classification path while the native entity, relation and record extraction heads are not exposed by this Swift manager.

## Where FluidUse is the wrong tool, and how a cloud model differs

Three limits are concrete. The first is that it only runs on Apple silicon, and the constraint is deeper than a build target. The whole latency story rests on the Neural Engine, and the model is a Core ML conversion, so this is macOS on Apple hardware and not a portable technique. The second is the accessibility dependency. Reading a form through the Accessibility API and typing into it means the app must expose its controls to that API, and a canvas-drawn interface, a custom-drawn control or a game with no accessible elements will not appear as fields to fill. That also explains the presence of a headless Tetris and 2048 harness: those are applications with no form fields at all, so the project had to build a different way of exposing state. The third is that the model matrix is a moving target. The README explicitly points to a model support document for the current Core ML model matrix and handoff notes, which is a way of saying the set of models and their Core ML conversions change, and a project with three releases in five days is very likely to reshape its API.

The realistic alternative is to send a screenshot of the form to a hosted vision model and parse the response, which is what most computer-use systems do. The difference in approach is where the inference happens. A cloud model can read anything visible on screen, including a form rendered in a way no accessibility tree describes, and it can be much larger, so it handles unexpected layouts better. The cost is a round trip to a data centre for every decision, which for a one millisecond on-device decision is several orders of magnitude, plus a screenshot of the user's screen leaving the machine on every field. FluidUse's pitch is the opposite trade: give up generality, gain local speed and no data leaving the device. For a tool that fills a known form repeatedly, that is the right side of the trade. For anything where the interface is unknown or the form is complex, a cloud model with a human in the loop is more likely to work, and the 706K parameter specialisation here is the constraint to check against your own forms.

## Conclusion

FluidUse fits a macOS on Apple silicon team that needs to fill the same forms repeatedly and quickly, and that needs the screen content to stay on the machine. It is the wrong choice for a cross-platform workflow, for interfaces that expose nothing to the Accessibility API, and for anything where a large model reading arbitrary screen content is needed. Before you adopt it, read Documentation/DecisionModelSupport.md for the current Core ML model matrix, check that the model you need has a Core ML conversion at all, and confirm the 291 MB and 385 MB GLiNER downloads are acceptable for the machines you will deploy to.

## FAQ

### What is FluidUse and how does it interact with applications?

It is a Swift package for local computer use on Apple silicon. It reads a form in a running Mac app through the Accessibility API, asks an on-device model what belongs in each field, and types the answer into the app.

### Which models does FluidUse ship with?

The first is CUA-S1-FORMS, a 706K parameter form specialist from Cua converted to Core ML. It also runs laya, a 322M mmBERT encoder plus decision head for typed choice, score and probability questions, and GLiNER 2.5 for on-device classification.

### How do I add FluidUse to a Swift project?

Add the Swift Package Manager dependency with `.package(url: "https://github.com/FluidInference/FluidUse.git", from: "0.3.0")`, then import FluidAudio and FluidUse and load a manager such as `CuaS1FormsManager` or `LayaManager`.

### Does FluidUse work on non-Apple platforms?

The README describes the project as local computer use on Apple silicon, driving the macOS Accessibility API and running Core ML models on the Neural Engine, so it is a macOS project. The current Core ML model matrix is documented in Documentation/DecisionModelSupport.md.

## Sources

- [FluidInference/FluidUse on GitHub](https://github.com/FluidInference/FluidUse)
- [License: Apache-2.0](https://github.com/FluidInference/FluidUse/blob/main/LICENSE)
- [Project website](https://huggingface.co/FluidInference)
- [README](https://github.com/FluidInference/FluidUse/blob/main/README.md)
- [Releases](https://github.com/FluidInference/FluidUse/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/fluidinference-fluiduse