activity-frames: compiling screen capture into replayable agent workflows over MCP

The gap activity-frames targets: agents that re-derive every task

The README states the problem directly: computer-use agents work every task out from scratch, even one you have done a hundred times, and between tasks the agent has no idea what you have been doing all day. That is two complaints bundled into one package. The first is cost. Re-deriving a familiar workflow means screenshot, reason, act, repeat, and the token spend lands on steps that have not changed since last Tuesday. The second is context. A fresh conversation starts blind because nothing in the model's input describes your actual day.

The intended user is someone running an agent that can drive a browser or a desktop, and who is willing to grant local screen recording. It is not a note-taking app and it is not a conversation memory layer. The README draws that line itself: agent memory today means conversation memory, what you told the model, and what you actually did is the missing half. If your agent's problem is that it forgets what you said, this project is aimed at a different problem.

Two tiers, and why the measured tier carries no model

The architecture described in the README is a compile pipeline, not a summarisation pipeline. Capture stores instants: the README gives the example of thousands of snapshot rows a day, each saying that at 22:53:05 Chrome showed a particular LinkedIn profile URL. Those rows are described as useless to reason over, which is fair. The compiler turns them into activity frames: bounded records with an app, a site, a start and end time, a duration, a list of pages with a kind and an entity, an input tally of keys and clicks, and an evidence field pointing back at the frame_ids range the record was derived from.

That evidence field is the part worth noting. A frame does not assert a fact without naming the span of raw capture it came from, so you can walk a claim back to instants. The README describes a two-tier contract in SPEC.md: tier 1 is measured and derivable by deterministic means, and the package itself is tier 1. The context block example carries the same discipline in its header, labelling itself as measured from screen capture with no interpretation.

The README states the compile is under a second, costs zero tokens, and has no LLM in the loop. Treat those as the project's own claims rather than independent measurements, but the design consequence is real: the same capture input should produce the same frame output, which is what makes a replay view possible at all.

The compiled context block and its token arithmetic

The output an agent consumes most often is a compact text block. The README's example begins with a date, a local-time note, a coverage line (10:29 to 22:55, 529 active minutes, 15 apps), an away span, and then one line per frame with a time range, an app or app/site pair, a duration, and a short entity summary such as dashboard:video x59 or user:bethvourc x5.

The README states that this block, for the day shown, compiles to 1,371 tokens while the raw snapshot rows for the same day are 247,563. That is roughly two orders of magnitude of reduction, and the ratio is the whole argument for the compile step. The README also notes the excerpt shown is partial: the full block has 44 frames. A reader should not assume 1,371 tokens is a fixed budget. It scales with how many frames your day produces, and a day spent in many short bursts will produce more frames than a day in three long sessions.

One design choice here is deliberate and slightly awkward. The block is a flat list of time-ranged lines, not a hierarchy of projects or intentions. The README's own demo question, asking what article was open around 9pm, is answerable from a flat list because the URL is in the line. A question like which project was I neglecting this week is not answerable from this shape. The block reports activity, and it is honest about reporting only activity.

aframes record, aframes context, aframes steps

Installation is a single pip command, and the README gives it as pip install activity-frames. The quickstart then shows two commands: aframes record to start capturing, noted as local with audio off by default, and aframes context to print your last two hours in an agent-ready form.

The replay path uses a third command. The README shows aframes steps --find "message john doe", which returns JSON with a steps array, a step_count and an unresolved_clicks field. Each step carries a timestamp, an op such as focus, click or type, a human-readable target, and where relevant a role and a url. Typed runs are recorded with a character count and the text itself. The README describes this as the replay view of a demonstrated run, ordered clicks grounded by element name and role, with typed runs and focus changes.

For agent integration, the README points at docs/mcp.md for six MCP tools, docs/python.md for the Python API, docs/cli.md for every flag, and docs/troubleshooting.md for errors. AGENTS.md is written for an agent doing the integration. The README does not enumerate the six tools in the excerpt available here, so the tool names and their schemas are something you would read from docs/mcp.md rather than guess at.

Replay is bounded, and the boundary is the interesting part

The README is explicit that replay happens on the happy path at zero model calls, and that anything unexpected halts and asks instead of guessing. That is the correct default for a system that clicks buttons on your behalf, and it also defines the failure mode you are buying. A compiled workflow is a recording of one specific path through one specific interface. If the target site renames a button, reorders a form, inserts a consent dialog or serves a different layout, the recorded step no longer matches and the run stops.

The unresolved_clicks field in the steps JSON is the visible surface of that risk. The README's example shows it as 0 for a clean LinkedIn run, which tells you the field exists because it is sometimes not 0. A click that cannot be grounded to an element name and role is a step an agent cannot faithfully repeat. There is no stated retry policy, no stated fallback to model-driven recovery, and no stated handling for multi-factor prompts or captchas in the material available here. If your workflows routinely cross those, this is the wrong tool for them.

The second limitation is environmental. Capture depends on screen recording permission at the OS level, and the README does not describe a headless or server-side mode. A CI box or a container without a display is outside what is described. The third is representational: frames are built from app, site, URL and accessibility-style element name and role. Canvas-heavy applications, games, remote desktop sessions and anything that draws its own UI will produce thin or empty element data, and thin data means frames with little to compile and steps with poor grounding.

Where it sits against conversation-memory tooling

The obvious comparison is with the memory layers that attach to chat: vector stores over past conversations, or memory servers that expose remember and recall style tools over MCP. Those systems store what was said. They answer questions like what did we decide about the schema, and they index text you or the model produced.

activity-frames stores what was on screen, compiled into time-bounded records with input counts and element-grounded steps. It answers what did I actually do, and it can hand an agent a sequence of operations to repeat. The difference in approach is not the storage backend, it is the source of truth. A conversation memory system is only as complete as the conversation, and long stretches of real work happen in windows nobody narrates. A screen-capture compiler sees those stretches but cannot see intent, so it reports a LinkedIn people search with two results, not a recruiting push.

The practical consequence is that the two are complements rather than substitutes, and the README's framing of conversation memory as the other half supports that reading. If you are choosing one, choose based on whether your agent's blind spot is what you said or what you did.

Licence, releases and what maintenance looks like

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence and it does not impose copyleft obligations on your own code. It also offers no patent grant language, which is typical of MIT and worth knowing if patent exposure is a concern for your organisation. None of this is legal advice; read the LICENSE file and your own counsel's view.

On cadence, the supplied material shows two releases in the recent list: v0.2.0 labelled communications view in July 2026 and v0.2.2 later that month, with the last push to the default branch in late August 2026. The version numbers are still in the 0.2 range, which is the honest signal here: the public interface, including the six MCP tools and the CLI flags, can still move between minor versions. Anything you build against the JSON shape of aframes steps output should be pinned to a version and re-checked at upgrade time rather than assumed stable.

The README also references a research directory containing an instrument, measurements and an executor for what it calls the Routine Overhead Ratio, plus an arXiv identifier and a HackerNoon write-up. Those are the project's own published artefacts. They are not independent benchmarks, and the numbers in them should be read as the authors' measurements on their own activity and on a public web-task dataset, which is what the README says they are.

Editorial conclusion

Adopt activity-frames if you already run a computer-use or MCP-connected agent and you want a deterministic, LLM-free record of what actually happened on screen, plus a replay view for tasks you repeat. Do not adopt it if you need capture to work without a permissive OS-level screen recording grant, or if your work is dominated by canvas apps, games and remote desktops where accessibility labels are thin: the frame model is built on element name, role and URL, and the README's own example shows unresolved_clicks as a field precisely because clicks can fail to resolve. Before committing, run aframes context and aframes steps --find on your own week, check the unresolved_clicks count in the returned JSON, and confirm the Python version range on PyPI matches what your agent host runs.

Official sources

1. Issues
2. License: MIT
3. nossa-y/activity-frames on GitHub
4. README
5. Releases