SOLO reorders rows so a decision model can reuse cached prefixes

The whole idea is that a one-token answer is not a cheap request

SOLO runs a natural-language decision over every record in a dataset. It arranges rows and fields so a prefix-caching backend can reuse more input computation, then returns the decisions in your original row order. The backend receives the complete record on every request.

That last constraint is what keeps the approach honest. SOLO is not summarising, truncating or dropping fields to make requests smaller. Every request still carries the full record, so the model sees the same information and the answers are the same kind of answer.

The documentation explains why this matters with a timing observation on JEV-9B. A decision model may return only one label, but it must first process the full contract, policy, user context or candidate. Median request latency rose from 0.280 seconds at roughly 1K input tokens to 0.897 seconds at 6.5K, even though every request produced exactly one token.

So the cost is in the prefill, not the decode, and prefill is exactly what prefix caching attacks. The optimisation is a scheduling trick: if records that share a long prefix are sent close together in time, the server can reuse the computed KV for that prefix instead of recomputing it.

The library name is SOLO, System One Layout Optimizer, and the tagline is put repeated context first. Make your decision model do less work.

Installing the client and running your first scan

The client installs from a source checkout with Python 3.10 or newer:

python -m pip install ".[pandas,hub]"

That gives you the client library. For inference you connect to a running JEV-9B service, or deploy one on your own GPU.

The API is two calls. Build a DataFrame of records, open a DecisionEngine, and scan it with your question:

import pandas as pd
from solo_layout import DecisionEngine

tickets = pd.DataFrame([
    {"id": "T-104", "policy": "Refunds within 30 days", "request": "Please refund my order."},
    {"id": "T-201", "policy": "Refunds within 30 days", "request": "Can I exchange the size?"},
])
with DecisionEngine() as engine:
    result = engine.scan(tickets, "Does the request ask for a monetary refund?")
print(result.to_pandas())  # Decisions + probabilities, aligned with tickets.index

The result converts back to a DataFrame and is aligned with the original index, which is the point of the design: you reorder internally, but your caller never sees reordered rows.

A useful detail is that the client downloads only the pinned model's two small decision-head metadata files on first inference. Model weights, vLLM and Torch stay on the server. You can point `model_dir` at a local path if you want to use metadata from disk instead.

There is also a way to see the layout optimiser work with no GPU and no network at all:

python examples/offline_layout.py

JSON, JSONL, DataFrames and arrays all go through one engine

The input contract is broad. JSON text, JSON and JSONL files, nested objects, DataFrames and NumPy arrays share the same engine, and the scan call takes a path just as happily as it takes a frame:

from pathlib import Path
from solo_layout import DecisionEngine

with DecisionEngine() as engine:
    result = engine.scan(Path("tickets.jsonl"), "Does this customer request a refund?")

The packaging reflects that range. The base dependencies are numpy, numba and requests, and pandas is an optional extra rather than a hard requirement. The extras are pandas, hub for huggingface-hub, demo which adds pandas and tokenizers, plot for matplotlib, and dev for pytest, pandas and build.

The project is published as `solo-layout` at version 0.2.0, MIT licensed, with authors Yingze Li and Dong Wang. The classifier calls it Development Status 3 - Alpha, and the version number is consistent with that, so it should be treated as an early library.

The intended workloads are structured rather than free text. Repeated policies, account context, catalog attributes and correlated fields are the cases the design targets, and the included backend serves AutoTrust JEV-9B, an open decision model built on Qwen3.5-9B, through vLLM, returning binary, categorical or scored decisions with probabilities. The layout optimiser itself is not tied to that backend, and the documentation notes it works independently with your own.

The benchmark that shows the effect is the one to read closely

The reported headline is up to 11.7x throughput over original order, and 3.28x over NDV sorting, measured on one RTX 4090.

The first demo is the cleaner case, and the documentation is unusually candid about what it is testing. Many records share policies or account context, but an ID-first serialization breaks the reusable prefix. The demo varies the length of the repeated fields deliberately, and the reuse percentage describes field-value tokens rather than the server's cache-hit rate.

| Reusable field-value tokens | Max prompt tokens | Original rows/s | SOLO rows/s | Speedup | | 80.0% | 1,401 | 5.99 | 8.94 | 1.49x | | 97.0% | 8,881 | 0.84 | 6.68 | 7.90x | | 98.3% | 15,611 | 0.47 | 5.45 | 11.71x |

The pattern is that the gain grows with context length and with reuse. At short prompts the speedup is modest, because there is little to reuse. In the longest-context case the same 128 records finish in 23.5 seconds instead of 274.9 seconds, and all measured layouts reach 100% accuracy on that synthetic refund task.

The second demo targets a different failure mode. All 14 fields have the same NDV of 8 and the same length of 128 tokens, and twelve of them encode correlated information. Sorting columns by marginal NDV leaves their original order unchanged, because every column looks equally diverse. SOLO groups fields using their joint prefix structure instead, reaching 13.64 rows per second and 77.8% cached input, against 4.15 rows per second and 0.0% cached for the NDV layout.

Both are controlled synthetic workloads with the model, complete input records and concurrency held constant, each method starting in a fresh cache namespace, with reported time including planning, serialization, HTTP and the first cache fills.

Where the time actually goes, and what the accuracy numbers mean

The profiling section is where the mechanism is confirmed rather than asserted.

On the ContractNLI run, the median interval from scheduling to the first decision token fell from 726.1 ms to 219.3 ms, a 3.31x reduction. Every request emits one decision token, so the subsequent decode interval was 0.0 ms in all 2,176 original and SOLO request measurements. In other words, the entire saving is in prefill and scheduling, which is exactly the theory.

The cache accounting is reported in the same way. With APC enabled, 85.19% of SOLO prompt tokens had an earlier exact prefix, 73.89% remained reusable after 528-token block quantization, and vLLM reported 72.48% actually cached, which is 98.1% of the block-level ceiling.

The accuracy figures need more care than the throughput ones, and this is where the caveat belongs. Across all 1,088 decisions from 64 selected ContractNLI contracts, original-order accuracy was 66.18% and SOLO reached 73.90%. Completing the same 17 batches of 64 took 293.64 seconds with the original layout and 115.73 seconds with SOLO, a 2.54x speedup.

Those numbers are better than the original, which is a good sign for the method, but they also mean a change in behaviour. SOLO keeps every field and leaves the model unchanged, so the difference comes from the order of records and fields changing the computation. A library that reorders your inputs and returns a different accuracy on the same task is doing more than scheduling, and the documentation does not explain why accuracy moves at all. That is worth investigating on your own data before you rely on the result.

The precondition that decides whether SOLO helps you at all

The single most important sentence in the documentation is that the speedup disappears when APC is disabled.

Automatic prefix caching is what the entire approach depends on. SOLO's job is to arrange records so that a request shares a long prefix with a recent request in the cache, and if the server does not cache prefixes, ordering the requests buys nothing but the cost of the reordering itself.

The project also documents a real problem in the upstream process. It notes that recently newer changes were appearing in the upstream package without the library version being changed, and that this created confusion and difficulty when it came to maintenance updates. The upstream developers have acknowledged the issue and are working on making the process more transparent.

Two operational notes follow from the design. The reordering is not free: planning and serialization are counted in the reported times, so on very short records SOLO may lose to doing nothing. And the requests are reordered even though the results are realigned, which matters if you have anything downstream that depends on the order of calls, such as a rate-limited downstream service, an audit log, or a model that carries state between requests.

The repository is built for reproducibility of exactly these claims. It carries a docs/profiling.md with the full protocol and cache controls, a validation/ directory with raw trial JSON including shared-live.json and the two summary aggregates, assets/profiling/ with prefill and decode charts, a planner-provenance.json, a demo.py, a deploy/ directory, and a CITATION.cff for citing the work.

Editorial conclusion

Adopt SOLO if you are running per-record decisions over a large relational table where many records share policy or account context, because the library keeps every field intact and only changes the order, so the model and the answers stay the same. Skip it if your backend has no prefix caching enabled, because the documentation is explicit that the speedup disappears when APC is off, or if you need a per-row audit trail that depends on request order, since the requests are reordered even though the results are realigned. Verify first that your inference server supports automatic prefix caching, that your records contain genuinely repeated long fields, and that the accuracy change on your own data is acceptable, since the reported ContractNLI figures come from one benchmark with a specific layout.

Frequently asked questions

Official sources

1. 0814wdwd/solo_jev on GitHub
2. Issues
3. License: MIT
4. README