OrcaBonsai 27B Uncensored: ablating refusals at runtime instead of editing weights

The problem with ablating a ternary model

The project is described as runtime behavioral ablation for compressed LLMs, with the first target being the Ternary Bonsai 2 27B pack from prism-ml, and the pitch in one line: keep the weights compressed, change the behavior at inference.

The obstacle is specific and interesting. Traditional abliteration modifies model weights by orthogonalizing matrices that write into the residual stream against a learned refusal direction, and the formula given is:

W ← W - r(rᵀW)

For a conventional BF16 or FP16 model, the resulting matrix can simply be saved as a new checkpoint, so the edit becomes permanent and the model is just a different model.

Ternary Bonsai 2 is different because its weights are aggressively quantized. The affine container stores a scale and a bias:

scale = s
bias  = -s

so the 2-bit codes decode to a three-valued set:

{0, 1, 2}

becoming

{-s, 0, +s}

The model's quality at approximately 1.72 bits per weight is the result of quantization-aware training, and that is the problem. Orthogonalizing one of these matrices produces a dense, full-precision matrix, and saving that back into the ternary representation would require re-quantization. And simply re-quantizing the edited weights does not reproduce the quantization-aware training process that produced the original model.

So the decision follows from the arithmetic: do not edit the weights. The repository's summary of its own claims is 27B parameters, about 1.72 bits per weight, 0 modified weights, 0 re-quantization, 0 additional weight quantization error, runtime-adjustable ablation strength, 129 residual intervention sites, and Apple Silicon with MLX.

The projection moves from the weight to the residual write

Instead of editing the matrix once, the equivalent projection is applied when each residual contribution is produced:

y ← y - α · dot(y, r) · r

where y is the residual contribution, r is the normalized refusal direction, and alpha is the intervention strength. At alpha = 1 the component of each residual write parallel to the refusal direction is removed.

The data flow is worth spelling out because it explains the whole design. The original Ternary Bonsai 2 27B pack with bit-identical weights goes into a packed matmul, that produces y, the runtime ablation block applies the correction to y, and the result continues into the residual stream and out to the output. The correction happens on the way, not on the weights.

Two implementation details are stated and both matter for anyone reproducing this. The projection uses an MLX-compiled correction with float32 accumulation, then restores the writer's output dtype. And the documentation is careful that compilation may change floating-point rounding, while stating that it does not change the projection formula or the ablation strength.

The claim that the original ternary weights are never modified is the whole point of the project, and it is also what makes the A and B comparison clean. Because the intervention happens entirely at runtime, the same model pack can be run with or without ablation, which the project calls straightforward A and B testing, since both configurations use the same underlying weights.

The alpha parameter is the dial. At alpha = 0 you get original model behaviour, at alpha = 1 the full refusal-direction projection, and the default is alpha = 1.

Running it, and how the pack is located

The quickstart is short, and the detail that trips people up is the pack path.

Install the dependencies:

pip install -r requirements.txt

and then run the model:

python run.py \
  --pack /path/to/Ternary-Bonsai-2-27B-mlx-2bit \
  "your prompt"

The --pack argument is the directory holding config.json and runtime/. The documentation is precise about which directory that is when you download with huggingface_hub: it is the snapshot directory, not the models--... cache entry above it. There is a one-liner for printing the right path:

python -c "from huggingface_hub import snapshot_download; \
print(snapshot_download('prism-ml/Ternary-Bonsai-2-27B-mlx-2bit'))"

A cache entry is accepted as well, in which case the newest snapshot is resolved for you.

The requirements file is worth reading because it explains the platform situation. The pins are the versions the pack's own runtime uses, and the comment notes that mlx-metal comes in automatically with mlx on Apple Silicon, while on x86 Linux you need to add mlx-cpu==0.32.0, because there the plain mlx wheel carries no libmlx.so and there is no CUDA kernel for the pack's quantized matmul. The pinned set is mlx 0.32.0, mlx-lm 0.31.3, mlx-vlm 0.6.3, transformers 5.5.0, numpy 2.0 or newer, tokenizers 0.21 or newer, jinja2 3.1 or newer and pillow 10.0 or newer.

So the project runs on Apple Silicon as the primary path, with an x86 Linux path that needs an extra package and loses the quantized matmul kernel.

The self-check is how you know the wiring is right

Because the intervention happens inside the residual writers, a broken setup does not crash, it just does nothing. The project ships a check for exactly that.

python scripts/selfcheck.py \
  --pack /path/to/Ternary-Bonsai-2-27B-mlx-2bit

The self-check measures how much of the residual stream lies along the refusal direction before and after the intervention. A correctly instrumented setup should drive the remaining component to approximately:

~1e-6 of the residual norm

If the number barely changes, the documentation says the residual writers were probably not wrapped correctly. That is the diagnostic value of the check: the failure mode is silence, and this is the number that tells you it happened.

The check also verifies that all expected intervention sites are present, and the expected number is:

129 residual writers

There is a second, model-free test for the compiled correction itself, run with python scripts/test_ablation.py, covering FP16 and FP32 outputs, changing directions, and alpha values of 0, 1 and 2. Separating the two is the right structure: one test says the numbers are right, the other says the patch is applied where it needs to be.

The evaluation is honest about its own method

The refusal evaluation is presented with a caveat that a reader should take seriously, and it is the most methodologically careful paragraph in the project.

Refusal is judged by a rule-based opening-phrase classifier, with a caveat value meaning the model answered but wrapped the answer in a disclaimer, and the documentation calls this indicative rather than an LLM judge or a publication-grade number. Thinking is off, decoding is greedy, and the token budget on the refusal sets is 64 tokens. The base and ablated rows are the same weights in the same process, alpha 0 against alpha 1.

The results are given for AdvBench, JailbreakBench, StrongREJECT, HarmBench, MaliciousInstruct, SimpleSafetyTests and ForbiddenQuestions, each with a sample size and the base rate, the ablated rate and the caveat rate. One methodological note is stated explicitly: no reply in any set ran out of its token budget, so none of these rates is inflated by truncation. That is the kind of check that is easy to omit and expensive to get wrong, since a truncated answer reads as a refusal.

There is a second table for the opposite risk, over-refusal on benign prompts, with XSTest-safe and JBB-benign. The base model turns down 5.2 percent of XSTest-safe prompts and 25.0 percent of JBB-benign ones, and the ablated model turns down 0.4 and 0.0 percent. So the projection does not only stop refusals on harmful prompts, it also stops the over-refusal behaviour that the published pack has.

One more implementation constraint is stated for the evaluation runs: they are executed on the unfolded fp16 expansion of the pack rather than the pack driving its own kernels, because the packed quantized matmul has no CUDA implementation and the CPU backend needs minutes per forward pass. The expansion is said to carry the pack's ternary values exactly, a container change rather than a requantization, and to reproduce the pack's own next-token distributions to three decimals on spot checks.

Where this repository is the wrong tool

The first limit is that the whole thing is a research rig, not a deployment artefact. The project itself says you supply the original Ternary Bonsai 2 pack and it supplies only the refusal direction and the runtime that applies it. There is no packaged model, no server, and no supported way to serve this. Anyone wanting a served model that behaves this way is looking at a different kind of project, and the alpha dial is a research control rather than a production setting.

The second is the platform. The primary path is Apple Silicon with MLX, and the x86 Linux path needs an extra package and has no quantized matmul kernel, so the CPU backend takes minutes per forward pass. The evaluation numbers were produced on the unfolded fp16 expansion for that reason. A deployment on x86 Linux is not what was measured.

The third is that the evaluation is narrow, and the project says so. A rule-based opening-phrase classifier on 50 to 250 sample prompts per set, with a 64 token budget and greedy decoding, is a reasonable smoke test and not a benchmark. The caveat column is the interesting part of the table: on AdvBench, 6.0 percent outright refusals becomes 56.0 percent answers wrapped in a disclaimer. Whether a disclaimer counts as refusal is exactly the judgement the method is not making for you, and any organisation that cares about the distinction needs a different evaluation than this one provides.

The fourth is the licence. The repository is classified as Other, a custom licence GitHub cannot classify, with the terms in the LICENSE file. For a model-weights-adjacent artefact that is worth reading rather than assuming, and the practical question for any organisation is what the terms permit for redistribution and for use in a product.

The fifth is scope of intent, and it is the one the project itself cannot answer for you. Removing refusal behaviour from a model is a research operation on a model you already have. Whether that is appropriate depends entirely on what the result is then used for and who is exposed to it, and nothing in the repository addresses that question. The same is true of the over-refusal reduction in the second table, which is a genuine improvement in behaviour and is also the removal of a behaviour some deployments rely on. The last push to the repository was on 2026-09-22, and there are no GitHub releases, so the code is a moving branch rather than a versioned distribution.

Editorial conclusion

OrcaBonsai 27B Uncensored is worth reading if you are researching refusal directions or want to see how a projection behaves on a quantized model, because the runtime approach preserves a bit-identical pack and gives an adjustable alpha you can compare against the untouched model in the same process, which removes most of the confounds a modified checkpoint introduces. It is the wrong tool for anything that needs a served, reproducible model artefact, and it is the wrong tool if your goal is to remove safety behaviour from a system other people will use. Before you rely on it, check the three things the project states: that the pack must come from the original source, since this repository supplies only the refusal direction and the runtime that applies it; that the self-check expects 129 residual writers and a remaining residual component of about 1e-6 of the residual norm, so a setup that barely changes the number is a setup that did not wrap the writers; and that the reported refusal rates are judged by a rule-based opening-phrase classifier rather than a language model judge, which the project itself calls indicative rather than publication-grade.

Frequently asked questions

Official sources

1. Continuum-AI-Corp/OrcaBonsai-27B-Uncensored on GitHub
2. Issues
3. Project website
4. README