MACE: Erasing Up to 100 Concepts from a Stable Diffusion U-Net

The Problem MACE Targets: Erasure at List Scale

Concept erasure asks a diffusion model to stop producing a given subject when prompted. The MACE abstract states that prior methods are typically restricted to handling fewer than five concepts simultaneously, and that they struggle to balance erasing concept synonyms against preserving unrelated concepts. Those two failure directions are named in the paper as generality and specificity. MACE's claim is that it scales the erasure scope up to 100 concepts while holding that balance. The intended user is therefore not someone who wants one celebrity removed from a checkpoint. It is a team with a long, enumerated blocklist: object categories, named people, explicit content, or artistic styles, evaluated across the four task families the abstract lists. If your blocklist is three entries long, the machinery here is heavier than the job.

Two Mechanisms: Cross-Attention Refinement and Per-Concept LoRA

The framework figure in the README breaks the method into four labelled parts. Part (a) says the framework tunes the prompt-related projection matrices inside cross-attention blocks rather than the whole U-Net. Part (b) refines the pretrained U-Net's cross-attention blocks with a closed-form solution, which the caption describes as discouraging the model from embedding residual information of the target phrase into surrounding words. That residual leakage is the mechanism behind synonym failure: erase the exact token and the model routes around it through neighbouring context. Part (c) learns a distinct LoRA module per concept to remove that concept's intrinsic information. Part (d) applies a second closed-form solution to merge multiple LoRA modules without mutual interference, and the caption adds that this avoids catastrophic forgetting. The two closed-form steps are the interesting part of the design. Merging LoRAs by ordinary weighted addition is where multi-concept erasure usually degrades, because separately trained adapters pull the same weights in conflicting directions. MACE's answer is an algebraic merge rather than a hyperparameter search over merge weights.

Data Preparation Is the Real Cost, Not Training

Before any LoRA is trained, MACE needs masks, and the README is explicit that this is where the work sits. It offers two Grounded-SAM routes. The official version is described as complex to install and able to run on a 24GB GPU. The HuggingFace version is presented as the easier path but the README states it likely requires above 28GB. That single sentence is the most useful capacity planning fact in the repository: the convenient route is the more memory-hungry one. The official build involves cloning Grounded-Segment-Anything, installing segment_anything and GroundingDINO as editable packages, initialising the grounded-sam-osx submodule and running its install script, then cloning recognize-anything for RAM and Tag2Text. Two weight files are fetched with wget: groundingdino_swint_ogc.pth from the GroundingDINO releases and sam_hq_vit_h.pth from the lkeab/hq-sam repository. The README also offers a Download Pre-cached Files section, which is the sane choice if you are evaluating MACE rather than reproducing the mask pipeline, though the README does not state what those cached files cover.

Getting It Running: Environment and LLM Provider

The environment is pinned and narrow. The README gives conda create -n mace python=3.10, then conda install pytorch==2.0.1 torchvision==0.15.2 pytorch-cuda=11.7 from the pytorch and nvidia channels. Application dependencies are pinned at diffusers==0.22.0, transformers==4.46.2 and huggingface_hub==0.25.2, with accelerate, openai, omegaconf and opencv-python installed unpinned. The gap between diffusers 0.22.0 and current releases is wide, so expect to work inside this environment rather than against it. Text augmentation for the erased concepts is generated by an LLM. The README states the default is OpenAI gpt-3.5-turbo, and that you can switch to MiniMax without code changes by setting LLM_PROVIDER to openai or minimax, MINIMAX_API_KEY to your key, and MINIMAX_MODEL to a model such as MiniMax-M2.7. Note the default: if you run the training script without setting these, caption generation goes to a third-party API. Air-gapped or privacy-sensitive deployments need to confirm whether the pipeline can run with pre-generated captions instead.

Where MACE Is the Wrong Tool

The mask dependency is a real constraint, not a footnote. MACE's LoRA stage is conditioned on segmentation masks produced by Grounded-SAM, so the erasure quality inherits the detector's behaviour on your target concepts. Abstract concepts, stylistic qualities and anything a grounding detector cannot localise cleanly are awkward inputs for this pipeline, even though the paper lists artistic style erasure among its evaluated tasks. The setup also assumes a single-GPU workflow with a specific CUDA build. There is no released package, no retrieved releases in the repository metadata, and no homepage. The primary language is Jupyter Notebook, which means the runnable logic is spread across notebooks rather than a library API you can import into a service. If you need concept suppression inside a serving path, MACE produces modified weights you then deploy; it is not a runtime filter. Finally, the repository is marked NOASSERTION, so the licence terms are not machine-readable from the metadata alone and must be read from the repository itself.

The Alternative: Steering the Denoising Trajectory Instead of Finetuning

The same group's later work, ANT, is referenced in the README with its own arXiv entry and code repository. The difference in approach is structural. MACE is a finetuning framework: it modifies the model, producing new weights that carry the erasure permanently, with per-concept LoRA modules merged by closed-form solution. ANT, described as auto-steering denoising trajectories to sidestep unwanted concepts, operates on the sampling trajectory rather than rewriting the checkpoint. That matters for operations. A finetuned checkpoint is a new artifact to version, distribute and re-validate; a trajectory-level intervention leaves the base weights untouched and is applied at inference. The trade-off runs the other way too: MACE's erasure is baked in and costs nothing extra per generation, while a trajectory method adds work to every sampling run and depends on the intervention being active. If your constraint is a fixed model artifact shipped to users, MACE's approach fits. If your constraint is keeping one canonical checkpoint and adjusting policy per deployment, the trajectory approach is the more natural shape.

Maintenance, Pins and Licence Status

The dependency pins are the main maintenance liability. diffusers 0.22.0 and transformers 4.46.2 are fixed versions, and the README pins huggingface_hub==0.25.2 as well, which suggests the code depends on interfaces that later releases changed. Upgrading any of the three is an untested change, and the repository ships no release artifacts to anchor a known-good combination, so the README's pip line is the only supported configuration. The Grounded-SAM install is the second liability: it pulls three external repositories, one of which is installed through a shell script in a submodule, and those upstream projects move independently of MACE. On licensing, the repository metadata reports NOASSERTION, which means the licence could not be identified automatically. That is not the same as having no licence, and it is not the same as permissive. Anyone intending to ship a MACE-modified checkpoint should read the repository's own licence file and check the terms of the Grounded-SAM components separately, since those carry their own licences. This is a description of the metadata, not legal advice.

Editorial conclusion

Adopt MACE if your task is suppressing a large, enumerated list of concepts in a Stable Diffusion checkpoint and you can afford a Grounded-SAM mask pipeline plus per-concept LoRA training. Do not adopt it if you need one concept removed in an afternoon, if you cannot run the official Grounded-SAM build on a 24GB card, or if you need a clearly stated licence before shipping. Verify the licence file and the LLM provider configuration first, because the repository is marked NOASSERTION and the default text augmentation path calls OpenAI.

Official sources

1. Issues
2. README
3. Shilin-LU/MACE on GitHub