# MACE's two-stage erasure: cross-attention refinement, then a LoRA per concept

> MACE, the official code for a CVPR 2024 paper, strips unwanted concepts out of a text-to-image diffusion model in two passes: a closed-form rewrite of the cross-attention projections, then one LoRA per concept. The bill comes due at mask preparation and at the LLM that writes augmentation captions.

**Shilin-LU/MACE** — [CVPR 2024] "MACE: Mass Concept Erasure in Diffusion Models" (Official Implementation)

- Repository: https://github.com/Shilin-LU/MACE
- Stars: 391 · Forks: 32
- Language: Jupyter Notebook
- License: NOASSERTION
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/shilin-lu-mace

## Closed-form cross-attention refinement, then one LoRA per concept

MACE touches two things inside a pretrained text-to-image U-Net. The first is the prompt-related projection matrices inside the cross-attention blocks, and those are refined with a closed-form solution rather than a finetuning loop. The stated purpose is to discourage the model from embedding residual information about the target phrase into the words around it in the prompt, a narrower job than retraining the whole attention stack. The second is per-concept LoRA: every concept on the erasure list gets its own low-rank module, trained to eliminate that concept's intrinsic information. Nothing else about the checkpoint is asked to change. Two stages, two shapes of edit, one entry script called `training.py` at the root and a `src/` directory holding the model code it calls into. Which stage a problem belongs to matters when something misbehaves. Residual concept leaking into a neighbouring word points at the cross-attention stage; a concept that still renders on a direct request points at the LoRA.

## The authors point at a different lever in their next paper

The README doubles as a pointer to the authors' next idea. Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts (arXiv 2504.12782) steers the denoising trajectory away from unwanted concepts rather than editing weights, and its code lives in a separate repository at github.com/lileyang1210/ANT. Same group, same target, opposite approach. MACE removes information already baked into a checkpoint, which is what you need when the only artifact you control is the weights file and nothing about how inference runs can change. Steering acts on the run instead of the file, leaves the checkpoint alone and pays for that at sampling time. The other comparison runs sideways rather than forward. The abstract states that existing concept erasure methods are typically restricted to fewer than five concepts at once and struggle to balance generality, meaning killing synonyms of the target, against specificity, meaning keeping unrelated concepts intact. MACE claims both at 100 concepts. Whether that balance holds for your own list is something you settle by generating images, not by reading the abstract.

## The conda recipe pins PyTorch 2.0.1 against CUDA 11.7

The environment is pinned hard, which helps reproducibility and hurts anyone on a newer stack. Python 3.10, PyTorch 2.0.1, torchvision 0.15.2, CUDA 11.7, all four fixed in the project's own conda lines rather than in a lockfile you can read at the top level.

```bash
git clone https://github.com/Shilin-LU/MACE.git
conda create -n mace python=3.10
conda activate mace
conda install pytorch==2.0.1 torchvision==0.15.2 pytorch-cuda=11.7 -c pytorch -c nvidia
```

Read the diffusion libraries as a deliberate combination rather than a frozen lockfile. `diffusers` is held at 0.22.0 while `transformers` sits at 4.46.2 and `huggingface_hub` at 0.25.2, so no single release covers the set. If a newer `diffusers` is already installed elsewhere on your machine, expect that pin to be the thing that fights you.

```bash
pip install diffusers==0.22.0 transformers==4.46.2 huggingface_hub==0.25.2
pip install accelerate openai omegaconf opencv-python
```

That second line is where the LLM client and `omegaconf` arrive, and both matter in the stages that follow.

## Grounded-SAM masks cost a 24GB GPU, or more than 28GB

LoRA tuning needs masks that say where the concept sits in each training image, and Grounded-SAM supplies them. Two routes are documented and both carry a price: the official Grounded-Segment-Anything build runs on a 24GB GPU, while the HuggingFace version likely requires above 28GB. The project's own advice is to skip the official build if 24GB is what you have, which is an unusual thing to find in an installation section. Two preparation scripts sit at the root, `data_preparation.py` and `data_preparation_transformers.py`, one for each route.

```bash
export AM_I_DOCKER=False
export BUILD_WITH_CUDA=True
# export CUDA_HOME=/path/to/cuda-11.7/

cd MACE
git clone https://github.com/IDEA-Research/Grounded-Segment-Anything.git
cd Grounded-Segment-Anything
```

The commented `CUDA_HOME` line ties this toolchain to the same CUDA 11.7 the PyTorch pin assumes, so a driver mismatch surfaces here first. Two checkpoints come down next: GroundingDINO's `groundingdino_swint_ogc.pth` from the v0.1.0-alpha release, and SAM's `sam_hq_vit_h.pth` from a HuggingFace repository. Treat the mask step as annotation rather than scaffolding. The LoRA stage learns against whatever regions these masks define, so a mask that clips the object wrong produces a LoRA that erased too much or too little, and the finetuning takes the blame.

## Augmentation captions come from an LLM you have to bring

Erasure needs captions, and MACE does not write them itself. An LLM generates text augmentations for the concepts being erased, and the default backend is OpenAI's `gpt-3.5-turbo`. Four environment variables control the choice. `LLM_PROVIDER` switches between `openai` and `minimax`, `OPENAI_API_KEY` and `MINIMAX_API_KEY` carry the credential for whichever you pick, and `MINIMAX_MODEL` selects a model, defaulting to `MiniMax-M2.7`, with `MiniMax-M2.7-highspeed`, `MiniMax-M2.5` and `MiniMax-M2.5-highspeed` offered as the alternatives. Swapping backends needs no code change, which is the entire reason the choice sits in the environment.

```bash
export LLM_PROVIDER=minimax
export MINIMAX_API_KEY=your_api
```

This is the one documented stage that calls a third-party API during training. No offline backend appears in the README, and no caching of generated captions is described, so a training run needs working network access plus a key you supply before it starts. Erase a hundred concepts and you are asking that endpoint for a hundred sets of paraphrases, which is the practical reason the provider switch matters on a machine that cannot reach OpenAI.

## training.py, inference.py and prompts_csv/ map the shape of a run

The repository root is a usable map of what a run touches. `training.py` and `inference.py` are the entry points, `configs/` holds the run configuration that `omegaconf` reads, `prompts_csv/` holds prompt lists for evaluation, and `metrics/` is where scoring lives. `demo.ipynb` is the only notebook, which is why the repository reports Jupyter Notebook as its primary language even though the pipeline is Python. `lean_proofs/`, `tests/` and `assets/` fill out the rest. Two entries in the table of contents shape the decision to retrain: sampling from the modified model is a separate step from training, and MACE Finetuned Model Weights is a separate section again, which reads as an offered starting point rather than a footnote. Nothing forces a finetuning run before you can generate images. One filename needs a second look, `transformer_gsam_utils.py`, named for the GroundingDINO and SAM work rather than for the diffusion model. Two model families, from two unrelated codebases, are both required before the first training step.

## The README stops mid-command before the training recipe

Setup, mask preparation, dependencies and the LLM provider section are complete enough to follow. Two blocks are not. The Grounded-SAM build ends on a line that stops at `pip inst`, and the provider block ends right after `export MINIMAX_API_KEY=your_api`. Everything the table of contents lists after that point exists only as a heading: Training MACE to Erase Concepts, Sampling from the Modified Model, MACE Finetuned Model Weights, Metrics Evaluation. You get the root file names to infer from, plus `configs/` to read, but no invocation, no flags, no hyperparameters, no metric definitions. The project page and poster links near the top of the README sit inside HTML comments, so the paper's own supplementary material is not linked from here either. Plan around that gap rather than through it: budget time to read `configs/` and the paper before starting, because the README alone will not get you to a running trainer.

## No releases, and the license field is unasserted

Two facts about upkeep matter before anything here goes into a product. The repository is not archived, and its last push landed on 2026-07-20 on the default branch `main`. It has no GitHub releases, so there is no version to pin, no tag to track and no changelog to read. What you install is whatever `main` holds when you clone it, which means the compatibility question comes back to those hand-written conda pins every single time. A LICENSE file does sit at the root, but the repository's license field reads NOASSERTION rather than a named license. Open that file and read it yourself before shipping anything derived from this code, because nothing in the repository metadata tells you which terms apply. None of this says the work is finished. A paper's reference implementation with a recent commit and no packaging is an ordinary shape for research code. It does mean you own the upgrade path for every dependency in the list.

## Conclusion

Researchers who want the paper's results should start from the published MACE finetuned model weights rather than a training run, because the README stops before it prints the training command. Anyone planning to erase a hundred concepts on their own checkpoint should measure the mask step against their GPU first: 24GB for the official Grounded-SAM build, more than 28GB for the HuggingFace one. Teams blocking a handful of celebrity names are paying for machinery built to scale past five.

## FAQ

### What does MACE erase, and how many concepts at once?

MACE is a finetuning framework for mass concept erasure in text-to-image diffusion models, aimed at stopping a model from generating images of concepts you do not want. The paper scales the erasure scope to 100 concepts, where earlier concept erasure methods are typically restricted to fewer than five at a time.

### Does MACE need an OpenAI API key to train?

Text augmentations for the erased concepts are generated by an LLM, and the default backend is OpenAI's gpt-3.5-turbo with OPENAI_API_KEY as its variable. Setting LLM_PROVIDER to minimax with MINIMAX_API_KEY and MINIMAX_MODEL switches backends through environment variables alone, with no code changes.

### How much GPU memory does MACE need before training?

Mask preparation is the expensive part. The official Grounded-SAM build for masks can run on a 24GB GPU, while the HuggingFace version likely requires above 28GB, and the README suggests skipping the official build if that is all you have.

### Is MACE maintained, and can I pin a version?

The repository is not archived and its last push was on 2026-07-20, on the default branch main. It has no GitHub releases, so there is no release to pin and no changelog, and the LICENSE file needs reading directly because the repository's license field reads NOASSERTION.

## Sources

- [Issues](https://github.com/Shilin-LU/MACE/issues)
- [README](https://github.com/Shilin-LU/MACE/blob/main/README.md)
- [Shilin-LU/MACE on GitHub](https://github.com/Shilin-LU/MACE)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/shilin-lu-mace