BrickGPT: text to stable brick models, and the Gurobi dependency behind it

The gap BrickGPT fills: text prompts that already know how to stand up

Most text-to-3D systems produce a shape and stop. Whether the shape can be built, whether a brick floats in mid-air, and whether the stack survives its own weight are left to whoever tries to assemble it. BrickGPT targets that second problem. The README describes it as "the first approach for generating physically stable toy brick models from text prompts", and the repository is the official code for the ICCV 2025 paper that won the Marr Prize. The audience is narrow on purpose: researchers working on structured 3D generation, and engineers who want a brick-by-brick output rather than a mesh. The output format makes that clear. You get an LDraw file, a text file listing each brick, and a rendered image, not a triangle soup.

How the generation loop actually works

The model is a fine-tune of meta-llama/Llama-3.2-1B-Instruct, so generation is autoregressive over a sequence of bricks rather than over pixels or voxels. The example interaction in the README shows what one run produces: a prompt about a table with a flat rectangular surface over four legs, 59 bricks placed, 98 brick rejections, and 4 regenerations before the run finished. That rejection count is the mechanism. Candidate bricks are proposed, checked, and thrown away when they collide with existing geometry or when they were already rejected earlier in the same run. The rejection reasons printed by the script are grouped by cause, and in the documented example the dominant reason is already_rejected rather than collision, which tells you the sampler revisits the same bad placements repeatedly instead of exploring new ones. Stability is the other half of the loop. With Gurobi enabled, the code performs physics-based stability analysis; the README states that without it you can pass --use_gurobi False to fall back on "a simpler but less accurate connectivity-based method". Connectivity asks whether bricks touch. Stability asks whether the structure holds. Those are different questions, and the project is honest that the cheap one is worse.

Getting it running: uv, a gated model, and an LDraw library in your home directory

The repository uses uv as its project manager. Clone it, then run uv sync to build a virtual environment from the dependencies in pyproject.toml. Two prerequisites come before that. First, Llama-3.2-1B-Instruct is a gated model, so you request access on Hugging Face, generate a user access token, and export it as HF_TOKEN; the weights download automatically when you run the code. Second, rendering requires ImportLDraw: run git submodule update --init, download a background exr file into the ImportLDraw/loadldraw subdirectory, and fetch the LDraw parts library with wget from library.ldraw.org, extracting it into your home directory. If you want the parts somewhere else, set LDRAW_LIBRARY_PATH to the ldraw directory. Inference is then a single command, uv run infer, which starts an interactive session asking for a prompt, an output filename, and a generation seed. The seed defaults to 42. To install the code into an existing project instead, the README gives uv add "https://github.com/AvaLovelace1/BrickGPT.git" or the pip equivalent. Note the asymmetry: the ImportLDraw step is marked optional and is only needed for the infer script and texturing, so a headless pipeline that just wants output.txt and output.ldr can skip the LDraw download entirely.

Gurobi is the real dependency, and the fallback is a downgrade

The README calls Gurobi "optional but recommended", which undersells the situation. Stability analysis is the project's central claim, and the licence situation is restrictive. The example output shows Gurobi printing "Academic license - for non-commercial use only", with an expiry date. Academics can request a free licence; everyone else needs a commercial one, and the README points to a support article about where to place the licence file, typically the home directory. If you cannot get a licence, --use_gurobi False keeps the code running, but you are no longer evaluating the thing the paper is about. That is the honest trade-off, and it is the first thing to settle before you spend an afternoon on setup. A second, quieter cost sits in the example run itself: 63.53 seconds for a single 59-brick structure, with 4 regenerations. Interactive use through the Gradio demo is plausible; batch generation of hundreds of prompts is a different budget question.

Where the approach runs out: rejection loops, fixed vocabulary, and the texturing edge

The rejection statistics are the most informative number in the README. Of 98 rejections in the documented run, 93 were already_rejected. The sampler is spending most of its rejections on placements it has already turned down, which suggests the search does not strongly avoid dead ends within a single generation. More regenerations means more wall-clock time, and the count is not bounded in any way the README describes. The brick vocabulary is also fixed. Output is written as lines of the form hxw (x,y,z), so the model emits standard brick footprints at integer grid positions. Prompts that imply curved pieces, hinges, or non-grid geometry have no representation in that format. And the texturing path is only partly documented here: the README's final section announces that src/texture contains code for UV textures or per-brick colour, and the supplied text cuts off mid-sentence. Treat texturing as unverified until you read that directory yourself. The README does not state how per-brick colour is stored in the LDraw output, or whether colour survives a round trip.

The alternative: a general text-to-3D generator plus your own buildability check

The obvious alternative is to use a general text-to-3D system and post-process the mesh into bricks yourself. The difference in approach is where the constraint lives. A general generator optimises for appearance and produces geometry with no notion of a discrete part inventory, so any brick decomposition you add afterwards is a separate solver on top of a result that was never asked to be buildable. BrickGPT constrains during generation: the model proposes bricks from a fixed vocabulary and the stability check gates each one. That is a meaningfully different architecture, and it is why the rejection counts exist at all. The cost is scope. You cannot ask BrickGPT for a shape outside its brick vocabulary, and you cannot swap the stability backend for a different physics engine without changing the code. If your goal is a pretty render, a general generator is the shorter path. If your goal is a structure someone can actually assemble, the constraint has to be inside the loop.

Maintenance, licence, and what the repository does not tell you

The project is MIT licensed, which covers the code in this repository. It does not cover the model weights, which are distributed separately on Hugging Face, and it does not cover Llama-3.2-1B-Instruct, which is a gated Meta model with its own terms and its own access request. Gurobi is commercial software with its own licence, academic or otherwise. Three licences stacked on one pipeline is normal for research code but worth mapping before you ship anything. On maintenance: there are no retrieved releases, the default branch is main, and the last push recorded is 2026-05-21. That is a single-commit-stream research repository, not a versioned library, so pin a commit hash if you build on it. The README also does not document training or fine-tuning, only inference and texturing, so reproducing the model from the released StableText2Brick dataset is not a path the repository lays out. Do not assume it does.

Editorial conclusion

Adopt BrickGPT if you have a Gurobi licence and want to study or extend text-to-structure generation with a physics check in the loop. Do not adopt it if you need a hosted service, a fast interactive toy, or a permissively licensed replacement for Gurobi. Before you commit, verify two things: that your Hugging Face token has been granted access to the gated meta-llama/Llama-3.2-1B-Instruct weights, and that a Gurobi academic licence is actually available to you, because the fallback connectivity check is documented as simpler and less accurate.

Frequently asked questions

Official sources

1. AvaLovelace1/BrickGPT on GitHub
2. Issues
3. License: MIT
4. Project website
5. README