Modly: local image-to-3D mesh generation on your own GPU

What problem Modly solves, and for whom

Image-to-3D is normally a cloud service. You upload a photo, a remote GPU runs a model, and you download a mesh. Modly inverts that: the README describes it as a desktop application that turns a photo into a 3D model "using open source AI models running entirely on your GPU." Nothing about the generation step is remote. That single design decision determines the audience.

The people this fits are the ones who cannot or will not send source imagery to a third party: hobbyist modelers working from reference photos, small studios under client NDAs, and engineers who want a repeatable local pipeline instead of metered API calls. It also fits anyone who already owns a capable GPU and would rather amortize it than rent one by the hour.

The repository describes Modly as a desktop application for Windows, Linux, and Apple Silicon macOS, so the target is a workstation with a discrete GPU, not a browser tab. The topics list on the repository includes self-hosted and ai-local, which matches the README's framing. If your constraint is not privacy or cost but raw throughput, the local-first design is a cost, not a feature: you are limited to whatever your card can hold.

How Modly is put together: Electron shell, Python backend, workflow graph

The repository layout makes the architecture legible without running anything. There is an electron/ directory and an src/ directory for the TypeScript side, an api/ directory for Python, and a tools/ directory holding the CLI. The build config is electron.vite.config.ts, and package.json names electron-vite as the dev and build driver, with the main entry at ./out/main/index.js. So the app is an Electron main process plus a renderer, and the Python service under api/ is a separate process the app talks to.

Generation is expressed as a graph, not a button. The README's own walkthrough says to start on the Workflows tab with Image -> Generate Mesh -> Add to Scene, and warns that there must be a connection between each step. The graph is validated before a run: the platform notes state that invalid graphs stay in place and surface inline or toast warnings rather than dropping the current mesh view. That is a deliberate choice to preserve in-progress work at the cost of letting a broken graph sit on screen.

Models are not baked into the binary. They arrive through the extension system, where each extension is a GitHub repository containing a manifest.json plus the runtime entry files its type requires. The README lists five official extensions: three Hunyuan3D 2 Mini variants (standard, Turbo, Fast), TripoSG, and Trellis2 GGUF. Process extensions, by contrast, are described as ready once installation and setup complete, with no model download step. The mesh viewport is built on @react-three/fiber and @react-three/drei, with @mkkellogg/gaussian-splats-3d present for splat rendering and @xyflow/react for the node graph itself.

Installing Modly and running a first image-to-mesh job

There are two paths. The README points to the Releases page for installers for Windows, Linux, or Apple Silicon macOS. The second path is to clone the repository and run it directly, which the README shows as launch.bat on Windows and ./launch.sh on Linux and macOS.

For a development checkout, the README gives four steps. First install the JavaScript dependencies:

npm install

Then create the Python environment for the backend. The README uses a venv inside api/ and installs from requirements.txt:

cd api
python -m venv .venv
.venv\Scripts\activate     # Windows
source .venv/bin/activate  # Linux / macOS
pip install -r requirements.txt

With both sides prepared, the dev command starts the app:

npm run dev

The README notes that dev is not a bare electron-vite call: package.json defines it as node scripts/build-builtins.mjs && electron-vite dev, so builtins are compiled first. The README's test sequence is npm test, then a type check with ./node_modules/.bin/tsc --noEmit -p tsconfig.node.json, then npm run build. Packaging is separate: npm run package for the general case and npm run package:mac for Apple Silicon, both of which run prepare-resources, which invokes scripts/download-python-embed.js.

For the actual first generation, follow the README literally. Open the Workflows tab and build Image -> Generate Mesh -> Add to Scene with a connection between every step, then go to the Generate tab, confirm the workflow is selected, and click Generate 3D Model. If nothing happens, the README directs you to Settings/Logs/Errors. Before any of that produces a mesh, you need a model: on the Models page, click Install from GitHub, paste the HTTPS URL of an extension repository, and then download the model or one of its variants if the extension exposes model nodes.

The CLI is the part that matters for automation

Modly ships a stdlib-only Python CLI at tools/modly-cli/agent.py that talks to a running desktop app. The README's examples show health checks, model listing, run status, and a friendly generate command:

python tools/modly-cli/agent.py health
python tools/modly-cli/agent.py model list
python tools/modly-cli/agent.py workflow-run status <run_id>
python tools/modly-cli/agent.py generate --image ./input.png --output ./export.glb

The README states that generate is a convenience wrapper: it starts POST /workflow-runs/from-image, polls the returned run, exports the final mesh when requested, and includes recovery metadata such as workflow-run status and workflow-run cancel in the JSON response. Final machine-readable output goes to stdout.

The distinction the README draws between canonical and non-canonical surfaces is worth taking seriously if you are scripting against this. The canonical root commands are health, model, workflow-run, capability, and process-run. Everything else is labeled: legacy wraps the old /generate/* job endpoints, dev serve-api and dev ensure-server start only the FastAPI backend and do not prove that the Electron or desktop bridge is ready, and experimental comfy-image and experimental generate-from-workflow are described as external ComfyUI orchestration helpers rather than the canonical agent contract. Hidden aliases like status, export, and batch remain parseable but are not presented as root commands. Build on the canonical five. The README also points to tools/modly-cli/SKILL.md for the agent workflow and output contract, which is where you should look before writing a client.

Where Modly is the wrong tool

The clearest boundary is hardware. macOS support targets Apple Silicon only, so Intel Macs are out. On Windows and Linux, generation quality and speed are bounded by your GPU, and the model you pick determines the floor: the README lists both a Trellis2 GGUF extension and a Hunyuan3D 2 Mini Fast variant, which implies a spread of memory footprints wide enough that the heaviest option may not fit on a mid-range card. The README does not publish per-model VRAM requirements, so you cannot size your machine from the documentation alone.

A second boundary is that the app is not self-sufficient. It is a shell plus a backend plus separately installed extensions plus separately downloaded weights. The README's own install flow has four distinct stages before a mesh exists, and the Models page step explicitly separates installing an extension from downloading its model or variant. If you want a single binary that works offline immediately after installation, this is not that.

A third is the automation surface. The CLI talks to a running desktop app. It is not a headless server mode. The README warns that dev serve-api and dev ensure-server start only the FastAPI backend and do not prove Electron or desktop bridge readiness, which tells you the desktop process is the real coordinator. If your target is a CI container with no display, the canonical CLI contract is not aimed at you. Finally, the README documents no rollback or version-pinning procedure for extensions, so an extension repository that changes under you is a real operational risk that the documentation is silent on.

Comparing Modly with a hosted image-to-3D API

The obvious alternative is a cloud image-to-3D service, where you send an image over HTTP and receive a mesh. The difference is not quality, it is where the compute and the data live. A hosted API gives you elastic GPU capacity, no driver installation, and no model downloads; it also gives you per-request cost, a network dependency, and a copy of your input on someone else's infrastructure. Modly gives you the inverse on every point.

Within the local category there is also a second kind of alternative: running the underlying models yourself, directly, without Modly. The models Modly wraps are open source and reachable through the extension repositories it links, so a team with ML infrastructure can call them from their own scripts. What Modly adds on top is the desktop shell, the node-graph workflow with pre-run validation, the mesh viewport, in-app smoothing and decimation (the platform notes say optimized results are written back into the workspace), and the CLI contract. If you already have a generation pipeline and only need the model, Modly's value is mostly the graph editor and the packaging, not the model itself.

The honest comparison point is ComfyUI. The README itself treats ComfyUI as adjacent enough to ship experimental helpers for it, including a generate-from-workflow command that takes a workflow name and an output path and downloads a produced 3D asset directly when the workflow yields one. Modly's own framing is that this is a compatibility path, not the canonical contract.

Maintenance, licensing and what a fork owes upstream

The repository is not archived, and the last push was on 2026-09-06, which is recent. Releases are versioned and dated: v0.4.0 on 2026-06-21, v0.4.1 on 2026-07-16, and v0.4.2 on 2026-08-28, all labeled Beta. The cadence visible in those three dates is roughly monthly, and package.json carries version 0.4.2, matching the latest release tag. The Beta label is worth noting on its own: this is pre-1.0 software, and the README's careful separation of canonical, legacy, dev and experimental CLI surfaces reads like a project that has already moved interfaces once.

Upgrade cost depends on which surface you depend on. The desktop app ships with electron-updater in its dependencies, so the packaged app has an update path. Extensions are the looser end: because they are GitHub repositories installed by URL, their contents can change independently of the app, and the README does not document pinning a commit or a version. If you rely on a specific extension, record the commit you installed from yourself.

The license situation needs care. The repository's license metadata is NOASSERTION, while the README states MIT License and points to LICENSE for details. The README also states a condition beyond plain MIT: if you fork this project and build your own app from it, you must credit the original project and its creator, with a specific attribution line given in the README. Treat that as a project requirement to read in full rather than a formality. This is a description of what the files say, not legal advice; if you plan to redistribute a fork commercially, have someone qualified read LICENSE and the extension licenses.

Editorial conclusion

Adopt Modly if you need image-to-3D generation that never leaves your machine, you have a GPU you are willing to keep busy, and you are comfortable installing a Python backend next to an Electron front end. Do not adopt it if you need a hosted API with an SLA, if you are on Intel macOS, or if you expect a one-click installer to also fetch model weights for you. Before committing, verify three things: that your GPU has enough VRAM for the specific extension's model, that the extension repository you want still resolves over HTTPS, and that the CLI's canonical commands (health, model, workflow-run, capability, process-run) cover the automation you plan to build, rather than the legacy or experimental surfaces.

Frequently asked questions

Official sources

1. Issues
2. lightningpixel/modly on GitHub
3. Project website
4. README
5. Releases