DreamLayer Eval: A Local Benchmark Harness for Diffusion Models
DreamLayer Eval: open-source benchmarking for image and video diffusion models. Automate prompts, seeds, and metrics for reproducible results. A research project from DreamLayer AI (dreamlayer.io).
At a glance
- What is it?
- DreamLayer Eval is a GPL-3.0 Python toolkit that sweeps prompts and seeds across image and video diffusion models, computes metrics during generation, and logs every run to SQLite. The design is sound for lab-internal comparisons; the setup path and the model licensing around what you benchmark are the parts to check before you commit.
- Who is it for?
- Adopt DreamLayer Eval if you already run ComfyUI locally and need repeatable, run_id-addressable comparisons of checkpoints, LoRAs, or samplers across many prompts and seeds. Do not adopt it if you need a hosted service, if your team cannot maintain a Python plus Node plus ComfyUI stack, or if you expect the project to ship model weights for you.
- Can I use it commercially?
- Yes, with conditions. GPL-3.0 is a copyleft licence: if you distribute software that includes it, you must release that software's source code under the same licence. Running it internally without distributing it does not trigger that obligation.
- Is it still maintained?
- Yes. The repository last received commits 7 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The comparison problem DreamLayer Eval tries to remove
Diffusion model evaluation is usually done by hand. Someone writes a prompt list, picks a seed, generates a grid, changes one variable, and repeats. The result is a folder of images and a spreadsheet that nobody can reproduce six weeks later, because the sampler, step count, CFG value, or LoRA stack was not recorded alongside the output. DreamLayer Eval targets exactly that gap. The README frames it as automating prompts, seeds, metrics, configs, and reproducible run logging so that researchers and developers can compare model quality consistently. The stated audience is narrow and specific: AI researchers comparing diffusion models, ML engineers evaluating generation quality, labs building internal benchmarking workflows, and open-source model creators testing checkpoints, LoRAs, and workflows. If your evaluation is a one-off visual check of two checkpoints, this tool is more machinery than the task needs. It pays off when the same sweep has to run again next month against a new checkpoint and produce numbers that can be placed next to the old ones.
What a run actually records, and why run_id matters
The reproducibility claim rests on one design decision: every run persists to SQLite. According to the README, the stored record includes the prompt, negative prompt, seed, sampler, steps, CFG, model hash, LoRA stack, ControlNet configuration, and all computed metrics. Any run can then be replayed by run_id. That is the mechanism that separates this from a script that dumps PNGs into a directory. The model hash is the field that does the real work, because it ties a result to a specific checkpoint rather than a filename that may have been overwritten. The trade-off is that SQLite is a single-file store. It is fine for a lab workstation or a shared internal box, and it is not a multi-user database. If two people run sweeps against the same file concurrently, the README does not describe any coordination, locking, or merge behaviour, so treat the database as a per-machine artifact and copy it deliberately if you need to share results.
Architecture: React, Flask, SQLite, ComfyUI
The stack is stated plainly in the README: a React frontend, Flask-based services, SQLite run storage, and ComfyUI integration for image workflows. The frontend is served on port 8080 and ComfyUI on port 8188. That split matters for anyone deploying this. DreamLayer Eval is not a library you import into a training script. It is a local application you start, and ComfyUI is a dependency of the image path rather than an optional backend. The README notes a DREAMLAYER_COMFYUI_CPU_MODE variable for machines without NVIDIA drivers, defaulting to false, which tells you the expected path is GPU-backed generation through ComfyUI. Video benchmarking is listed among the capabilities, but the README does not describe a separate video backend, so how video generation is wired into ComfyUI is not something you can confirm from the supplied material. Verify that before planning a video evaluation around it.
Getting it running: scripts, ports, and environment keys
Installation is script-driven per platform. Linux uses ./install_linux_dependencies.sh, macOS uses ./install_mac_dependencies.sh, and Windows uses .\install_windows_dependencies.ps1, with a note to run Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass if script execution is blocked for the session. Starting the application is ./start_dream_layer.sh on Linux and macOS, and start_dream_layer.bat on Windows. Two environment variables are documented. DLVENV_PATH sets the Python virtual environment location and defaults to /tmp/dlvenv, which is worth overriding on any machine where /tmp is cleared on reboot. DREAMLAYER_COMFYUI_CPU_MODE switches to CPU-only generation when no NVIDIA driver is present and defaults to false. The README also offers a Cursor-based path: open the folder in Cursor and type run it, and it walks through dependency installation, virtual environment creation, and starting both services, with a stated time of about 5 to 10 minutes. It warns that PyTorch setup on macOS may take a few retries. That warning is honest and worth taking at face value. A /tmp default for the virtual environment plus a multi-service startup is a setup that will break in ways unrelated to the benchmarking code, so budget time for the environment before you judge the metrics.
Models and metrics: what ships and what you supply
DreamLayer Eval ships without weights. Open-source checkpoints are placed in auto-created folders (Checkpoints, Lora, ControlNet, VAE) and appear in the UI after Settings > Refresh Model List. The README suggests symbolic links if your checkpoints live on another drive. Closed-source API models are enabled by presence of a key in ./.env: OPENAI_API_KEY, BFL_API_KEY, IDEOGRAM_API_KEY, STABILITY_API_KEY. The rule is explicit: no key, the feature stays hidden. On the metric side, the README lists CLIPScore (ViT-L/14), FID, LAION aesthetic, color harmony, sharpness, and YOLOv8 composition F1 for images, and FVD (I3D), SSIM, PSNR, LPIPS, temporal flickering, subject and background consistency via DINO, and motion smoothness for video. Custom metrics are described as pluggable. The YOLO model (yolov8n.pt, about 6MB) auto-downloads on first use, and FID scoring requires a separate step: python scripts/fetch_datasets.py pulls the CIFAR-10 reference set. Note the mismatch there. FID is distribution-level, so scoring against CIFAR-10 tells you about a reference distribution that almost certainly is not your target domain. The script is useful for smoke-testing the metric pipeline, not for a meaningful FID number on modern text-to-image output.
Where it is the wrong tool, and what to use instead
Two limits deserve direct statements. First, DreamLayer Eval is a local application with a GPU dependency on the image path, so it does not fit teams that want a hosted evaluation endpoint or CI-native scoring without a ComfyUI instance. Second, the FID reference set is CIFAR-10, which is a weak proxy for diffusion output quality. If distributional fidelity against a domain-specific reference is the question, a harness like torch-fidelity is the more direct route: it is a Python library you call from your own code, it takes an arbitrary reference image directory, and it computes FID, IS, and KID without a frontend, a Flask service, or ComfyUI in the loop. The difference in approach is the whole point. torch-fidelity is a metric function you own and wire into whatever pipeline you already have. DreamLayer Eval is an application that owns the sweep, the generation, the storage, and the metric computation, and gives you a UI and run history in return. Pick based on whether you want a function or a workbench.
Licence, maintenance, and what to confirm before adopting
The repository is GPL-3.0. If you modify DreamLayer Eval and distribute it, or distribute a derivative that links against it, the GPL obligations attach to your distribution. Running it internally to produce benchmark numbers is a different situation from shipping a product built on its code, but the boundary depends on your use, and this is not legal advice. Check with your own counsel if you plan to redistribute anything derived from it. The repository is not archived and the last push is dated 2026-09-09, but no releases were retrieved, so there is no versioned artifact to pin. That means you are tracking main. For a benchmarking tool, that is a real cost: a change to how a metric is computed can invalidate comparisons against your earlier runs, and without releases there is no tag to freeze against. Before you build a comparison baseline on this, record the commit hash you installed, keep the SQLite file with it, and re-run one small sweep after any pull to confirm the numbers still line up. If they do not, you have a metric change, not a model change.
Editorial conclusion
Adopt DreamLayer Eval if you already run ComfyUI locally and need repeatable, run_id-addressable comparisons of checkpoints, LoRAs, or samplers across many prompts and seeds. Do not adopt it if you need a hosted service, if your team cannot maintain a Python plus Node plus ComfyUI stack, or if you expect the project to ship model weights for you. Before trusting any number it produces, verify three things on your own hardware: that your ComfyUI install matches the path the start script expects, that the metric you care about (FID, FVD, CLIPScore, or a custom one) runs end to end on a small sweep, and that the licence terms of every checkpoint you place in Checkpoints/ permit the evaluation you intend to publish.
Community notes