comfyui_controlnet_aux: ControlNet Preprocessor Nodes for ComfyUI Workflows

What comfyui_controlnet_aux Does and Who It Is For

ControlNet is a technique for guiding diffusion model image generation through structural hint images rather than text alone. A hint image encodes some property of a source image, such as the positions of edges, the depth from camera, the skeleton of a pose, or the boundaries of semantic regions. The hint is then passed alongside the text prompt to a ControlNet-aware sampler, which uses it to constrain the spatial structure of the output.

Producing that hint image is the job of a preprocessor. comfyui_controlnet_aux is a set of plug-and-play ComfyUI nodes that each run one type of preprocessor. The repository was created by Fannovel16 and, according to its README, the annotator code is copied from lllyasviel's ControlNet repository and connected to the Hugging Face Hub for model weights. All credit for the underlying methods belongs to lllyasviel.

The pack is intended for ComfyUI users who build node-graph workflows and need to convert a photograph or illustration into one of the hint types that ControlNet models accept. It does not include the ControlNet models themselves. Those must be obtained and placed in the correct ComfyUI models folder separately.

How the Preprocessor Nodes Work: Weights from Hugging Face, Results as Tensors

Each preprocessor node accepts an image tensor in ComfyUI's standard format and returns a processed hint image in the same format. Internally, the node loads an annotator model, runs inference on the input, and returns the result. The annotator weights are not bundled with the repository. They download automatically from the Hugging Face Hub on first use, which means an internet connection is required the first time any given preprocessor is run.

All preprocessors except Inpaint are wrapped into a single AIO Aux Preprocessor node that exposes a dropdown for selecting the preprocessor type. This allows quick switching between hint types without rewiring the graph. The trade-off is that the AIO node does not expose the individual threshold parameters each preprocessor normally accepts. If you need to adjust the low or high threshold on the Canny Edge preprocessor, for example, you must use the dedicated Canny Edge node rather than the AIO wrapper. The README is explicit about this limitation.

The project version in pyproject.toml is 1.1.6. The dependency list in that file includes torch, torchvision, opencv-python, mediapipe, scipy, einops, scikit-image, trimesh, albumentations, scikit-learn, and omegaconf, among others.

Installing comfyui_controlnet_aux in ComfyUI

The recommended installation method is through ComfyUI Manager. If ComfyUI Manager is already installed in your ComfyUI setup, open it, search for comfyui_controlnet_aux, and install from there. No manual steps are needed.

For a manual installation on Linux or on a Windows account without administrator rights, clone the repository into your ComfyUI custom_nodes folder:

git clone https://github.com/Fannovel16/comfyui_controlnet_aux/

Then install the Python dependencies. On a portable ComfyUI installation that ships with an embedded Python runtime, use:

path/to/ComfUI/python_embeded/python.exe -s -m pip install -r requirements.txt

On a system Python installation, use:

pip install -r requirements.txt

The README also notes that Windows users on a portable install can run install.bat, which detects the portable setup and installs to the correct location.

After installation, restart ComfyUI. The nodes appear in the node menu grouped under ControlNet Preprocessors. A minimal test workflow connects a Load Image node to a Canny Edge node, then passes the Canny Edge output to a ControlNet Apply node and from there to a sampler. This confirms the node is installed and that the Canny annotator model downloads and runs correctly.

Preprocessor Categories and Model Pairings

The README organizes the preprocessors into several categories.

The line extractor category includes Canny Edge, HED Soft-Edge Lines, PiDiNet Soft-Edge Lines, Standard Lineart, Realistic Lineart, Anime Lineart, Manga Lineart, M-LSD Lines, TEED Soft-Edge Lines, AnyLine Lineart, and several scribble variants including Scribble Lines, Scribble XDoG Lines, and Scribble PiDiNet Lines. Each preprocessor maps to one or more ControlNet models listed in the README tables, such as control_v11p_sd15_canny for Canny or control_v11p_sd15_softedge for HED.

The depth and normal category includes MiDaS Depth Map, LeReS Depth Map, Zoe Depth Map, Depth Anything, Depth Anything V2, Zoe Depth Anything, Metric3D Depth, BAE Normal Map, MiDaS Normal Map, Normal DSINE, and Metric3D Normal. Depth Anything and its Zoe variant pair with the Depth-Anything diffusion model checkpoint.

The face and pose category includes DWPose Estimator and OpenPose Estimator. DWPose maps to control_v11p_sd15_openpose. OpenPose has variants for body-only, body plus hand, face-only, and full detection. A MeshGraphormer Hand Refiner node handles depth-based hand refinement and pairs with control_sd15_inpaint_depth_hand_fp16.

Further categories cover semantic and panoptic segmentation, tile and recolor processing, optical flow estimation through UniMatch, animal pose estimation, and face parsing. Example workflow screenshots for many of these are stored in the repository's examples folder.

Limitations: No Bundled Models, Dependency Conflicts, No ONNX on ARM

The pack produces hint images only. It does not supply the ControlNet models that consume those hints. A depth hint produced by the Zoe Depth Map node must be paired with a compatible depth ControlNet checkpoint, not a softedge or pose model. The README tables document the correct pairings, but the pairing responsibility rests entirely with the user.

The Inpaint preprocessor is not included in the AIO Aux Preprocessor node. If your workflow needs inpaint preprocessing, you must use its dedicated node, and it is not covered by the AIO wrapper.

Dependency conflicts are a practical problem in larger ComfyUI setups. The requirements.txt lists packages such as mediapipe, trimesh with the easy extra, albumentations, and scikit-learn. Other custom node packs in the same environment may require different versions of these packages, and pip cannot always satisfy both constraint sets simultaneously.

The requirements.txt specifies onnxruntime-gpu conditionally, limited to Windows and Linux on x86_64 architecture. GPU-accelerated ONNX inference on ARM platforms, including macOS Apple Silicon, does not have a supported install path in the requirements.

The repository has no GitHub release tags. Update history is tracked only in UPDATES.md. There is no semantic versioning release channel to subscribe to for change notifications.

Alternative: sd-webui-controlnet

The main alternative for ControlNet preprocessing is the sd-webui-controlnet extension for the Automatic1111 stable-diffusion-webui interface. That extension integrates preprocessors directly into a form-based web UI. Configuration is done through dropdown menus and sliders rather than a node graph, and no explicit wiring between nodes is needed.

comfyui_controlnet_aux takes the node-graph approach. Each preprocessor is an explicit node, and its output tensor can be routed to multiple downstream nodes, saved to disk, or combined with other hint images. This makes multi-hint workflows, conditional branching, and batch processing straightforward to express.

The README maps each comfyui_controlnet_aux node to its sd-webui-controlnet counterpart in the preprocessor tables. Engineers migrating an existing webui workflow to ComfyUI can use those tables to identify which node replaces each webui preprocessor setting.

License and Maintenance Status

The repository is licensed under Apache-2.0. The README credits all underlying annotator methods to lllyasviel, whose original ControlNet repository and Hugging Face Hub models are the source of both the code and the weights.

The last push to the repository was on 2026-08-27. The repository is not archived.

Because annotator weights are fetched from the Hugging Face Hub at runtime rather than stored in the repository, continued operation depends on those model files remaining available at the expected paths. If Hugging Face removes or moves the annotator checkpoints for a given preprocessor, that preprocessor will fail when run on a machine that has not already cached the weights locally.

Editorial conclusion

comfyui_controlnet_aux is the right tool for any ComfyUI workflow that requires a preprocessor to produce ControlNet hint images. It is the wrong tool if you need the ControlNet models themselves, which must be downloaded separately and matched to the correct preprocessor. Before adopting it, verify that your Python environment can install the full dependency list in requirements.txt, particularly mediapipe, trimesh, and albumentations, all of which have version constraints that may conflict with other custom nodes in the same ComfyUI setup.

Frequently asked questions

Official sources

1. Fannovel16/comfyui_controlnet_aux on GitHub
2. Issues
3. License: Apache-2.0
4. README