hls4ml: Compiling Keras and PyTorch Models into FPGA Firmware

The Latency Problem hls4ml Was Built Around

A trained model served from a CPU or GPU passes through a runtime: memory copies, kernel launches, batching. For most applications that overhead is invisible. For a trigger decision inside a particle detector, or a feedback loop in a fusion or quantum control system, it is the whole problem. The README frames hls4ml as a package for machine learning inference in FPGAs, creating firmware implementations of machine learning algorithms using high level synthesis, with ultra-low-latency inference as the design goal. The origin is explicit: the project has strong roots in high-energy physics applications such as L1 trigger systems at the CERN Large Hadron Collider. That origin explains the design. hls4ml does not serve models; it emits hardware descriptions of them. The audience is therefore narrower than the pip install suggests: engineers who have a trained network, a latency or power budget that rules out a general-purpose processor, and access to an FPGA synthesis toolchain. The README also lists adoption outside physics, including quantum computing control systems, nuclear fusion feedback loops, satellite environmental monitoring and biomedical signal processing such as arrhythmia classification. Those domains share one property: the inference has to complete within a fixed, short window.

From Keras Config to HLS Source: The Conversion Path

The mechanism is a converter plus a configuration dictionary. In the getting-started example, hls4ml.utils.fetch_example_model('KERAS_3layer.json') downloads an example model and returns a configuration file, which is then printed and passed to hls4ml.converters.keras_v2_to_hls(config). The result is an hls_model object that represents an HLS project on disk. The configuration is the control surface: it carries the default parameters that the converter reads when deciding how each layer is expressed in HLS. That is the central abstraction. hls4ml does not run your model and record its behaviour; it walks the model graph and emits C++-like HLS code whose numeric types, parallelism and pipelining are set by the config. The repository topics name the supported front ends: keras, pytorch and onnx appear alongside the backends vivado and vivado-hls. The README mentions Keras, PyTorch and ONNX models, and the citation list points to separate papers for specific feature families, including convolutional networks, distributed arithmetic, and binary and ternary precision networks. That structure matters for evaluation: the core converter handles a baseline set of layer types, while CNNs, quantization schemes and arithmetic implementations arrived through later work with their own papers. If your model depends on one of those features, the relevant question is not whether hls4ml supports it in general but whether the converter you are calling exposes it through the config keys you have.

Backends, Synthesis and the Report Reader

Conversion produces a project; building it produces firmware. The README states that the project is built using Xilinx Vivado HLS, and that alongside it hls4ml supports Vitis HLS, Intel HLS, Catapult HLS and has some experimental support for Intel oneAPI. The backend is selected with the backend argument when building the model. The example calls hls_model.build() with a comment noting it might take several minutes, and then hls4ml.report.read_vivado_report('my-hls-test') to print the synthesis report. Two things follow from this. First, the build step is not self-contained: it shells out to a vendor tool that you must install and license separately, and the README links to Xilinx's download page rather than bundling anything. Second, the feedback loop is long. Synthesis of a small network takes minutes according to the README's own comment, and the useful output arrives as a report rather than an exception. The report reader is the practical interface for the iteration cycle: you change the configuration, rebuild, and read the resource and timing numbers back through hls4ml.report. The README does not describe how errors surface when a layer is unsupported or a config key is wrong, so plan on discovering the failure modes empirically rather than from the documentation.

The Vendor Toolchain Is the Real Dependency

The pip install is one line. The rest of the setup is not, and the README is honest about the split without dwelling on it. pip install hls4ml gives you the Python package; pip install hls4ml[profiling] adds the extra dependencies for profiling. Neither installs a synthesis tool. The README tells you to download and install Vivado HLS from Xilinx's page, and names Vitis HLS, Intel HLS, Catapult HLS and experimental Intel oneAPI support as alternatives. This is the wrong tool for anyone who wants a self-contained deployment artifact. There is no path here to a Docker image that runs your model; the output is HLS source that must be compiled by a vendor compiler into a bitstream for a specific part. That has consequences the README does not spell out. Your CI cannot validate a build unless the runner has the vendor tool and a licence. Reproducing a build on a different machine means reproducing the tool version, not just the Python environment. And the experimental label on Intel oneAPI support is a signal worth taking literally when choosing a backend. The Apache-2.0 licence covers the hls4ml source. It does not cover Vivado, Vitis, Intel HLS or Catapult, each of which carries its own licence terms, and the README does not attempt to describe those. This article is not legal advice; read the vendor agreements.

Where the Abstraction Leaks

The honest limitation is that hls4ml converts a model graph, and model graphs contain operations that do not map cleanly onto fixed-function hardware. The README's own citation list is the evidence. Separate papers exist for CNNs, for distributed arithmetic, and for binary and ternary precision, which means these were not part of the original conversion path and were added as distinct bodies of work. The README also flags Intel oneAPI support as experimental, which is the only explicit maturity qualifier in the text. Beyond that, the material does not describe how unsupported layers are handled, whether the converter raises or silently approximates, or what the numeric consequences of a given configuration are. That gap is the thing to probe before adopting. A second limitation is structural rather than technical: the project is designed for inference. Nothing in the README suggests on-device training or dynamic model updates, and the fixed nature of synthesized hardware makes those poor fits by construction. If your model changes shape per request, or your input dimensions vary at runtime, the conversion-plus-synthesis pipeline is the wrong shape of tool. The README's example models are fetched from a repository and converted, which is a static flow; treat that as the intended mode of use.

How hls4ml Differs from ONNX Runtime and TensorRT

The obvious comparison is a software inference runtime such as ONNX Runtime or TensorRT. The difference is not speed, it is what gets produced. ONNX Runtime loads an ONNX graph and executes it on whatever hardware is present, with the graph remaining data that the runtime interprets. hls4ml reads a Keras, PyTorch or ONNX model and emits HLS source, which a synthesis tool compiles into a circuit. Once synthesized, the model is no longer data; it is the hardware. That changes the operational profile in both directions. You gain a fixed, short latency that does not depend on a scheduler or a memory hierarchy, which is why the L1 trigger use case exists. You lose the ability to swap models without resynthesizing, and you take on the vendor toolchain as a build dependency. The comparison to a hand-written HLS or RTL implementation is the other relevant axis. Writing the datapath by hand gives full control over numeric formats and pipelining, and hls4ml's value is that it generates that code from a model you already trained, with the configuration dictionary as the knob. Whether the generated code matches a hand-tuned implementation is not something the README claims, and it is not something this article can assert without running synthesis.

Release Cadence and What Maintenance Actually Costs

The release history is legible: v1.1.0 (gladiolus) in March 2025, v1.2.0 (hyacinth) in November 2025, v1.3.0 (iris) in March 2026, with the repository showing activity into September 2026. Roughly two releases a year, each carrying a botanical codename. That cadence is neither alarming nor a guarantee; it tells you the project is maintained without telling you how much churn an upgrade brings, and the README does not include a compatibility policy for the configuration format. Since the config dictionary is the interface between your model and the generated hardware, a change to how a config key is interpreted can change your synthesized result without changing your Python code. That is the maintenance cost specific to this project, and it is different from upgrading a runtime library. Budget for re-running synthesis and re-reading the report after each version bump, not just for re-running a test suite. The citation guidance reinforces the point: the README asks users to cite additional papers when they use features developed later, such as CNNs, distributed arithmetic, or binary and ternary networks. Those features are the ones most likely to move between releases. The Apache-2.0 licence permits commercial use and modification of the hls4ml source, but it says nothing about the vendor synthesis tools, and it does not obligate the maintainers to keep the config format stable across minor versions.

Editorial conclusion

Adopt hls4ml if your inference budget is measured in clock cycles and you already have a Vivado, Vitis or Intel HLS install to build against; skip it if you need dynamic shapes, training, or a deployment path that does not involve an FPGA vendor toolchain. Before committing, run hls4ml.utils.fetch_example_model on your own exported model, check which layers survive conversion, and confirm that the backend you intend to use is the one your synthesis licence covers.

Official sources

1. fastmachinelearning/hls4ml on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases