Holistic Trace Analysis finds where distributed GPU training wastes time

A post-hoc analyser, not a profiler

HTA, abbreviated HTA, is described as a performance analysis tool to identify performance bottlenecks in distributed training workloads, and the mechanism is that it achieves this by analyzing traces collected through the PyTorch Profiler, also known as Kineto. That distinction decides how you should evaluate it. HTA does not start a run, does not instrument your model and does not hook into your training loop. It takes the trace files that the profiler already produced and computes a set of derived views over them. The library is MIT licensed, the primary language is Python, the documentation is at hta.readthedocs.io, and the project is maintained by a named group at Meta Platforms with past contributors listed separately. The release history shows recent work, with v0.5.0 in May 2025 and both v0.6.0 and v0.6.1 published on the same day in April 2026, and the last push to the repository was on 2026-10-08.

The eleven analyses, from time to counters

The feature list is the substance of the project, so it is worth reading as a set of questions you can now answer about a training run. Temporal breakdown splits GPU time across all ranks into computation, communication, memory events and idle time. Kernel breakdown finds the longest running kernels on each rank, and kernel duration distribution shows how that average varies between ranks, which is how you spot one straggler. Idle time breakdown splits GPU idle time into waiting for the host, waiting for another kernel, and an unknown cause, which is the breakdown that tells you whether you are input bound, synchronisation bound or just blocked. Communication computation overlap computes the percentage of time when communication overlaps computation, the number that decides whether all those communication calls are free. Frequent CUDA kernel patterns finds the kernels most often launched by a given PyTorch or user defined operator, and CUDA kernel launch statistics gives distributions of kernels with very small duration, large duration and excessive launch time. Augmented counters add memory bandwidth and queue length views, which are described as augmented trace files. Trace comparison identifies and visualises differences between traces. And CUPTI counter analysis is an experimental API for GPU performance counters, which by attributing measurements from kernels to PyTorch operators makes roofline analysis possible.

Installing the package and setting up a conda environment

HTA runs on Linux and Mac with Python 3.10 or later. The documentation offers an optional conda environment and then two install routes. The stable route is a pip install from PyPI.

pip install HolisticTraceAnalysis

The source route clones the repository and installs it in editable mode.

git clone https://github.com/facebookresearch/HolisticTraceAnalysis.git
cd HolisticTraceAnalysis
git submodule update --init
pip install -r requirements.txt
pip install -e .

The submodule step matters and is easy to miss, because the parser for trace events and the symbol table handling live in the third_party/ directory that the submodule pulls in. There is also a naming detail worth flagging, because the distribution on PyPI and the project name are not the same string. The pyproject.toml names the project traceinsight at version 0.6.1, while the installation command uses the HolisticTraceAnalysis name, so a reader who types the name from the package metadata will get a different result. The runtime requirements are modest and are listed in requirements.txt: jupyterlab, numpy, networkx, pandas, plotly and pydot, the last of which the file annotates as required by the third_party parameter directory.

Analysing a job in a notebook, one method at a time

The usage model is a Python object over a directory of traces, and every analysis is a method on it. The data preparation rule is simple and load bearing: all traces collected from a job must reside in a unique folder. The import and the object are two lines.

from hta.trace_analysis import TraceAnalysis
analyzer = TraceAnalysis(trace_dir = "/path/to/folder/containing/the/traces")

Then the analyses are called by name, and each returns a data frame.

# Temporal breakdown
temporal_breakdown_df = analyzer.get_temporal_breakdown()

# Kernel breakdown
kernel_breakdown_df = analyzer.get_gpu_kernel_breakdown()

# Idle time breakdown
idle_time_df = analyzer.get_idle_time_breakdown()

# Communication computation overlap
comm_comp_overlap_df = analyzer.get_comm_comp_overlap()

The later methods take more arguments, which is where the tool starts to shape output rather than just summarise it. The frequent patterns call needs an operator name and an output directory, the memory bandwidth series and summary are separate calls, and the queue length series and summary are likewise separate. That is a lot of methods to call, which is why the documentation points at the trace_analysis_demo and trace_diff_demo notebooks in the examples folder for a fuller picture. The examples directory is broad, with notebooks for critical path analysis, kernel breakdown, memory analysis, timeline plotting, trace filtering, straggler identification, custom trace parsing, and the CUPTI work.

Where the numbers come from, and the augmented file caveat

Two design points explain why some of these views are not simply a SQL query over the trace. The first is that the augmented counters are described as augmented trace files, which provide insights into memory bandwidth utilized and the number of outstanding operations on each CUDA stream. That means the analysis works over a modified trace, not the original one, and the CUPTI counters come from an experimental API that attributes performance measurements from kernels to PyTorch operators so that roofline analysis can be performed and kernels optimised. The second is that the idle time breakdown needs a categorisation step, splitting idle time into waiting for the host, waiting for another kernel, or an unknown cause, and that last bucket is where a large share of real findings land. The trace comparison side is a separate entry point, trace_diff.py, and the trace filter examples suggest you can narrow a trace before analysing it. Both entry points sit in the hta package alongside the analyzers directory that holds the core logic for each analysis and the common directory for code shared between them.

Configuration, the repo map and how to run the tests

Configuration is handled through files rather than code. The logging level is set in hta/configs/logging.config and can be changed in the logger_hta section of that file, and a different logging file can be pointed at by modifying hta/configs/trace_analyzer.json. The repository map in the documentation is worth reading before you navigate, because it shows the shape of the package. The hta directory contains the analyzers for each analysis, a common directory for shared code, the configs directory, and the two entry points trace_analysis.py for the TraceAnalysis API and trace_diff.py for the TraceDiff API, plus a utils directory. Alongside it sit examples with the demo notebooks, scripts described as generic tools for traces, and tests. The Makefile has two targets and a comment showing the intended order: install does pip install -e ., and test depends on install and then runs the unit tests with python3 -m unittest -v tests/*.py. The CI runs on CircleCI with codecov coverage, and the project asks that new features be discussed in an issue before the work starts.

Where HTA is the wrong tool

The limits are clear from the design. The first is that you need the trace first. If the run you want to understand produced no Kineto trace, HTA has nothing to read, and collecting a trace from a large distributed job is itself a decision with a cost in disk and in overhead. The second is that the multi-rank views are only as good as the trace set. The requirement that all traces from a job live in a single folder is convenient for the tool, but it means a partially collected job, or a job where one rank's trace is missing, produces cross-rank numbers that are quietly wrong rather than obviously broken. The third is the scope of the operators. The analysis is centred on CUDA kernels and GPU timelines, and the CUPTI counter path is described as experimental, so a question about host CPU time or network behaviour is not what this tool answers. The fourth is scale. Analysing a large trace is memory hungry, and the tool's own requirements include jupyterlab, pandas and plotly, which is a heavy set for something you might want to run inside a CI container. And the fifth is the honest one: for a single GPU job, the PyTorch profiler's own summaries will often settle the question, and HTA's advantage is in the cross-rank attribution that only exists once a job is distributed.

Editorial conclusion

Adopt HTA if you run distributed PyTorch training on GPUs and already produce Kineto traces, because the library turns a wall of trace events into an attribution of where the time actually went. Do not adopt it as a profiler, since it is a post-hoc analysis tool that consumes traces rather than collecting them, and do not adopt it for a single GPU job where the PyTorch profiler's own summaries are enough. Verify two things before you rely on it: that your traces cover every rank in one folder, since the documentation requires all traces from a job to reside in a unique folder for the cross-rank analysis to mean anything, and that the tool's memory bandwidth and queue length numbers line up with what you observe, since those come from augmented trace files that add counters to the original trace. The practical first step is to install the package, then point a TraceAnalysis object at one collected job's trace directory and call get_temporal_breakdown to see whether the numbers match your sense of the run.

Frequently asked questions

Official sources

1. facebookresearch/HolisticTraceAnalysis on GitHub
2. License: MIT
3. Project website
4. README
5. Releases