torch-fidelity computes six generative metrics over one cached feature set

Why six metric implementations needed one PyTorch package

Generative image evaluation is a pile of small formulas that most groups end up rewriting for themselves: Inception Score, Fréchet Inception Distance, Kernel Inception Distance, Precision and Recall, Perceptual Path Length, and Monge Inception Distance. torch-fidelity gathers all six into one PyTorch package and makes the accuracy claim its selling point, that the values match reference implementations up to floating point's machine precision. That claim only matters when a number is going into a table beside somebody else's number, which narrows the audience to people training GANs, diffusion models or flow-matching setups who need ISC, FID and KID on a schedule rather than once at the end. One executable handles the command line, one function handles the library path, and both emit the same metric names.

Feature sharing and a cache sit between the inputs and the score

The expensive stage in any of these metrics is feature extraction, not the algebra on top of it. torch-fidelity computes features once and shares them across whichever metrics you request, then adds a second caching level so features and per-metric statistics survive between calls whenever that is possible. Ask for ISC, FID, KID and PRC in the same call and you pay for one pass over the images rather than four. The package description names InceptionV3, CLIP, DINOv2 and VGG16 among the supported feature extractors, which means the value you get is a function of the extractor you select and not only of the formula being implemented. The stated design target is the training loop, where the cache is what makes end-of-epoch evaluation affordable at all.

Two inputs, and a wrapper whose fourth argument is unnamed

Every call takes `input1` and `input2`, and they are not interchangeable. A bundled dataset alias such as `cifar10-train` can stand in for real images with nothing on disk, while a generative model is supplied as `~/generator.onnx` and described by flags that pin the latent behaviour: `--input1-model-z-type normal`, `--input1-model-z-size 128` and `--input1-model-num-samples 50000`. In Python the same job is done by `torch_fidelity.GenerativeModelModuleWrapper(generator, 128, 'normal', 0)`, whose four positional arguments are the model, the latent dimension, the latent type, and then a bare 0 that the README never explains. A generator that consumes a different latent type or dimension cannot be described by those three flags at all. The wrapper is the only documented route for passing an in-memory model to the package.

Three command line examples that grow in cost

The PyPI name and the import name differ, so install one and import the other:

pip install torch-fidelity

The README points to a separate documentation page for installing the latest GitHub code as a nightly version. With no image files at all, one flag and one dataset alias are enough:

fidelity --gpu 0 --isc --input1 cifar10-train

The printed result is given as `inception_score_mean: 11.23678` and `inception_score_std: 0.09514061`. Swapping the alias for a directory of your own pictures is the entire difference between the simple and the medium example:

fidelity --gpu 0 --isc --input1 ~/images/

The third example adds a generator and a reference set, and asks for all six metrics over one set of features:

fidelity \
  --gpu 0 \
  --isc \
  --fid \
  --kid \
  --ppl \
  --prc \
  --mind \
  --input1 ~/generator.onnx \ 
  --input1-model-z-type normal \
  --input1-model-z-size 128 \
  --input1-model-num-samples 50000 \ 
  --input2 cifar10-train 

Each additional metric flag reuses features already extracted, which is the recomputation saving the package describes, though the marginal cost per flag is not documented. Note what has to be supplied by hand: `--gpu 0` appears in every example, and nothing in the interface selects a device for you.

Attaching the same call to the end of an epoch

The library path exists for the case where per-epoch numbers matter more than a single final report:

import torch_fidelity

Wrap the model and call the metric function from an epoch callback:

wrapped_generator = torch_fidelity.GenerativeModelModuleWrapper(generator, 128, 'normal', 0)

metrics_dict = torch_fidelity.calculate_metrics(
    input1=wrapped_generator, 
    input2='cifar10-train', 
    cuda=True, 
    isc=True, 
    fid=True, 
    kid=True, 
    prc=True, 
    verbose=False,
)

What comes back is an ordinary dictionary rather than a log line, so tensorboard, wandb or a plain console all accept it with no parsing in between, and `verbose=False` keeps progress output out of a training log. One gap between the two examples is easy to miss: the command line demo covers `--ppl` and `--mind`, while neither appears among the keyword arguments of this Python call. A complete training script lives in `examples/sngan_cifar10.py`, and a generator checkpoint from that run is published as a download attached to tag v0.2.0.

The sample output dictionary would not parse

Paste the printed dictionary from the Python example into a script and the interpreter stops on it:

{
    'inception_score_mean': 11.23678, 
    'inception_score_std': 0.09514061, 
    'frechet_inception_distance': 18.12198, 
    'kernel_inception_distance_mean': 0.01369556, 
    'kernel_inception_distance_std': 0.001310059
    'precision': 0.51369556, 
    'recall': 0.501310059
}

The `kernel_inception_distance_std` line ends without a comma, so the value that follows it is a syntax error rather than a sixth key. Read the figures and a second oddity turns up: `precision` at 0.51369556 repeats the tail of `kernel_inception_distance_mean` at 0.01369556, and `recall` at 0.501310059 repeats the tail of `kernel_inception_distance_std` at 0.001310059. Two values carrying the exact trailing digits of the KID values above them is what a hand edited sample looks like, not what independent computation gives. The two Inception Score entries are the exception, matching the command line output exactly at 11.23678 and 0.09514061, so treat this block as a shape reference for the returned keys rather than as numbers to assert against.

Unpinned dependencies and no device fallback

Five runtime requirements are named in `requirements.txt`: numpy, pillow, torch, torchvision, tqdm. None carries a version bound, and `setup.py` reads that file directly at build time, taking the package version by executing the version module inside the package directory. Nothing there fixes a minimum Python version or a CUDA build, and `pyproject.toml` carries only a black configuration with `line-length = 120`, so what you install is whatever your index resolves on the day. That is the first thing to check when a metric value moves after an unrelated upgrade. Two operational limits follow from the interface itself: device placement is explicit everywhere, `--gpu 0` on the command line and `cuda=True` in Python, so a machine without a usable GPU needs code changes rather than a flag, and the alias `cifar10-train` brings a data fetch with it while the README documents no cache location and no offline switch. The licence is also something to read directly: GitHub classifies it as Other and points at the LICENSE file, while the repository root ships `LICENSE.md`, and the classification line says nothing about the actual terms. On timing, the last push was on 2026-05-11 and the newest tag is v0.4.0 dated 2026-02-17, with the tag before that, v0.3.0, dated 2021-06-15.

A metric aggregator that depends on this one

The package description states that torch-fidelity has been adopted as a foundational dependency by torchmetrics, which makes the alternative path easy to name. torchmetrics fronts many metric families behind one API and pulls torch-fidelity in for generative image work, so choosing it gives a single interface across vision, text and audio metrics at the cost of the direct control over which feature extractor and which latent settings feed the computation. Choosing the standalone package gives you the cache, the executable and the generator wrappers, and leaves the wiring to tensorboard or wandb in your own code. The other alternative named in the README is running the original TensorFlow implementations, described there as scattered and slow reference implementations, which is the same complaint that motivates the machine precision claim. For a metric destined for a results table, that is the trade that matters: matching a reference to machine precision removes the whole class of disagreement about whose environment produced the number, and moves the cost onto installing an ONNX generator or writing the wrapper yourself.

Editorial conclusion

Adopt torch-fidelity when your numbers have to sit next to published FID, KID or Inception Score values, because matching reference implementations to machine precision is the entire premise and the command line needs no code of yours. Skip it when no usable GPU is present, since `--gpu 0` and `cuda=True` are set by hand, or when you need a bounded dependency set, since `requirements.txt` names torch and torchvision without version limits. Before trusting a result, run `fidelity --gpu 0 --isc --input1 cifar10-train` and check the printed mean reads 11.23678.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. toshas/torch-fidelity on GitHub