TorchCodec: media decoding that hands you tensors instead of a subprocess

Three approaches, and FFmpeg does the actual work

TorchCodec is described as a PyTorch-native library for decoding and encoding media, videos, audio and images, on CPU and CUDA GPU. The stated aim is to be fast, easy to use and well integrated into the PyTorch ecosystem, and the framing is that if you want to use PyTorch to train models on video, audio or images, or run inference, TorchCodec is how you turn these into tensors and back.

The three bullet points in the README are the design brief, and the middle one carries the most weight.

The APIs are Pythonic and mirror Python and PyTorch conventions. The data comes back as PyTorch tensors, ready to be fed into PyTorch transforms or used directly to train models.

And the implementation relies on FFmpeg to do the video and audio decoding and encoding. Importantly, TorchCodec uses the version of FFmpeg you already have installed, and the reasoning is given: FFmpeg is a mature library with broad coverage available on most systems, but it is not easy to use, so TorchCodec abstracts FFmpeg's complexity to ensure it is used correctly and efficiently.

That is the whole thesis of the project in two sentences. The alternative approaches in a training pipeline are to shell out to an FFmpeg binary and parse its output, or to bind to FFmpeg yourself and get the lifetime and threading hazards that come with it. TorchCodec takes the binding and puts a Pythonic interface and tensor semantics on top.

One qualification is stated in the same section: FFmpeg is optional, and the image decoders and encoders do not need it.

The homepage is `meta-pytorch.org/torchcodec/`, and the licence is BSD-3-Clause with a `LICENSE` file at the repository root.

The decoder API is index-based, with timestamps attached

The video decoding example is the clearest picture of the interface, and the shape of the returned data matters more than the call syntax.

from torchcodec.decoders import VideoDecoder

device = "cpu"  # or e.g. "cuda" !
decoder = VideoDecoder("path/to/video.mp4", device=device)

decoder.metadata
# VideoStreamMetadata:
#   num_frames: 250
#   duration_seconds: 10.0
#   bit_rate: 31315.0
#   codec: h264
#   average_fps: 25.0

The metadata object reports the frame count, duration, bit rate, codec and average frame rate, which is enough to compute sampling strategies without decoding anything.

The indexing API is deliberately Python-like. `decoder[0]` returns a uint8 tensor of shape `[C, H, W]`, a single frame, and `decoder[0 : -1 : 20]` returns a uint8 stacked tensor of shape `[N, C, H, W]`, which is the shape a model wants. The slice syntax works the way you would hope, including a step.

There are two additional entry points that matter for anything where wall-clock position is the thing you care about. `decoder.get_frames_at(indices=[2, 100])` returns a `FrameBatch` with the frame data plus `pts_seconds` and `duration_seconds` tensors, so you know the presentation timestamp of each frame you actually got. `decoder.get_frames_played_at(seconds=[0.5, 10.4])` does the reverse, giving you the frames that were on screen at particular times.

That distinction is the substance of the library. Frame index is not time. A variable frame rate video has no reliable frame-to-time mapping, and a training pipeline that samples by index will silently train on the wrong moments. Returning `pts_seconds` and `duration_seconds` alongside the data is the library making that distinction explicit at the API boundary rather than leaving it to the caller.

The device argument accepts `"cpu"` or, as the comment puts it, e.g. `"cuda"`.

Encoding is a stream-based API, not a call with a tensor

The encoder example shows a different shape, and it is worth understanding why.

from torchcodec.encoders import Encoder

encoder = Encoder()
video_stream = encoder.add_video(
    height=height, width=width, frame_rate=frame_rate,
)
audio_stream = encoder.add_audio(
    sample_rate=sample_rate, num_channels=num_channels,
)
with encoder.open_file("output.mp4"):
    video_stream.add_frames(frames_batch_0)
    audio_stream.add_samples(samples_batch_0)
    video_stream.add_frames(frames_batch_1)
    audio_stream.add_samples(samples_batch_1)

The encoder is built up in three steps. You create an `Encoder`, you add one or more streams to it with the shape and format parameters, and then you open a file and feed the streams batches. The `add_video` call takes height, width and frame rate, and `add_audio` takes sample rate and channel count.

The nesting of `add_frames` and `add_samples` inside the `open_file` context is the point. Video and audio are written as interleaved batches into a single output, so a caller that has two streams writes them in the order they should appear on the timeline rather than concatenating two separate files afterwards.

The image side is a different API again, and much simpler. The decoders and encoders are separate functions and classes:

from torchcodec.decoders import decode_image, decode_jpeg
from torchcodec.encoders import JpegEncoder

# JPEG, PNG, WebP, GIF, AVIF and HEIC, with the format detected automatically.
image = decode_image("path/to/image.jpg")  # uint8 tensor of shape [C, H, W]

# Or use the format-specific decoders, e.g. to decode JPEGs on GPU:
image = decode_jpeg("path/to/image.jpg", device="cuda")

# JPEG and PNG encoding. JPEGEncoder also supports CUDA encoding!
JpegEncoder(image).to_file("output.jpg")  # also .to_tensor() and .to_file_like()

`decode_image` detects the format automatically across JPEG, PNG, WebP, GIF, AVIF and HEIC. `decode_jpeg` is the format-specific variant, and the comment notes it is for decoding JPEGs on GPU. `JpegEncoder` writes to a file, to a tensor, or to a file-like object, and it supports CUDA encoding.

Installing means FFmpeg with shared libraries, on a supported version

The install section has two steps and one important caveat that catches people out.

The first step is FFmpeg. TorchCodec supports all major FFmpeg versions in [4, 9], and Linux distributions usually come with FFmpeg pre-installed. The caveat is that you need FFmpeg that comes with separate shared libraries, and the documentation says this is especially relevant for Windows users, where those releases are usually called the shared releases.

That requirement is not pedantic. A static FFmpeg build has no shared libraries to load, and the loader in a compiled wheel expects to find them. If you install FFmpeg from a source build with everything linked statically, TorchCodec will fail in a way that looks like a library bug rather than a packaging mismatch.

If FFmpeg is not already installed, or you need a more recent version, the documentation suggests conda:

conda install "ffmpeg"
# or
conda install "ffmpeg" -c conda-forge

The note that follows is the one to internalise. FFmpeg is an optional dependency. It is needed for the video and audio decoding and encoding, the `VideoDecoder`, `AudioDecoder`, `VideoEncoder` and `AudioEncoder` family, but the image decoders and encoders, `decode_image`, `decode_jpeg`, `JpegEncoder` and `PngEncoder`, do not require FFmpeg at all, so if you only need images you can skip the step.

The second step is the Python install, and it is short:

pip install torch torchcodec

On Linux x86 and aarch64 this installs CUDA-enabled wheels by default, matching the default behaviour of `pip install torch`, and the documentation notes those wheels should still work even if you have no GPU. On macOS and Windows it installs CPU-only wheels, and CPU wheels are available for Linux on x86_64 and aarch64, macOS and Windows.

CUDA support means NVDEC and NVENC hardware, and a matching wheel index

The CUDA section is where the practical constraints live, and there are two of them: hardware capability and wheel selection.

On CUDA GPUs, TorchCodec supports decoding and encoding of videos and jpeg images, and CUDA-enabled wheels are installed by default on Linux. For Windows you need to pass an index URL, and for video you need a GPU with NVDEC and NVENC hardware that supports the formats you want. The documentation points at Nvidia's GPU support matrix for the list.

The index URL selects the CUDA toolkit version, and it has to match your PyTorch install:

# This corresponds to CUDA Toolkit version 13.0.
pip install torch torchcodec --index-url=https://download.pytorch.org/whl/cu130

The FFmpeg build has to have the hardware codecs too, and there are two commands to check. `ffmpeg -decoders | grep -i nvidia` should show a line like `V..... h264_cuvid Nvidia CUVID H264 decoder (codec h264)`, and `ffmpeg -encoders | grep -i nvidia` should show `V....D h264_nvenc NVIDIA NVENC H.264 encoder (codec h264)`. A stock FFmpeg without the NVIDIA patches will show nothing for both.

There is also an end-to-end check that the FFmpeg libraries work with NVDEC correctly, by decoding a generated test video with hardware acceleration:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -f lavfi -i testsrc2=duration=1 -f null -

The CPU-only route is a different index URL, and it is worth knowing it exists even if you are on Linux where CUDA wheels are the default:

pip install torchcodec --index-url=https://download.pytorch.org/whl/cpu

The README also has an XPU section for Intel GPUs.

To try the decoder out without a video file, the README gives an FFmpeg command that generates one: `ffmpeg -f lavfi -i testsrc2=size=640x400:duration=10:rate=25 /tmp/output_video.mp4`.

A compiled wheel with a hand-rolled build and version handling

The packaging tells you what kind of project this is, and it is not a pure Python package.

The build backend is `scikit-build-core` with `pybind11`, and the project requires Python 3.10 or newer. The `wheel.packages` setting points at `src/torchcodec`, so the C++ extension lives under a `src/` layout and the Python package wraps it.

The version is dynamic, handled by a custom metadata plugin. The `pyproject.toml` points `provider` at `torchcodec_version` with `provider-path` set to `packaging`, which is described as restoring the version-handling logic that used to live in `setup.py`. The logic honours a `BUILD_VERSION` environment variable, set by the test-infra release and nightly jobs, and otherwise appends the short git sha to the contents of `version.txt` for dev builds. The generated `version.py` is written both into the install tree and into the source tree from the same template.

There is a detail in the build config that is worth reading, because it explains a class of packaging bug. `install.strip = false` is set explicitly, with a comment explaining that scikit-build-core strips by default but the post-build script inspects the `.so` symbol table with `objdump --syms` and greps for GLIBCXX to enforce manylinux GLIBCXX compatibility, and stripping leaves that check with no symbols to read. The comment also notes this matches the behaviour of the previous setuptools build, which did not strip.

The repository also has `benchmarks/`, `docs/`, `examples/`, `test/`, `packaging/`, a `CMakeLists.txt`, a `.clang-format`, a `.flake8`, a `mypy.ini` and a `pre-commit-config.yaml`. The examples directory is split into `decoding/`, `encoding/`, `low_level/` and `migration/`, with a `README.rst` alongside, and the `migration/` directory is a hint that there is a documented path from an older API to the current one.

The release history is regular. v0.15.0 came on 2026-07-15, v0.16.0 on 2026-08-13 with image decoding and encoding, and v0.17.0 on 2026-09-30 with low-level decoding APIs. The last push was on 2026-10-15 and the repository is not archived, so the project is actively developed, and the release notes in the tag titles show the surface area growing in a deliberate order, images first and then the low-level API.

Editorial conclusion

TorchCodec fits a PyTorch team training or serving models on video, audio or images that wants decoded frames as tensors with sensible metadata rather than a pipe from an external tool. It does not fit a workload that only needs images and wants to avoid a native dependency chain, since the image path still installs a compiled wheel, and it does not fit an environment where the FFmpeg version in the image is outside the supported range. Verify first that the FFmpeg build you have is a shared-library build in the supported major version range, since a static build or an out-of-range version is the most likely cause of an import-time failure, that your GPU has the NVDEC and NVENC hardware for the codecs you intend to decode and encode, and that the wheel index you use matches the CUDA toolkit version of your PyTorch install.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. meta-pytorch/torchcodec on GitHub
3. Project website
4. README
5. Releases