ffmpeg-rockchip, hardware transcoding for the RK3588 family

A vendor fork of FFmpeg for one SoC family

The project aims to provide a full hardware transcoding pipeline in the FFmpeg CLI for Rockchip platforms that support MPP, the Media Process Platform, and RGA, 2D Raster Graphic Acceleration. It includes hardware decoders, encoders and filters, and a typical target platform is a device based on the RK3588 or RK3588s.

The scope is worth stating plainly, because it is narrow. Everything this fork adds is Rockchip specific, and a build of it on a machine without a Rockchip VPU is just FFmpeg with extra code compiled out. That is not a criticism so much as a description of what a hardware enablement fork is. The value is that for the target hardware, the whole path from container to compressed output runs on the dedicated blocks rather than on the CPU.

The status is also stated in the disclaimer. This project is an UNOFFICIAL fork of FFmpeg and is NOT the property of Rockchip Electronics Co., Ltd. And the MPP and RGA components used are compatible with FFmpeg by enabling the LGPLv3 license, with the configure flag given as ./configure --enable-version3.

That last point is a licensing detail with a practical consequence. FFmpeg's default build is LGPLv2.1 with optional GPL components, and the MPP and RGA code in this fork requires the version3 option, so a build that has to stay on the older LGPL terms cannot include the hardware support. If your distribution ships a fixed FFmpeg licence configuration, that is the first thing to check.

The upstream FFmpeg README is reproduced below the project header, which is the normal way to present a fork: the library breakdown, with libavcodec providing the codec implementations, libavformat handling streaming protocols and container formats, libavutil for hashers and utility functions, libavfilter for the directed graph of filters, libavdevice for capture and playback abstraction, libswresample for audio mixing and resampling, and libswscale for colour conversion and scaling.

The codec and filter surface this fork adds

The codec and filter list is what you actually run, and it is where the hardware support shows itself.

The decoders are all suffixed rkmpp, and they cover a wide range: av1_rkmpp, h263_rkmpp, h264_rkmpp, hevc_rkmpp, mjpeg_rkmpp, mpeg1_rkmpp, mpeg2_rkmpp, mpeg4_rkmpp, vp8_rkmpp and vp9_rkmpp. In the ffmpeg listing they appear alongside the software codecs, and the names follow the pattern of the software codecs with the rkmpp suffix.

The encoders are narrower: h264_rkmpp, hevc_rkmpp and mjpeg_rkmpp. So on the decode side the fork is broad, and on the encode side it is the two codecs that matter in practice for a transcode pipeline, H.264 and HEVC.

The filters are the RGA ones, and there are three. scale_rkrga is the video resizer and format converter, vpp_rkrga is the video post-process doing scale, crop and transpose, and overlay_rkrga is the video compositor for blending two images. The README lists their input and output in the usual ffmpeg filter notation, with the compositor taking two video inputs and the other two taking one.

What the hardware blocks add beyond a plain software path is listed in the Highlights section, and it is the more interesting part of the project than the codec names. The MPP decoders support up to 8K 10-bit H.264, HEVC, VP9 and AV1 decoding, and they can produce AFBC, which is ARM Frame Buffer Compression, image. They support de-interlace using IEP, the Image Enhancement Processor, and they support allocator half-internal and pure-external modes. The MPP encoders support up to 8K H.264 and HEVC encoding, async encoding which is frame-parallel, and consuming AFBC image.

The RGA filters support image scaling and pixel format conversion, cropping, transposing, blending two images, async operation, and producing and consuming AFBC image. And the summary line for all of it is zero-copy DMA in the above stages, which is the property that makes the whole thing coherent.

Why AFBC and async are the two features to understand

Two of the highlights carry most of the technical weight, and both are about the boundary between the CPU and the hardware rather than about the hardware itself.

AFBC is ARM Frame Buffer Compression, a compressed surface format that the hardware blocks can both produce and consume. The significance is the round trip. If the decoder can output AFBC and the RGA filters can consume and produce AFBC, then a frame can go from decode to scaling to compositing without ever being expanded into a linear pixel buffer in system memory. Each expansion and recompression is a memory bandwidth cost and a copy, and on a device with a modest memory bandwidth that is often the actual bottleneck rather than the compute.

So the pipeline where the decoder writes AFBC, the RGA filter reads AFBC and writes AFBC, and the encoder consumes AFBC is a pipeline where the frames stay compressed for the whole trip. That is what the two separate mentions of AFBC, one on the decoder side and one on the filter and encoder side, are describing.

Async encoding, which the README calls AKA frame-parallel, changes the encoder's relationship to the input. Rather than the encoder waiting for a frame to be submitted and then returning a result, multiple frames are in flight at once. The same applies to the RGA filters, which support async operation. On a pipeline with several stages, a synchronous stage forces every other stage to wait for it, and the async design is what stops a slow conversion from stalling the whole chain.

The de-interlace support through IEP is the other feature worth knowing, because it is not something the ffmpeg CLI would otherwise reach for on this hardware, and doing it on the Image Enhancement Processor rather than in a software filter is the difference between a usable frame rate and an unusable one.

The MPP decoders' allocator modes are worth a mention too. Half-internal means the decoder allocates part of its own buffers and part is supplied by the caller, and pure-external means the caller supplies everything. The external mode is what makes zero-copy possible, since the caller is the one who owns and maps the memory.

What has to be true before it runs

The Important section of the README is short and consists of three requirements, and each of them is a real obstacle.

First, a Rockchip BSP or vendor kernel is necessary, and 5.10 and 6.1 are the two tested versions. This is the point where a lot of people stop, because a mainline kernel does not carry the Rockchip VPU driver stack this depends on.

Second, for the supported maximum resolution and FPS you should refer to the datasheet or TRM. That is a sensible instruction, because the 8K figures in the highlights describe what the hardware blocks are capable of, not what any particular device sustains with a real pipeline.

Third, the user MUST be granted permission to access the device files, and the README lists them by what they are used for:

# DRM allocator
/dev/dri

# DMA_HEAP allocator
/dev/dma_heap

# RGA filters
/dev/rga

# MPP codecs
/dev/mpp_service

# Optional, for compatibility with older kernels and socs
/dev/iep
/dev/mpp-service
/dev/vpu_service
/dev/vpu-service
/dev/hevc_service
/dev/hevc-service
/dev/rkvdec
/dev/rkvenc
/dev/vepu
/dev/h265e

That list is a good map of the driver stack. The DRM allocator and the DMA_HEAP allocator are what the zero-copy paths need, since both are how the kernel hands out memory buffers to user space. The RGA node is the 2D accelerator, and the MPP service node is the media process platform. The optional block, marked as being for compatibility with older kernels and SoCs, lists the older naming conventions that the vendor drivers used before they were unified, with both underscore and hyphen variants of mpp-service, vpu-service and hevc-service, plus the rkvdec, rkvenc, vepu and h265e device names.

Separately, the documentation lives on the project Wiki page rather than in the repository, and the README simply points there for how to use it.

One more thing to note is the Todo section, which is short. Support for the MPP VP8 video encoder is listed, followed by an ellipsis. So the encoder side of VP8 is not there, even though the VP8 decoder is.

Editorial conclusion

ffmpeg-rockchip suits someone running transcoding workloads on Rockchip hardware where the CPU cannot keep up, since the MPP and RGA offload covers the decode, encode and the scaling path in one pipeline, and the asynchronous encoder and zero-copy DMA are what make the hardware path worthwhile rather than decorative. It is the wrong choice on any other SoC, since every decoder, encoder and filter here is Rockchip specific, and it is the wrong choice if you need a stock FFmpeg build, because this is a fork you compile yourself. Before building it, confirm your kernel is a Rockchip BSP with 5.10 or 6.1, check that the user running the transcode can reach the DRM, DMA heap, RGA and MPP device nodes, and read the configure step for enabling the LGPLv3 build that the MPP and RGA components require.

Frequently asked questions

Official sources

1. Issues
2. nyanmisaka/ffmpeg-rockchip on GitHub
3. README