VAAPI Hardware Encoding on Linux — Accelerate FFmpeg with Intel and AMD GPUs

What You Will Learn

• What VAAPI is and which GPUs can use it
• Prerequisites for enabling VAAPI on Linux
• Basic usage of h264_vaapi, hevc_vaapi, and av1_vaapi
• How to build a GPU decode + encode pipeline
• Common errors and their fixes

Tested with: FFmpeg 6.1 (Linux VAAPI environment)
Target OS: Linux (Intel iGPU / AMD GPU supported)

What Is VAAPI

VAAPI (Video Acceleration API) is the standard hardware video acceleration interface on Linux. It is available on Intel integrated GPUs, AMD GPUs, and some NVIDIA GPUs. Because it calls the GPU’s encoder directly, it is dramatically faster than CPU software encoding.

Prerequisites

Installing the Required Packages

Ubuntu / Debian:

※ This command requires a VAAPI environment
sudo apt install vainfo libva-dev intel-media-va-driver  # Intel
sudo apt install vainfo libva-dev mesa-va-drivers         # AMD

Arch Linux:

※ This command requires a VAAPI environment
sudo pacman -S intel-media-driver libva   # Intel
sudo pacman -S mesa libva                 # AMD

Verifying the VAAPI Device

※ This command requires a VAAPI environment
vainfo

If /dev/dri/renderD128 appears and the supported entry points (such as VAEntrypointEncSlice) are listed, you are ready to go.

Basic h264_vaapi Command

※ This command requires a VAAPI (Linux) environment
ffmpeg -vaapi_device /dev/dri/renderD128 -i input.mp4 \
  -vf 'format=nv12,hwupload' \
  -c:v h264_vaapi -qp 23 \
  -c:a aac -b:a 128k output_vaapi.mp4

Key points:

• -vaapi_device /dev/dri/renderD128: specifies the VAAPI device (you may have several, such as /dev/dri/renderD129)
• -vf 'format=nv12,hwupload': required to upload frames into GPU memory
• -qp 23: quality parameter (0–51; lower means higher quality)

hevc_vaapi (H.265)

※ This command requires a VAAPI (Linux) environment
ffmpeg -vaapi_device /dev/dri/renderD128 -i input.mp4 \
  -vf 'format=nv12,hwupload' \
  -c:v hevc_vaapi -qp 28 \
  -c:a aac -b:a 128k output_hevc_vaapi.mp4

av1_vaapi (AV1) — Intel Arc / Xe and Later

※ This command requires a VAAPI (Linux / Intel Arc) environment
ffmpeg -vaapi_device /dev/dri/renderD128 -i input.mp4 \
  -vf 'format=nv12,hwupload' \
  -c:v av1_vaapi -rc_mode CQP -global_quality 30 \
  -c:a aac -b:a 128k output_av1_vaapi.mp4

Note: av1_vaapi does not accept the -qp option that h264_vaapi/hevc_vaapi use (FFmpeg reports “Codec AVOption qp has not been used”). Use -rc_mode CQP -global_quality 30 for constant-quality, or a bitrate target such as -b:v 4M. AV1 is supported on Intel Arc (Alchemist) and later, as well as AMD RDNA3 and later GPUs.

Full-GPU Pipeline: VAAPI Decode + Encode

By performing decoding on the GPU (VAAPI) as well, you can keep CPU usage almost at zero:

※ This command requires a VAAPI (Linux) environment
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
  -hwaccel_output_format vaapi \
  -i input.mp4 \
  -c:v h264_vaapi -qp 23 \
  -c:a copy output_fullgpu.mp4

With this approach, no frame transfer from GPU to CPU is needed.

Bitrate-Targeted Encoding

※ This command requires a VAAPI (Linux) environment
ffmpeg -vaapi_device /dev/dri/renderD128 -i input.mp4 \
  -vf 'format=nv12,hwupload' \
  -c:v h264_vaapi -rc_mode CBR -b:v 4M \
  -c:a aac -b:a 128k output_cbr.mp4

Common Errors and Fixes

Cannot open library: libva.so.2

• libva is not installed
• Ubuntu: sudo apt install libva2

Device creation failed (/dev/dri/renderD128 not found)

• Check the device with ls /dev/dri/
• Verify that the kernel module is loaded (Intel: i915, AMD: amdgpu)

Error without format=nv12,hwupload

• VAAPI encoders require frames in GPU memory, so hwupload is mandatory

Provided pixel format nv12 is not supported

• Some GPUs only accept the p010le (10-bit) format
• Try changing -vf 'format=nv12,hwupload' to -vf 'format=p010le,hwupload'

VAAPI vs. NVENC

Speed and Quality Ballparks (Typical Ranges)

The figures below are typical ranges observed in public benchmarks. Note they vary widely with generation and driver (on Intel, the legacy i965 versus the newer iHD/media-driver).

• Encode speed: An Intel UHD/Iris Xe iGPU often encodes 1080p H.264 at roughly 4–8× realtime; discrete GPUs like Intel Arc or AMD RDNA can run higher. That is typically several times faster than CPU libx264 -preset medium (around 1.5–3× realtime). Exact numbers depend on GPU, driver, content, preset, and resolution.
• Quality trade-off: At a matched quality target, VAAPI generally needs a higher bitrate than libx264 to reach the same VMAF; the exact gap depends on generation and content. Intel’s newer media-driver (iHD) carries QSV-grade tuning and is more consistent than the legacy i965.
• AV1: AV1 encoding on Intel Arc / AMD RDNA3 can deliver comparable quality at a noticeably smaller size than H.264, a meaningful win for streaming.

Common Pitfalls

Permission error on /dev/dri/renderD128 (Permission denied)

• Symptom: vainfo or an encode fails with failed to open /dev/dri/renderD128: Permission denied.
• Cause: The running user is not in the render (or video) group. This is especially common inside containers.
• Fix: Add the user with sudo usermod -aG render $USER and log back in. In Docker, pass --device /dev/dri and use --group-add with the host’s render GID.

Impossible to convert between the formats when format=nv12,hwupload is missing

• Symptom: Specifying -c:v h264_vaapi without the filter stops with a conversion error.
• Cause: VAAPI encoders require surfaces in GPU memory, so CPU-side frames can’t be passed directly.
• Fix: Always insert -vf 'format=nv12,hwupload'. In a full-GPU pipeline (-hwaccel vaapi -hwaccel_output_format vaapi) frames are already on the GPU, so hwupload is not needed.

Picking the wrong device on multi-GPU systems

• Symptom: You want the iGPU but the discrete GPU is selected, or vice versa.
• Cause: Which of /dev/dri/renderD128 and renderD129 maps to which GPU varies by system.
• Fix: Run ls -l /dev/dri/by-path/ to map PCI addresses to devices, then pass the intended GPU’s renderDxxx explicitly via -vaapi_device.

nv12 is not supported on 10-bit input

• Symptom: HDR or 10-bit sources fail with Provided pixel format nv12 is not supported.
• Cause: You’re trying to receive 10-bit material in 8-bit nv12.
• Fix: Change to -vf 'format=p010le,hwupload' and specify a 10-bit-capable output profile (such as -profile:v main10).

FAQ

How is VAAPI different from QSV (Quick Sync)?

Both drive Intel GPU hardware encoding, but VAAPI is the standard Linux API while QSV (h264_qsv) goes through Intel’s own SDK. Use VAAPI for portable Linux use; reach for QSV when you want to extract Intel-specific features or also target Windows.

Should I control quality with -qp or -rc_mode?

Use -qp for fixed-quality output (lower is higher quality; around 23 is a good start). Use rate control such as -rc_mode CBR -b:v 4M when you need consistent bitrate for streaming or size budgets. To balance both, combine -rc_mode VBR -b:v with -maxrate.

Can AMD GPUs use VAAPI too?

Yes. Install mesa-va-drivers (Mesa Gallium) and, with the kernel amdgpu module enabled, h264_vaapi/hevc_vaapi work. AV1 encoding requires RDNA3 or later.

Quality looks worse than my CPU encode

Even at the same fixed-quality value, VAAPI is slightly less efficient than libx264, so at equal size it can look softer. Lower -qp by 1–2 or raise the bitrate by 10–20% to close the perceived gap.

How do I use VAAPI inside a Docker container?

Pass the device with docker run --device /dev/dri ... and add the container user to the host’s render group GID. If vainfo runs correctly inside the container, you’re ready.

Related Articles

• NVENC (NVIDIA GPU) Hardware Encoding
• VideoToolbox (macOS) Hardware Encoding
• Compress Video — CRF and Target Bitrate

Primary sources: trac.ffmpeg.org/wiki/HWAccelIntro / ffmpeg.org/ffmpeg-codecs.html