AV1 Video Conversion Pipeline

A comprehensive, production-ready pipeline for converting video libraries to AV1 codec with quality verification, automatic HDR handling, and intelligent duplicate detection.

⚠️ Hardware Requirements

This pipeline requires an Intel CPU or GPU with AV1 hardware encoding support.

Minimum Requirements:

• For AV1 Encoding:
  • Intel Arc Series GPU (A310, A380, A750, A770, B570, B580)
  • Intel Core Ultra processors (1st gen, Meteor Lake, 2023+)
  • Intel 14th Gen Core Mobile processors (2023+)

📖 Official Intel Documentation: Intel Media Capabilities

Overview

This pipeline automates the conversion of large video collections to AV1, the modern successor to H.264/H.265. It handles HDR content (including Dolby Vision and HDR10+), performs quality verification before deleting originals, and processes files individually for better progress tracking.

Key Features

• 🎯 Smart Processing: Checks for existing AV1 versions before converting
• 🔍 Quality Verification: PSNR-based quality checks before deleting originals
• 🌈 HDR Support: Automatic detection from file metadata and intelligent tonemapping of Dolby Vision/HDR10+ to static HDR10
• 🚀 Hardware Acceleration: Intel QSV (Quick Sync Video) AV1 encoding
• 📊 Progress Tracking: Process files one at a time with detailed logging
• 🔧 Modular Design: Individual scripts can be used standalone or as part of pipeline

Architecture

The pipeline consists of two main stages:

Stage 1: Normalize Filenames (Batch)

fix-av1-filenames
└── Renames misnamed AV1 files with correct codec and HDR tags

Stage 2: Process Files (Individual)

For each non-AV1 file:
├── Check if AV1 version exists?
│   ├── Yes → verify quality → delete source (if quality good)
│   └── No → convert to AV1 → verify quality → delete source
│
├── convert-to-av1: Hardware-accelerated AV1 encoding
├── verify-av1-quality: Quality checks (PSNR, duration, resolution)
└── Automatic cleanup of originals after verification

System Requirements

Hardware (Required)

• Intel GPU with AV1 encoding support (see Hardware Requirements section above)

Software

• Docker with a running FFmpeg container
  • Recommended: linuxserver/ffmpeg (recent version)
  • Container must include:
    • FFmpeg with Intel QSV support
    • libplacebo for HDR tonemapping
    • Vulkan support
  • Container name: ffmpeg (or set CTN environment variable)
• Bash 4.0+
• Standard Unix tools: bc, jq, numfmt, stat

Docker Container Setup

Example setup for the linuxserver/ffmpeg container:

# Pull the image
docker pull linuxserver/ffmpeg:latest

# Run the container with hardware acceleration
docker run -d \
  --name=ffmpeg \
  --device=/dev/dri:/dev/dri \
  -v /path/to/media:/media \
  --entrypoint /bin/tail \
  linuxserver/ffmpeg:latest \
  -f /dev/null

# Verify hardware acceleration is available
docker exec ffmpeg ffmpeg -hide_banner -encoders | grep av1_qsv
# Should output: av1_qsv

Important: The container needs access to /dev/dri for Intel QSV hardware acceleration and must have Vulkan support for libplacebo tonemapping.

Usage

Basic Usage

Process a single directory:

./av1-pipeline /path/to/movies

Process multiple directories:

./av1-pipeline /path/to/movies /path/to/tvshows /path/to/documentaries

Dry Run Mode

Preview what would be done without making changes:

DRY_RUN=1 ./av1-pipeline /path/to/movies

Skip Quality Verification

Process faster by skipping PSNR verification (not recommended for archival):

SKIP_VERIFY=1 ./av1-pipeline /path/to/movies

Skip Stages

Skip the rename stage if already done:

SKIP_RENAME=1 ./av1-pipeline /path/to/movies

Only run rename stage:

SKIP_PROCESS=1 ./av1-pipeline /path/to/movies

Individual Scripts

convert-to-av1 -- Core Conversion Script

Converts video files to AV1 with automatic HDR detection and tonemapping.

./convert-to-av1 input.mkv
# Output: input [AV1].mkv (or input [AV1] [HDR10].mkv for HDR content)

Configuration:

# Quality settings (19-23 recommended for archival)
QUALITY=22 ./convert-to-av1 movie.mkv

# Encoding speed (slower = better compression)
PRESET=veryslow ./convert-to-av1 movie.mkv  # slowest, best quality
PRESET=slow ./convert-to-av1 movie.mkv      # balanced
PRESET=medium ./convert-to-av1 movie.mkv    # faster, larger files

# Debug mode
DEBUG=1 ./convert-to-av1 movie.mkv

process-single-file -- Smart File Processor

Checks for existing AV1 version, converts if needed, verifies quality.

./process-single-file movie.mkv

Workflow:

1. Check if movie [AV1].mkv already exists
2. If exists: verify quality → delete source (if good)
3. If not: convert → verify → delete source

verify-av1-quality -- Quality Verification

Verifies encoding quality before deleting originals.

./verify-av1-quality original.mkv "encoded.mkv"

Checks performed:

• ✅ Video codec is AV1
• ✅ Duration matches (within 0.1s tolerance)
• ✅ Resolution unchanged
• ✅ PSNR ≥ 40 dB (configurable)
• ✅ HDR metadata preserved (for HDR content)

Configuration:

# Require higher quality threshold
MIN_PSNR=45 ./verify-av1-quality original.mkv encoded.mkv

# Allow larger duration difference
MAX_DURATION_DIFF=0.5 ./verify-av1-quality original.mkv encoded.mkv

fix-av1-filenames -- Filename Normalizer

Renames AV1 files that have incorrect codec names or HDR tags.

./fix-av1-filenames /path/to/directory

Examples:

• Movie [HEVC].mkv (but codec is AV1) → Movie [AV1].mkv
• Movie [AV1] [DV HDR10].mkv (but no DV metadata) → Movie [AV1] [HDR10].mkv
• Movie [x265] [HDR10+].mkv (tonemapped to static HDR10) → Movie [AV1] [HDR10].mkv

remove-duplicate-sources =- Duplicate Finder

Finds source files where AV1 version already exists and removes them after verification.

./remove-duplicate-sources /path/to/directory

Useful for cleanup after interrupted pipeline runs.

convert-and-replace -- Batch Converter

Legacy batch conversion script (kept for compatibility).

./convert-and-replace movie1.mkv movie2.mkv movie3.mkv

Configuration

Environment Variables

Variable	Default	Description
DRY_RUN	0	Set to 1 for dry-run mode (no changes)
SKIP_VERIFY	0	Set to 1 to skip quality verification before deletion
QUALITY	22	AV1 quality (19-23 for archival, lower = better)
PRESET	veryslow	Encoding preset (veryslow/slow/medium/fast)
LOOKAHEAD	40	Lookahead frames for rate control
MIN_PSNR	40.0	Minimum acceptable PSNR in dB
MAX_DURATION_DIFF	0.1	Maximum duration difference in seconds
CTN	ffmpeg	Docker container name
LOG_DIR	/mnt/user/downloads/av1	Log file directory

Quality Guidelines

QUALITY values:

• 19-21: Near-transparent quality (large files)
• 22-23: High quality archival (recommended)
• 24-26: Good quality, smaller files
• 27+: Noticeable quality loss

PSNR thresholds:

• 50+ dB: Excellent (nearly transparent)
• 45+ dB: Very high quality
• 40+ dB: High quality (default minimum)
• 35-40 dB: Acceptable quality
• <35 dB: Noticeable artifacts

Output

File Naming Convention

The pipeline automatically generates appropriate filenames:

SDR content:

Movie (2023) [1080p].mkv → Movie (2023) [1080p] [AV1].mkv

HDR10 content:

Movie [HEVC] [HDR10].mkv → Movie [AV1] [HDR10].mkv

Dolby Vision (auto-detected and tonemapped):

Movie [HEVC] [DV HDR10].mkv → Movie [AV1] [HDR10].mkv
Movie [HEVC].mkv (with DV metadata) → Movie [AV1] [HDR10].mkv

HDR10+ (auto-detected and tonemapped):

Movie [HDR10+].mkv → Movie [AV1] [HDR10].mkv
Movie [HEVC].mkv (with HDR10+ metadata) → Movie [AV1] [HDR10].mkv

Note: HDR type is detected from video stream metadata, not filename. Filenames are corrected based on actual content.

Log Files

All operations are logged to /mnt/user/downloads/av1/:

• converted_<timestamp>.log -- Successfully converted files
• deleted_<timestamp>.log -- Deleted original files
• skipped_<timestamp>.log -- Files skipped (with reasons)
• encode_errors_<timestamp>.log -- Encoding errors
• encode_failed_<timestamp>.log -- Failed conversions
• encode_success_<timestamp>.log -- Successful encodings with size stats

HDR Handling

All HDR detection is automatic based on video stream metadata, not filename. The pipeline analyzes color transfer characteristics, dynamic metadata, and side data to determine the correct HDR type.

Dolby Vision & HDR10+

Dynamic HDR formats (Dolby Vision, HDR10+) are automatically tonemapped to static HDR10 using libplacebo.

What happens:

1. ✅ Pipeline detects Dolby Vision/HDR10+ metadata in video stream
2. ✅ libplacebo uses the dynamic metadata (including DV RPU) to intelligently tonemap to HDR10
3. ✅ Output is static HDR10
4. ⚠️ Dolby Vision dynamic metadata is removed (not supported in av1_sqv)
5. ✅ Resulting file is universally compatible with HDR10 displays

Benefits:

• Wide compatibility with all HDR10-capable displays
• Preserves HDR appearance using content's own metadata for mapping
• Significantly smaller file sizes than original
• No licensing requirements (DV requires licensing)

Trade-offs:

• Dynamic metadata is lost (scene-by-scene adjustments)
• Static HDR10 is still significantly better than SDR

Static HDR10

Static HDR10 content is detected automatically and passed through with all metadata preserved:

• Color primaries (BT.2020)
• Transfer characteristics (PQ/SMPTE 2084)
• Mastering display metadata
• Content light levels (MaxCLL/MaxFALL)

No tonemapping is applied to static HDR10 content.

SDR Content

SDR content is automatically detected and encoded as-is with original color space preserved (typically BT.709 or BT.601).

HDR Type Detection

• Color transfer characteristic (color_trc)
• Color primaries and color space
• Dolby Vision side data (RPU)
• HDR10+ dynamic metadata (SMPTE2094-40)

"Quality verification failed"

The encoded file didn't meet quality thresholds. Check:

1. PSNR value in logs (should be ≥40 dB)
2. Duration difference (should be <0.1s)
3. Try lowering QUALITY value (e.g., QUALITY=20)

"Black frames in output"

This can happen with certain codecs (VC1, MPEG2) when using hardware decode. The script should handle this automatically by using software decode for these codecs. If you encounter black frames, please report the source codec.

Files not being deleted

Check logs in /mnt/user/downloads/av1/ for reasons. Common causes:

• Quality verification failed
• Permission issues
• File in use

Remote Usage

If your encoding server is remote (e.g., tower):

1. Ensure the linuxserver/ffmpeg container is running on the remote server:

ssh tower docker ps | grep ffmpeg

2. Copy scripts to server:

scp av1-pipeline process-single-file convert-to-av1 \
    verify-av1-quality av1-common.sh tower:/tmp/

3. SSH and run:

ssh tower
cd /tmp
chmod +x av1-pipeline process-single-file convert-to-av1 verify-av1-quality
./av1-pipeline /path/to/movies

License

MIT License. See LICENSE file for details.

Credits

Built with:

• FFmpeg: The backbone of video processing
• Intel QSV: Hardware-accelerated encoding
• libplacebo: High-quality HDR tonemapping
• Docker: Containerized ffmpeg environment