Install Jellyfin on Proxmox Using a Debian 12 VM + Intel UHD 600 iGPU Passthrough for Hardware Acceleration

I wanted a simple, low power media server that I could run 24 by 7 in my homelab without buying a full NAS. So I used a Dell Wyse 5070, installed Proxmox, and deployed Jellyfin inside a Debian VM using Docker. The key part was passing through the Intel UHD 600 iGPU to the VM so Jellyfin could use VAAPI for hardware acceleration. In this post, I will walk through the exact steps and the small details that usually cause problems, so you can replicate the same setup quickly.

On the ProxMox Host

Setup the VM 

Prepare a VM on proxmox, give it a static IP address. In this case I am using Debian 12 with very minimal setup. Some tools such as sudo, htop, openssh-server and also Debian repository may need to manually configured.

apt install sudo htop openssh-server

Add to /etc/apt/sources.list just in case you need it.

deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

Enable IOMMU on Proxmox host

IOMMU is a CPU and chipset feature that lets a VM access PCIe devices through a protected memory mapping layer.

Without IOMMU, Proxmox usually cannot map the iGPU (00:02.0) into a VM safely, so PCI passthrough often fails.

The iommu=pt option improves performance by using pass through mappings where possible.

Edit grub:
nano /etc/default/grub

Find GRUB_CMDLINE_LINUX_DEFAULT= then add
intel_iommu=on iommu=pt

Update grub and then reboot:

update-grub
reboot

Load VFIO modules

VFIO is the Linux kernel framework that takes control of a PCIe device from the host and presents it to a VM.

Key modules:

• vfio core framework
• vfio_pci binds to the PCI device you passthrough
• vfio_iommu_type1 provides IOMMU based memory isolation
• vfio_virqfd helps handle interrupts efficiently in the VM

If these are not loaded, the device may remain owned by the host or the VM may fail to start when passthrough is enabled.

On the Proxmox host, create a file nano /etc/modules-load.d/vfio.conf
with the following content:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

then reboot again

Blacklist i915 (prevent iGPU taken by Proxmox host)

Create a file nano /etc/modprobe.d/blacklist-intel-gpu.conf 

then add the following line

update initramfs then reboot again

update-initramfs -u

reboot

Assign PCI device to Debian VM

Check the hardware availability

lspci | grep -i -E "vga|display"

add the PCE device

reboot the VM then check PCI device on the VM

On Debian VM

Install VAAPI on the VM

sudo apt update

sudo apt install -y vainfo intel-media-va-driver i965-va-driver

run this command to verify

vainfo

Install Docker

sudo apt update

sudo apt install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(. /etc/os-release && echo $VERSION_CODENAME) stable" \

| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

sudo usermod -aG docker $USER

Prepare requirements for Jellyfin

sudo mkdir -p /opt/jellyfin/{config,cache}

sudo mkdir -p /data/media

cd /opt/jellyfin

sudo nano compose.yml

services:

  jellyfin:

    image: jellyfin/jellyfin:latest

    container_name: jellyfin

    network_mode: host

    environment:

      - TZ=Asia/Makassar

    volumes:

      - /opt/jellyfin/config:/config

      - /opt/jellyfin/cache:/cache

      - /data/media:/media

    devices:

      - /dev/dri:/dev/dri

    restart: unless-stopped

Start jellyfin

docker compose up -d (yes, i forgot to add -d if you refer to the screenshot)

docker ps

You may also install this tools to verify transcoding later

sudo apt install -y intel-gpu-tools

On Client

Accessing Jellyfin for the First Time

I actually re-run the jellyfin container from standard user instead of root.

Open http://IP-address:8096 from web browser, do the initial setup on jellyfin

Specify the admin account for jellyfin

Speficy the media folder, you may also want to setup the metada preferences.

Complete by clicking Finish then login using the jellyfin admin account, later you can create different account to setup on TV or the clients.

Go to Dashboard > Playback > Transcoding, set the hardware accelleration to use the iGPU.

Set the hardware accelleration options from none to intel QSV. You can set the other parameter as needed. Then click save

I have put some movie files under /data/media. If it not showing on jellyfin, then you need to rescan the library

Some Verification Steps

You can check and verify transcoding procces using the following command. I open 3 stream (Chrome, Edge, Jellyfin Desktop), there are 3 ffmpeg proccess appear there

intel_gpu_top

Serving 3 stream with transcoding, the 2 vCPU looks very busy. The experience from client side were not so bad, there are a little buffering at the beginning of transcoding process

But with all stream using direct play mode, the CPU looks very chill

I think that's all for today! The improvement for this setup is on how to manage the media folder, especially if I am using external drive and I want to add more external drive in the future, the scenario of using NFS might need to be considered aswell.