Running LLMs on Strix Halo about as fast as it'll go

This one's for the nerds! See my lament (and final acceptance) of the Strix Halo here.

Uninstall Ubuntu Rocm drivers if necessary

sudo apt remove rocm
sudo apt remove rocminfo

Install lastest Rocm drivers

cd Downloads/
wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/noble/amdgpu-install_7.2.3.70203-1_all.deb
sudo apt install ./amdgpu-install_7.2.3.70203-1_all.deb
amdgpu-install -y --usecase=rocm --no-dkms
sudo usermod -aG render,video $USER

Increase available GPU memory

sudo nano /etc/default/grub

Set:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash ttm.pages_limit=30720000 amdgpu.gttsize=120000 amd_iommu=off"

sudo update-grub
sudo reboot

In BIOS, change set memory to lowest possible (2GB on my machine)

Check:

sudo dmesg | grep "amdgpu.*memory"
rocm -v
rocminfo

Performance tuning

sudo udevadm control --reload-rules && sudo udevadm trigger
sudo apt update && sudo apt install tuned -y
sudo systemctl enable --now tuned
tuned-adm list | grep accelerator
sudo tuned-adm profile accelerator-performance
tuned-adm active

Rocm Nightly

export RELEASE_ID=20260515-25898286141 # Find the latest here: https://rocm.nightlies.amd.com/packages-multi-arch/deb/
sudo apt install -y ca-certificates
echo "deb [trusted=yes] https://rocm.nightlies.amd.com/packages-multi-arch/deb/${RELEASE_ID} stable main"   | sudo tee /etc/apt/sources.list.d/rocm-multiarch-nightly.list
sudo apt install amdrocm
sudo apt update
sudo apt install amdrocm

Distrobox

amd-strix-halo-toolboxes
StrixHalo_Toolboxes

sudo apt install distrobox
distrobox create llama-rocm-7.2.3 --image docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-7.2.3 -- --device /dev/dri --device /dev/kfd --group-add video --group-add render --group-add sudo --security-opt seccomp=unconfined
distrobox enter llama-rocm-7.2.3
llama-cli --list-devices
curl -LsSf https://hf.co/cli/install.sh | bash
exit
distrobox enter llama-rocm-7.2.3

Install HF, exit and go back in to Distrobox

curl -LsSf https://hf.co/cli/install.sh | bash
exit
distrobox enter llama-rocm-7.2.3

Download a model (some examples - use "high performance" if you don't care about anyone else on your network)

HF_TOKEN=hf_xxxx HF_XET_HIGH_PERFORMANCE=1 hf download unsloth/Qwen3.5-35B-A3B-GGUF BF16/Qwen3.5-35B-A3B-BF16-00001-of-00002.gguf --local-dir models/qwen3.5-35b-a3b
HF_TOKEN=hf_xxx HF_XET_HIGH_PERFORMANCE=1 hf download unsloth/Qwen3.5-35B-A3B-GGUF BF16/Qwen3.5-35B-A3B-BF16-00002-of-00002.gguf --local-dir models/qwen3.5-35b-a3b # If there are two parts
hf download unsloth/Qwen3.5-35B-A3B-MTP-GGUF Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf --local-dir models/qwen3.5-35b-a3b-mtp # unsloth example

Run it

llama-cli --no-mmap -ngl 999 -fa 1 -m models/unsloth-qwen3.5-35b-a3b-mtp/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf  -p "Write a Strix Halo toolkit haiku."