Setup 🔗

From [Github] - AMD Strix Halo Llama.cpp Toolboxes:

• setup kernel to allow dynamic allocation of RAM to the GPU:

  add amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856" to your grub kernel line:

  sudo vi /etc/default/grub

  # general: amdgpu.gttsize=MB*1024 ttm.pages_limit=MB*262144
  # e.g.:
  # 104G: amdgpu.gttsize=106496 ttm.pages_limit=27262976
  # 120G: amdgpu.gttsize=122880 ttm.pages_limit=31457280
  GRUB_CMDLINE_LINUX="rd.luks.uuid=<partition-id> rhgb quiet \
      amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856" # 124G

  run sudo grub2-mkconfig to activate new config

  • more info: [Github] - Framework-strix-halo-llm-setup

• install and run nvtop to see available RAM/VRAM

Build Llama.cpp 🔗

sudo dnf install -y \
    glslang glslang-devel glslang-tools glslc \
    libcurl-devel \
    spirv-tools-devel spirv-headers-devel

git clone https://github.com/ggerganov/llama.cpp.git
( cd llama.cpp;
cmake -B build \
  -DGGML_VULKAN=ON \
  -DGGML_NATIVE=ON \
  -DCMAKE_BUILD_TYPE=Release \
  . \
  &&
  time make -C build -j \
)

llama.cpp/build/bin/llama-server  \
    --alias "llama.cpp" \
    --model "models/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf" \
    --kv-offload \
    --gpu-layers all \
    --no-mmap \
    --flash-attn on \
    --ctx-size "131072" \
    --temp "0.6" \
    --top-p "0.95" \
    --top-k "20" \
    --min-p "0.00" \
    --presence-penalty "0.0" \
    --repeat-penalty "1.0" \
    --batch-size "4096" \
    --ubatch-size "4096" \
    --reasoning off \
    --host 0.0.0.0 \
    --port 9000 \
    --threads 16

Models 🔗

• LLM Stats
• LLM Stats - SWE-Bench Verified
• Local LLMs on Strix Halo 128GB Shared Ram: My Tests
• AMD Ryzen AI MAX+ 395 “Strix Halo” — Benchmark Grid

Qwen3.6-35B-A3B 🔗

[huggingface/unsloth] - Qwen3.6-35B-A3B-GGUF

wget https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf

params: temp=1.0, top_p=0.95, 200K

links

• Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7