← All guides AI Release
RU Subscribe to @ai_release1

How to Run a Local LLM Server on AMD Strix Halo: HALOGEN and Qwen on Fedora 43

Updated: 25.10.2026 · AI Release · @ai_release1
How to Run a Local LLM Server on AMD Strix Halo: HALOGEN and Qwen on Fedora 43

TL;DR

- AMD Ryzen AI Max ("Strix Halo") machine with a Radeon 8060S-class integrated GPU

- Fedora 43 Workstation, root access via sudo

- Build toolchain: git, cmake, gcc-c++, make

- GPU runtime: ROCm runtime + Vulkan loader/drivers (rocminfo, vulkaninfo must both work)

- A Qwen model in GGUF format — a single file that contains the model weights, usually quantized (compressed) to save memory

- Free disk space: at least twice the size of the model file (build tree + weights)

> The channel posts do not pin a HALOGEN commit hash, a Qwen model file name, or benchmark numbers. Wherever a value depends on a specific release, this guide says so and tells you what to check.

ComponentCheck commandExpected result
CPU/GPU model`lscpu \grep "Model name"`Ryzen AI Max entry
GPU nodes`ls /dev/kfd /dev/dri/renderD*`paths exist
ROCm target`rocminfo \grep -m1 gfx`a `gfx…` target (Strix Halo prints `gfx1151`; note what yours prints)
Vulkan device`vulkaninfo --summary \grep deviceName`Radeon 8060S-class iGPU
Unified memory`free -h`shared RAM pool used by the iGPU

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

If you have never run a model locally before, read How to run AI models locally on your PC: a beginner's guide first — it covers the vocabulary (weights, quantization, context window) this guide assumes.

Why Strix Halo is the interesting part: on this platform the iGPU has no separate VRAM. It reads from the same system RAM as the CPU, so the "GPU memory" available to the model is a slice of your total RAM rather than a fixed 8 or 16 GB pool. That is why a Qwen model that would not fit on a discrete card can still run here — and why you must watch total RAM usage instead of VRAM usage.

Step 1. Prepare Fedora 43 and confirm the GPU is visible

What to do: install the toolchain and GPU runtime, add your user to the GPU groups, reboot, and confirm the runtime sees the iGPU.

sudo dnf update -y
sudo dnf install -y git cmake gcc-c++ make python3-devel \
  rocminfo vulkan-loader vulkan-tools mesa-vulkan-drivers
sudo usermod -aG video,render "$USER"

Log out and back in (or reboot) so the new group membership applies, then:

rocminfo | grep -m1 gfx
vulkaninfo --summary | grep -i deviceName
ls -l /dev/kfd /dev/dri/renderD*

Why: HALOGEN talks to the GPU through the ROCm/Vulkan stack. If the runtime cannot enumerate the iGPU, the server will start but silently fall back to CPU inference, which is far slower.

How to check: rocminfo prints a target such as gfx1151; vulkaninfo prints a device name containing Radeon 8060S (the exact strings vary by driver build — write yours down); ls shows /dev/kfd and a renderD* node without permission errors.

Example: on a Ryzen AI Max machine the three commands print, respectively, gfx1151, deviceName = AMD Radeon 8060S Graphics, and crw-rw---- 1 root render … /dev/dri/renderD128.

If a ROCm package is missing from your Fedora mirror, add AMD's official repository for your Fedora release and re-run the install — check the ROCm installation documentation for the current repository URL, because it changes between releases.

Step 2. Build HALOGEN and start the Qwen server

Illustration: Step 2. Build HALOGEN and start the Qwen server
Illustration: Step 2. Build HALOGEN and start the Qwen server

What to do: clone HALOGEN, compile it in Release mode, download a Qwen GGUF file, and launch the OpenAI-compatible HTTP server on port 8080.

git clone <halogen-repo-url> ~/halogen
cd ~/halogen
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"

<halogen-repo-url> is the repository listed on the HALOGEN project page — take the current URL from there, since the posts do not pin a commit.

Download the model (a GGUF file is a single binary blob plus a small metadata header, so one file is the whole model):

python3 -m pip install --user huggingface_hub
mkdir -p ~/models
huggingface-cli download <qwen-repo-id> <qwen-file>.gguf --local-dir ~/models

Start the server:

cd ~/halogen
./build/bin/halogen-server \
  -m ~/models/<qwen-file>.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 999 -c 8192

Why: -ngl 999 means "offload as many transformer layers to the GPU as possible" (999 is a ceiling, not a literal count); -c 8192 sets an 8K-token context window. Both flag names follow the llama.cpp server convention that HALOGEN's CLI mirrors — run ./build/bin/halogen-server --help and confirm the exact spellings for your build before filing a bug.

How to check: the startup log reports the GPU backend and a layer count. You want a line stating that all (or nearly all) layers were offloaded, and a final server listening on 0.0.0.0:8080. A log line reading offloaded 0/33 layers to GPU means the GPU was not used — see Troubleshooting.

Example: a healthy startup ends with main: server is listening on http://0.0.0.0:8080 - starting the main loop.

If you later want to move this server to a different machine or a smaller box, the hardware trade-offs are covered in Open-weight AI models: how to run them for free in 2026.

Step 3. Verify the endpoint and run a real request

What to do: query the server two ways — list models, then send a chat completion — and confirm you get generation, not an error.

curl -s http://localhost:8080/v1/models | python3 -m json.tool

Example output (abridged):

{
    "object": "list",
    "data": [{ "id": "qwen", "object": "model" }]
}

Then a real prompt:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Explain a GGUF file in one sentence."}],
    "max_tokens": 64
  }'

Why: /v1/models proves routing works; /v1/chat/completions proves the model actually loads, the GPU backend is active, and token generation finishes cleanly.

How to check: the chat response contains a non-empty choices[0].message.content string. Any OpenAI-compatible client works too:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Name one use case for a local LLM server."}],
)
print(r.choices[0].message.content)

Example: print outputs a normal sentence about private/offline inference. If it hangs for more than a minute on a short prompt, the model is almost certainly running on the CPU — go back to the startup log and check the offload line.

Troubleshooting

Problem. The server starts but every request is extremely slow and the log shows no GPU offload

rocminfo | grep -m1 gfx && vulkaninfo --summary | grep -i deviceName

Problem. Immediate crash at startup with a permission error on the GPU node

sudo usermod -aG video,render "$USER"
id | tr ' ' '\n' | grep -E 'video|render'

Problem. curl cannot reach the server from the same machine

ss -ltnp | grep 8080
./build/bin/halogen-server --help | grep -E '\-\-host|\-\-port'

To reach the server from another machine on the LAN, open the port: sudo firewall-cmd --add-port=8080/tcp --permanent && sudo firewall-cmd --reload.

Problem. Model load fails with an allocation error

Summary and next steps

You now have HALOGEN serving a Qwen GGUF model over an OpenAI-compatible API at http://localhost:8080/v1, with the iGPU doing the inference on Fedora 43. Next, wrap the server in a systemd --user unit so it survives logout, and point your editor or chat client at the same base_url. For a side-by-side look at other open models you can serve the same way, see How to Run Meta's New Open Model Locally and Compare It with DeepSeek and Qwen.

FAQ

Does the Qwen model need to fit in dedicated VRAM?

No. On Strix Halo the iGPU uses system RAM, so the model occupies a slice of your total memory pool. Watch total RAM consumption (free -h) while the server runs, not a VRAM counter.

Can I call this server from another computer on my network?

Yes, if you start it with --host 0.0.0.0 and open the port in firewalld (sudo firewall-cmd --add-port=8080/tcp --permanent && sudo firewall-cmd --reload). Then use http://<server-ip>:8080/v1 as the base URL.

Which quantization of Qwen should I download?

The posts do not recommend a specific file, so check the model card on Hugging Face: smaller quantizations use less memory and lose some quality. Start with the largest file that comfortably fits your available RAM alongside your desktop session.

How do I keep the server running after I close the terminal?

Run it in a tmux session, or create a systemd --user unit that starts halogen-server with the same flags and enable it with systemctl --user enable --now halogen.service.

🤖 AI summary

Related guides

How to Run Meta's New Open Model Locally and Compare It with DeepSeek and Qwen

Где скачать свежую открытую модель Meta, какие нужны GPU и VRAM, как настроить запуск и в каких задачах она выигрывает и

How to run AI models locally on your PC: a beginner's guide

Пошаговое руководство по установке и запуску локальных нейросетей на домашнем ПК без облачных сервисов.

How to Run an AI Model Locally on Android: Apps and Setup

Инструкция по запуску LLM на смартфоне с Android: выбор приложений, установка моделей и ограничения по памяти.