# How to Run a Local LLM Server on AMD Strix Halo: HALOGEN and Qwen on Fedora 43

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-zapustit-lokalnyy-llm-server-na-amd-strix-halo-halo_en.html

## TL;DR

- **Time to complete:** 45–60 minutes (the first compile is the slowest part)
- **Difficulty:** Intermediate
- **You will need:**
  - AMD Ryzen AI Max ("Strix Halo") machine with a Radeon 8060S-class integrated GPU
  - Fedora 43 Workstation, root access via `sudo`
  - Build toolchain: `git`, `cmake`, `gcc-c++`, `make`
  - GPU runtime: ROCm runtime + Vulkan loader/drivers (`rocminfo`, `vulkaninfo` must both work)
  - A Qwen model in GGUF format — a single file that contains the model weights, usually quantized (compressed) to save memory
  - Free disk space: at least twice the size of the model file (build tree + weights)

> The channel posts do not pin a HALOGEN commit hash, a Qwen model file name, or benchmark numbers. Wherever a value depends on a specific release, this guide says so and tells you what to check.

| Component | Check command | Expected result |
|---|---|---|
| CPU/GPU model | `lscpu \| grep "Model name"` | Ryzen AI Max entry |
| GPU nodes | `ls /dev/kfd /dev/dri/renderD*` | paths exist |
| ROCm target | `rocminfo \| grep -m1 gfx` | a `gfx…` target (Strix Halo prints `gfx1151`; note what yours prints) |
| Vulkan device | `vulkaninfo --summary \| grep deviceName` | Radeon 8060S-class iGPU |
| Unified memory | `free -h` | shared RAM pool used by the iGPU |

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-zapustit-lokalnyy-llm-server-na-amd-strix-halo-halo-en-1.jpg?v=1792918235)


- [ ] Fedora 43 installed and updated (`sudo dnf update -y`).
- [ ] Your user is in the `video` and `render` groups — the GPU device nodes are group-owned.
- [ ] Build toolchain installed: `git`, `cmake`, `gcc-c++`, `make`, `python3-devel`.
- [ ] GPU stack installed and working: ROCm runtime plus Vulkan loader and Mesa Vulkan drivers.
- [ ] A Qwen GGUF file downloaded from Hugging Face into `~/models`.
- [ ] Port `8080/tcp` free on the host (`ss -ltnp | grep 8080` prints nothing).

If you have never run a model locally before, read [How to run AI models locally on your PC: a beginner's guide](https://ai-release.org/guides/kak-zapustit-ii-model-lokalno-na-svoem-kompyutere-gayd.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) first — it covers the vocabulary (weights, quantization, context window) this guide assumes.

**Why Strix Halo is the interesting part:** on this platform the iGPU has no separate VRAM. It reads from the same system RAM as the CPU, so the "GPU memory" available to the model is a slice of your total RAM rather than a fixed 8 or 16 GB pool. That is why a Qwen model that would not fit on a discrete card can still run here — and why you must watch total RAM usage instead of VRAM usage.

## Step 1. Prepare Fedora 43 and confirm the GPU is visible

**What to do:** install the toolchain and GPU runtime, add your user to the GPU groups, reboot, and confirm the runtime sees the iGPU.

```bash
sudo dnf update -y
sudo dnf install -y git cmake gcc-c++ make python3-devel \
  rocminfo vulkan-loader vulkan-tools mesa-vulkan-drivers
sudo usermod -aG video,render "$USER"
```

Log out and back in (or reboot) so the new group membership applies, then:

```bash
rocminfo | grep -m1 gfx
vulkaninfo --summary | grep -i deviceName
ls -l /dev/kfd /dev/dri/renderD*
```

**Why:** HALOGEN talks to the GPU through the ROCm/Vulkan stack. If the runtime cannot enumerate the iGPU, the server will start but silently fall back to CPU inference, which is far slower.

**How to check:** `rocminfo` prints a target such as `gfx1151`; `vulkaninfo` prints a device name containing `Radeon 8060S` (the exact strings vary by driver build — write yours down); `ls` shows `/dev/kfd` and a `renderD*` node without permission errors.

**Example:** on a Ryzen AI Max machine the three commands print, respectively, `gfx1151`, `deviceName = AMD Radeon 8060S Graphics`, and `crw-rw---- 1 root render … /dev/dri/renderD128`.

If a ROCm package is missing from your Fedora mirror, add AMD's official repository for your Fedora release and re-run the install — check the ROCm installation documentation for the current repository URL, because it changes between releases.

## Step 2. Build HALOGEN and start the Qwen server
![Illustration: Step 2. Build HALOGEN and start the Qwen server](https://ai-release.org/guides/img/kak-zapustit-lokalnyy-llm-server-na-amd-strix-halo-halo-en-2.jpg?v=1792918240)


**What to do:** clone HALOGEN, compile it in Release mode, download a Qwen GGUF file, and launch the OpenAI-compatible HTTP server on port 8080.

```bash
git clone <halogen-repo-url> ~/halogen
cd ~/halogen
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"
```

`<halogen-repo-url>` is the repository listed on the HALOGEN project page — take the current URL from there, since the posts do not pin a commit.

Download the model (a GGUF file is a single binary blob plus a small metadata header, so one file is the whole model):

```bash
python3 -m pip install --user huggingface_hub
mkdir -p ~/models
huggingface-cli download <qwen-repo-id> <qwen-file>.gguf --local-dir ~/models
```

Start the server:

```bash
cd ~/halogen
./build/bin/halogen-server \
  -m ~/models/<qwen-file>.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 999 -c 8192
```

**Why:** `-ngl 999` means "offload as many transformer layers to the GPU as possible" (999 is a ceiling, not a literal count); `-c 8192` sets an 8K-token context window. Both flag names follow the llama.cpp server convention that HALOGEN's CLI mirrors — run `./build/bin/halogen-server --help` and confirm the exact spellings for your build before filing a bug.

**How to check:** the startup log reports the GPU backend and a layer count. You want a line stating that all (or nearly all) layers were offloaded, and a final `server listening on 0.0.0.0:8080`. A log line reading `offloaded 0/33 layers to GPU` means the GPU was not used — see Troubleshooting.

**Example:** a healthy startup ends with `main: server is listening on http://0.0.0.0:8080 - starting the main loop`.

If you later want to move this server to a different machine or a smaller box, the hardware trade-offs are covered in [Open-weight AI models: how to run them for free in 2026](https://ai-release.org/guides/kakie-ii-modeli-imeyut-otkrytye-vesa-i-kak-ih-zapustit.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## Step 3. Verify the endpoint and run a real request

**What to do:** query the server two ways — list models, then send a chat completion — and confirm you get generation, not an error.

```bash
curl -s http://localhost:8080/v1/models | python3 -m json.tool
```

**Example output (abridged):**

```json
{
    "object": "list",
    "data": [{ "id": "qwen", "object": "model" }]
}
```

Then a real prompt:

```bash
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Explain a GGUF file in one sentence."}],
    "max_tokens": 64
  }'
```

**Why:** `/v1/models` proves routing works; `/v1/chat/completions` proves the model actually loads, the GPU backend is active, and token generation finishes cleanly.

**How to check:** the chat response contains a non-empty `choices[0].message.content` string. Any OpenAI-compatible client works too:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "Name one use case for a local LLM server."}],
)
print(r.choices[0].message.content)
```

**Example:** `print` outputs a normal sentence about private/offline inference. If it hangs for more than a minute on a short prompt, the model is almost certainly running on the CPU — go back to the startup log and check the offload line.

## Troubleshooting

**Problem. The server starts but every request is extremely slow and the log shows no GPU offload**
- Error: `load_tensors: offloaded 0/33 layers to GPU`
- Cause: the GPU backend was not compiled in or the runtime cannot see the device, so HALOGEN falls back to CPU.
- Solution: re-run Step 1 checks, then rebuild with the GPU backend enabled. Confirm the device is visible first:
```bash
rocminfo | grep -m1 gfx && vulkaninfo --summary | grep -i deviceName
```

**Problem. Immediate crash at startup with a permission error on the GPU node**
- Error: `failed to open /dev/kfd: Permission denied`
- Cause: your user is not in the `render`/`video` groups, or the session predates the `usermod`.
- Solution: add the groups and start a fresh login session (a reboot is the reliable way):
```bash
sudo usermod -aG video,render "$USER"
id | tr ' ' '\n' | grep -E 'video|render'
```

**Problem. `curl` cannot reach the server from the same machine**
- Error: `curl: (7) Failed to connect to 127.0.0.1 port 8080`
- Cause: the server bound to a different interface (`--host 127.0.0.1` vs `0.0.0.0`), exited on a model-load error, or the port is taken.
- Solution: check the process and the port, then re-read the server log tail:
```bash
ss -ltnp | grep 8080
./build/bin/halogen-server --help | grep -E '\-\-host|\-\-port'
```
To reach the server from another machine on the LAN, open the port: `sudo firewall-cmd --add-port=8080/tcp --permanent && sudo firewall-cmd --reload`.

**Problem. Model load fails with an allocation error**
- Error: `ggml_backend_alloc_ctx_tensors: failed to allocate buffer`
- Cause: the requested context window (`-c`) plus the model exceeds the memory the iGPU can address from the shared pool.
- Solution: lower the context (`-c 4096`) or use a more heavily quantized Qwen file — check the model card for the memory footprint of each quantization before downloading.

## Summary and next steps

You now have HALOGEN serving a Qwen GGUF model over an OpenAI-compatible API at `http://localhost:8080/v1`, with the iGPU doing the inference on Fedora 43. Next, wrap the server in a `systemd --user` unit so it survives logout, and point your editor or chat client at the same `base_url`. For a side-by-side look at other open models you can serve the same way, see [How to Run Meta's New Open Model Locally and Compare It with DeepSeek and Qwen](https://ai-release.org/guides/kak-zapustit-novuyu-otkrytuyu-model-meta-lokalno-i-srav.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## FAQ

**Does the Qwen model need to fit in dedicated VRAM?**
No. On Strix Halo the iGPU uses system RAM, so the model occupies a slice of your total memory pool. Watch total RAM consumption (`free -h`) while the server runs, not a VRAM counter.

**Can I call this server from another computer on my network?**
Yes, if you start it with `--host 0.0.0.0` and open the port in firewalld (`sudo firewall-cmd --add-port=8080/tcp --permanent && sudo firewall-cmd --reload`). Then use `http://<server-ip>:8080/v1` as the base URL.

**Which quantization of Qwen should I download?**
The posts do not recommend a specific file, so check the model card on Hugging Face: smaller quantizations use less memory and lose some quality. Start with the largest file that comfortably fits your available RAM alongside your desktop session.

**How do I keep the server running after I close the terminal?**
Run it in a `tmux` session, or create a `systemd --user` unit that starts `halogen-server` with the same flags and enable it with `systemctl --user enable --now halogen.service`.