# How to download a model from Hugging Face and run it locally: filters, GGUF and quantization

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-skachat-model-s-hugging-face-i-zapustit-lokalno-fil_en.html

## TL;DR

- **Time to complete:** 30–45 minutes (the download itself depends on file size and your connection)
- **Difficulty:** Intermediate
- **You need:** a terminal, Python 3 with `pip`, the `huggingface_hub` package, `llama.cpp` (prebuilt release or built from source), free disk space equal to the model file plus headroom, a Hugging Face account (free)

The guide uses placeholders like `<org>/<model>-GGUF` for repository IDs and file names. Repository IDs and file names change weekly, so copy the real values from the model page you open — never from this text.

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-skachat-model-s-hugging-face-i-zapustit-lokalno-fil-en-1.jpg?v=1793868179)


**What to do:** prepare the machine before touching any model.

- **OS:** Linux, macOS, or Windows 10/11. On Windows use either the prebuilt `llama.cpp` binaries or WSL2 (Windows Subsystem for Linux — a real Linux kernel running inside Windows).
- **Python 3** with `pip`. Check the version your tooling supports in the documentation; this guide states no version number on purpose.
- **Disk space:** at least twice the size of the `.gguf` file you plan to download (the file itself plus cache). Check with `df -h`.
- **RAM/VRAM:** you need enough memory to hold the whole file plus the context window. The file size is shown on the model page — use it as your floor.
- **Hugging Face account + access token:** create one under Settings → Access Tokens.

**Why:** every failure later in this guide — 401 errors, out-of-memory kills, truncated downloads — is cheaper to prevent here than to debug at step 3.

**How to check:**

```bash
python3 --version
pip --version
df -h .
which git && which curl
```

Expected: three version lines and paths to `git` and `curl`. Any "command not found" means install that package first.

Then create an isolated environment and install the client:

```bash
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -U huggingface_hub
huggingface-cli --help
```

**Example:** on a fresh Ubuntu box the last command prints a usage block starting with `usage: huggingface-cli <command> [<args>]`. On newer releases of the library the binary is renamed to `hf`; if `huggingface-cli` is not found, run `hf --help` and use `hf download` wherever this guide says `huggingface-cli download` — check the documentation for your installed version.

- [ ] `python3 --version` returns a line
- [ ] `huggingface-cli --help` (or `hf --help`) prints usage
- [ ] `df -h .` shows free space greater than 2× your target file
- [ ] Access token copied to a password manager

If you have never run a model on your own hardware before, start with the beginner walkthrough [How to run AI models locally on your PC: a beginner's guide](https://ai-release.org/guides/kak-zapustit-ii-model-lokalno-na-svoem-kompyutere-gayd.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) and come back here for the download and quantization details.

## Step 1. Find the right file: filters, GGUF and quantization

**What to do:** locate a repository that ships a `.gguf` file and pick one quantization level.

**Why:** a Hugging Face repository is a folder, not a single file. Most repositories hold the original weights (`.safetensors`) divided into many shards; those are for training and for GPU servers, not for a laptop. GGUF is a single-file format designed for local inference: the weights, the tokenizer and the metadata live in one file that `llama.cpp` can memory-map. **Quantization** is the process of storing each weight in fewer bits, which shrinks the file at some cost in answer quality.

Open `https://huggingface.co/models` and set:

1. **Task** → Text Generation.
2. Search field → `GGUF`.
3. Sort → Most downloads.
4. **Size** filter → your parameter range (a 7B model is roughly a quarter of the memory of a 30B model; bigger means slower and hungrier).

Then open a repository and switch to the **Files and versions** tab. You will see a list of `.gguf` files with their sizes in GB.

| File suffix | Bits per weight | What you get |
|---|---|---|
| `Q2_K` | ~2 | Smallest file, most visible quality loss |
| `Q3_K_M` | ~3 | Small; degradation shows up on reasoning tasks |
| `Q4_K_M` | ~4 | Frequently used middle ground |
| `Q5_K_M` | ~5 | Closer to the original, larger |
| `Q6_K` | ~6 | Near-original |
| `Q8_0` | 8 | Almost lossless, large |
| `F16` | 16 | Unquantized reference |

`K` marks the k-quant family; `S`, `M` and `L` are small/medium/large mixes inside that family — `M` is the usual pick. Choose the largest quant that fits in your memory.

**Example:** you open a repository, see `model-Q4_K_M.gguf` at a given size in the file list, and your machine has more free RAM than that number. That file is your candidate; note the exact name, you will pass it to `--include` in step 2.

**How to check:** the **Files and versions** tab lists at least one file ending in `.gguf`, and the repository is not marked "gated" (a lock icon next to the name). Gated repositories require you to accept the license on the model page with your account logged in.

## Step 2. Download the file and start a local server
![Illustration: Step 2. Download the file and start a local server](https://ai-release.org/guides/img/kak-skachat-model-s-hugging-face-i-zapustit-lokalno-fil-en-2.jpg?v=1793868181)


**What to do:** authenticate, download only the file you need, then serve it.

**Why:** downloading one `.gguf` instead of the whole repository saves tens of gigabytes and avoids pulling the `.safetensors` shards you will never load. Serving the model over HTTP gives you an OpenAI-compatible endpoint, so existing client code works unchanged.

**Authenticate once:**

```bash
huggingface-cli login
```

Paste the token from step 0 when prompted. The token is stored locally; subsequent downloads use it automatically. This is what makes gated repositories work.

**Download a single file:**

```bash
huggingface-cli download <org>/<model>-GGUF \
  --include "*Q4_K_M.gguf" \
  --local-dir ./models
```

`--include` is a glob filter — only matching files are fetched. `--local-dir` puts them in a plain folder instead of the shared cache, which makes the next command shorter.

If the CLI fails, the same file is reachable over plain HTTP using the repository URL pattern:

```bash
curl -L -H "Authorization: Bearer $HF_TOKEN" \
  -o ./models/<model>-Q4_K_M.gguf \
  "https://huggingface.co/<org>/<model>-GGUF/resolve/main/<model>-Q4_K_M.gguf"
```

**Start the server** (from the `llama.cpp` build directory):

```bash
./llama-server \
  -m ./models/<model>-Q4_K_M.gguf \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8080
```

- `-c 4096` sets the context window in tokens (the amount of text the model can see at once). Larger context costs more RAM.
- `--host 127.0.0.1` binds to localhost only. Do not expose the port to the internet without an auth proxy.
- `-ngl <N>` offloads N layers to the GPU. `-ngl 0` forces CPU only; on a GPU machine raise it until memory is full. For deep GPU offload and quantization trade-offs see [How to Run a 125B-Parameter LLM on an RTX 4090: Quantization, Offload and 100 Tokens/s](https://ai-release.org/guides/kak-zapustit-model-na-125b-parametrov-na-rtx-4090-kvant.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

The binary is called `llama-server` in current builds and `server` in older ones — check the documentation of the release you downloaded.

**Example:** the terminal prints model metadata (architecture, parameter count, quantization type) followed by a line announcing the listening address, e.g. `listening on http://127.0.0.1:8080`. The process stays in the foreground; leave it running and open a second terminal.

**How to check:** in the second terminal run `curl -s http://127.0.0.1:8080/health`. Expected body: `{"status":"ok"}`. A connection refused error means the server is not up — read the first terminal's output.

## Step 3. Verify the model actually answers
![Illustration: Step 3. Verify the model actually answers](https://ai-release.org/guides/img/kak-skachat-model-s-hugging-face-i-zapustit-lokalno-fil-en-3.jpg?v=1793868186)


**What to do:** send one chat request and one completion request, and read the timing block the server logs.

**Why:** a running port proves nothing about generation quality. You want to confirm three things: the model loads, it produces coherent text, and the throughput matches your hardware expectations.

```bash
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Reply with the single word: OK"}],
    "max_tokens": 16
  }'
```

Expected: a JSON object where `choices[0].message.content` contains `OK`, and `usage` reports prompt and completion token counts. The `model` field value is arbitrary for a single-model server.

Then the same endpoint from Python, which is how most applications will call it:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Summarise the GGUF format in one sentence."}],
    max_tokens=64,
)
print(r.choices[0].message.content)
```

Finally, look at the server terminal. After each request it prints prompt-eval and eval timings in tokens per second. That number is your baseline: if you re-run with a higher `-ngl` or a larger quant, compare against it.

**Example:** the first `curl` returns within a couple of seconds on CPU for a small model; the Python snippet prints one English sentence about single-file packaging. If you get an empty `content` and `finish_reason: "length"`, your `max_tokens` is too small for the model to finish its first sentence.

**How to check:** all three of these hold — `/health` returns `ok`, the curl response contains a non-empty `content`, the server log shows a tokens-per-second figure after the request. That is a working local model.

## Troubleshooting

**Problem. Download fails with a permission error on a gated model**

- Error: `401 Client Error: Unauthorized for url: https://huggingface.co/<org>/<model>/resolve/main/<file>.gguf`
- Cause: the repository is gated and your account has not accepted its license, or the token was never sent — the CLI silently falls back to anonymous access if you are not logged in.
- Solution: open the model page in a browser while logged in, accept the license, then re-run `huggingface-cli login` and repeat the download. For raw `curl`, confirm the variable is set: `echo $HF_TOKEN` must print a token, not an empty line.

**Problem. The model loads but the server exits with a tensor shape error**

- Error: `llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape`
- Cause: your `llama.cpp` build is older than the GGUF format version used by the file, or the file is truncated because the download was interrupted.
- Solution: update to a current `llama.cpp` release and rebuild, then verify the file size matches the size shown on the model page (`ls -lh ./models`). If it is smaller, delete it and download again — `huggingface-cli download` resumes incomplete transfers.

**Problem. Generation dies immediately on a GPU machine**

- Error: `ggml_backend_cuda_buffer_type_alloc_buffer: allocating <N> MiB on device 0: cudaMalloc failed: out of memory`
- Cause: the weights plus the KV cache for your `-c` context do not fit in VRAM.
- Solution: lower the context (`-c 2048`), reduce offload (`-ngl` to a smaller number so more layers stay on the CPU), or switch to a smaller quant such as `Q3_K_M`. Restart and re-check `/health`.

## Summary and next steps

You now have a single `.gguf` file downloaded from a filtered Hugging Face search and served locally on `127.0.0.1:8080` through an OpenAI-compatible API. Next, benchmark two quantization levels on the same prompt and keep the smaller one if the answers stay usable, then compare other open-weight families as described in [How to Run Meta's New Open Model Locally and Compare It with DeepSeek and Qwen](https://ai-release.org/guides/kak-zapustit-novuyu-otkrytuyu-model-meta-lokalno-i-srav.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## FAQ

**Do I need a GPU to run a GGUF model?**
No. `llama.cpp` runs on CPU; `-ngl 0` forces that mode. A GPU only raises tokens per second and lets you fit larger quants, because the whole file must fit in memory either way.

**What is the difference between GGUF and safetensors?**
Safetensors is a general weight-storage format, usually split across many shards and loaded by training or serving frameworks. GGUF is a single self-contained file with weights, tokenizer and metadata, built for local inference.

**Which quantization should I pick first?**
Start with `Q4_K_M`. If the model answers poorly on your task, move up to `Q5_K_M` or `Q6_K` while it still fits in memory; if it does not fit at all, move down to `Q3_K_M`.

**Can I convert my own fine-tuned model to GGUF?**
Yes — `llama.cpp` ships a conversion script that takes a Hugging Face-format directory and writes a GGUF file, followed by a separate quantize step. Script names and flags change between releases, so check the documentation in the repository you cloned.