← All guides AI Release RU Subscribe to @ai_release1

How to download a model from Hugging Face and run it locally: filters, GGUF and quantization

Updated: 05.11.2026 · AI Release · @ai_release1

TL;DR

Time to complete: 30–45 minutes (the download itself depends on file size and your connection)
Difficulty: Intermediate
You need: a terminal, Python 3 with pip, the huggingface_hub package, llama.cpp (prebuilt release or built from source), free disk space equal to the model file plus headroom, a Hugging Face account (free)

The guide uses placeholders like <org>/<model>-GGUF for repository IDs and file names. Repository IDs and file names change weekly, so copy the real values from the model page you open — never from this text.

Requirements and setup

What to do: prepare the machine before touching any model.

OS: Linux, macOS, or Windows 10/11. On Windows use either the prebuilt llama.cpp binaries or WSL2 (Windows Subsystem for Linux — a real Linux kernel running inside Windows).
Python 3 with pip. Check the version your tooling supports in the documentation; this guide states no version number on purpose.
Disk space: at least twice the size of the .gguf file you plan to download (the file itself plus cache). Check with df -h.
RAM/VRAM: you need enough memory to hold the whole file plus the context window. The file size is shown on the model page — use it as your floor.
Hugging Face account + access token: create one under Settings → Access Tokens.

Why: every failure later in this guide — 401 errors, out-of-memory kills, truncated downloads — is cheaper to prevent here than to debug at step 3.

How to check:

python3 --version
pip --version
df -h .
which git && which curl

Expected: three version lines and paths to git and curl. Any "command not found" means install that package first.

Then create an isolated environment and install the client:

python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -U huggingface_hub
huggingface-cli --help

Example: on a fresh Ubuntu box the last command prints a usage block starting with usage: huggingface-cli <command> [<args>]. On newer releases of the library the binary is renamed to hf; if huggingface-cli is not found, run hf --help and use hf download wherever this guide says huggingface-cli download — check the documentation for your installed version.

[ ] python3 --version returns a line
[ ] huggingface-cli --help (or hf --help) prints usage
[ ] df -h . shows free space greater than 2× your target file
[ ] Access token copied to a password manager

If you have never run a model on your own hardware before, start with the beginner walkthrough How to run AI models locally on your PC: a beginner's guide and come back here for the download and quantization details.

Step 1. Find the right file: filters, GGUF and quantization

What to do: locate a repository that ships a .gguf file and pick one quantization level.

Why: a Hugging Face repository is a folder, not a single file. Most repositories hold the original weights (.safetensors) divided into many shards; those are for training and for GPU servers, not for a laptop. GGUF is a single-file format designed for local inference: the weights, the tokenizer and the metadata live in one file that llama.cpp can memory-map. Quantization is the process of storing each weight in fewer bits, which shrinks the file at some cost in answer quality.

Open https://huggingface.co/models and set:

1. Task → Text Generation.

2. Search field → GGUF.

3. Sort → Most downloads.

4. Size filter → your parameter range (a 7B model is roughly a quarter of the memory of a 30B model; bigger means slower and hungrier).

Then open a repository and switch to the Files and versions tab. You will see a list of .gguf files with their sizes in GB.

File suffix	Bits per weight	What you get
`Q2_K`	~2	Smallest file, most visible quality loss
`Q3_K_M`	~3	Small; degradation shows up on reasoning tasks
`Q4_K_M`	~4	Frequently used middle ground
`Q5_K_M`	~5	Closer to the original, larger
`Q6_K`	~6	Near-original
`Q8_0`	8	Almost lossless, large
`F16`	16	Unquantized reference

K marks the k-quant family; S, M and L are small/medium/large mixes inside that family — M is the usual pick. Choose the largest quant that fits in your memory.

Example: you open a repository, see model-Q4_K_M.gguf at a given size in the file list, and your machine has more free RAM than that number. That file is your candidate; note the exact name, you will pass it to --include in step 2.

How to check: the Files and versions tab lists at least one file ending in .gguf, and the repository is not marked "gated" (a lock icon next to the name). Gated repositories require you to accept the license on the model page with your account logged in.

Step 2. Download the file and start a local server

What to do: authenticate, download only the file you need, then serve it.

Why: downloading one .gguf instead of the whole repository saves tens of gigabytes and avoids pulling the .safetensors shards you will never load. Serving the model over HTTP gives you an OpenAI-compatible endpoint, so existing client code works unchanged.

Authenticate once:

huggingface-cli login

Paste the token from step 0 when prompted. The token is stored locally; subsequent downloads use it automatically. This is what makes gated repositories work.

Download a single file:

huggingface-cli download <org>/<model>-GGUF \
  --include "*Q4_K_M.gguf" \
  --local-dir ./models

--include is a glob filter — only matching files are fetched. --local-dir puts them in a plain folder instead of the shared cache, which makes the next command shorter.

If the CLI fails, the same file is reachable over plain HTTP using the repository URL pattern:

curl -L -H "Authorization: Bearer $HF_TOKEN" \
  -o ./models/<model>-Q4_K_M.gguf \
  "https://huggingface.co/<org>/<model>-GGUF/resolve/main/<model>-Q4_K_M.gguf"

Start the server (from the llama.cpp build directory):

./llama-server \
  -m ./models/<model>-Q4_K_M.gguf \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8080

-c 4096 sets the context window in tokens (the amount of text the model can see at once). Larger context costs more RAM.
--host 127.0.0.1 binds to localhost only. Do not expose the port to the internet without an auth proxy.
-ngl <N> offloads N layers to the GPU. -ngl 0 forces CPU only; on a GPU machine raise it until memory is full. For deep GPU offload and quantization trade-offs see How to Run a 125B-Parameter LLM on an RTX 4090: Quantization, Offload and 100 Tokens/s.

The binary is called llama-server in current builds and server in older ones — check the documentation of the release you downloaded.

Example: the terminal prints model metadata (architecture, parameter count, quantization type) followed by a line announcing the listening address, e.g. listening on http://127.0.0.1:8080. The process stays in the foreground; leave it running and open a second terminal.

How to check: in the second terminal run curl -s http://127.0.0.1:8080/health. Expected body: {"status":"ok"}. A connection refused error means the server is not up — read the first terminal's output.

Step 3. Verify the model actually answers

What to do: send one chat request and one completion request, and read the timing block the server logs.

Why: a running port proves nothing about generation quality. You want to confirm three things: the model loads, it produces coherent text, and the throughput matches your hardware expectations.

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Reply with the single word: OK"}],
    "max_tokens": 16
  }'

Expected: a JSON object where choices[0].message.content contains OK, and usage reports prompt and completion token counts. The model field value is arbitrary for a single-model server.

Then the same endpoint from Python, which is how most applications will call it:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Summarise the GGUF format in one sentence."}],
    max_tokens=64,
)
print(r.choices[0].message.content)

Finally, look at the server terminal. After each request it prints prompt-eval and eval timings in tokens per second. That number is your baseline: if you re-run with a higher -ngl or a larger quant, compare against it.

Example: the first curl returns within a couple of seconds on CPU for a small model; the Python snippet prints one English sentence about single-file packaging. If you get an empty content and finish_reason: "length", your max_tokens is too small for the model to finish its first sentence.

How to check: all three of these hold — /health returns ok, the curl response contains a non-empty content, the server log shows a tokens-per-second figure after the request. That is a working local model.

Troubleshooting

Problem. Download fails with a permission error on a gated model

Error: 401 Client Error: Unauthorized for url: https://huggingface.co/<org>/<model>/resolve/main/<file>.gguf
Cause: the repository is gated and your account has not accepted its license, or the token was never sent — the CLI silently falls back to anonymous access if you are not logged in.
Solution: open the model page in a browser while logged in, accept the license, then re-run huggingface-cli login and repeat the download. For raw curl, confirm the variable is set: echo $HF_TOKEN must print a token, not an empty line.

Problem. The model loads but the server exits with a tensor shape error

Error: llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape
Cause: your llama.cpp build is older than the GGUF format version used by the file, or the file is truncated because the download was interrupted.
Solution: update to a current llama.cpp release and rebuild, then verify the file size matches the size shown on the model page (ls -lh ./models). If it is smaller, delete it and download again — huggingface-cli download resumes incomplete transfers.

Problem. Generation dies immediately on a GPU machine

Error: ggml_backend_cuda_buffer_type_alloc_buffer: allocating <N> MiB on device 0: cudaMalloc failed: out of memory
Cause: the weights plus the KV cache for your -c context do not fit in VRAM.
Solution: lower the context (-c 2048), reduce offload (-ngl to a smaller number so more layers stay on the CPU), or switch to a smaller quant such as Q3_K_M. Restart and re-check /health.

Summary and next steps

You now have a single .gguf file downloaded from a filtered Hugging Face search and served locally on 127.0.0.1:8080 through an OpenAI-compatible API. Next, benchmark two quantization levels on the same prompt and keep the smaller one if the answers stay usable, then compare other open-weight families as described in How to Run Meta's New Open Model Locally and Compare It with DeepSeek and Qwen.

FAQ

Do I need a GPU to run a GGUF model?

No. llama.cpp runs on CPU; -ngl 0 forces that mode. A GPU only raises tokens per second and lets you fit larger quants, because the whole file must fit in memory either way.

What is the difference between GGUF and safetensors?

Safetensors is a general weight-storage format, usually split across many shards and loaded by training or serving frameworks. GGUF is a single self-contained file with weights, tokenizer and metadata, built for local inference.

Which quantization should I pick first?

Start with Q4_K_M. If the model answers poorly on your task, move up to Q5_K_M or Q6_K while it still fits in memory; if it does not fit at all, move down to Q3_K_M.

Can I convert my own fine-tuned model to GGUF?

Yes — llama.cpp ships a conversion script that takes a Hugging Face-format directory and writes a GGUF file, followed by a separate quantize step. Script names and flags change between releases, so check the documentation in the repository you cloned.

🤖 AI summary

Updated: 2026-11-05
Topics: hugging face как скачать модель, как выбрать gguf квантизацию, запустить модель с huggingface локально
Summary: Пошаговое руководство: как искать модели на Hugging Face по лицензии и размеру, выбирать GGUF-квантизацию под свою видеокарту и запускать её в LM Studio или Ollama.
Key facts:
- TL;DR Time to complete: 30–45 minutes (the download itself depends on file size and your connection) Difficulty: Intermediate You need: a terminal, Python 3 with pip , the huggingface_hub package, llama.cpp (prebuilt release or built from source), free disk space equal to the model file plus headroom, a Hugging Face account (free) The guide uses placeholders like <org>/<model>-GGUF for repository IDs and file names.
- Repository IDs and file names change weekly, so copy the real values from the model page you open — never from this text.
- Requirements and setup Illustration: Requirements and setup What to do: prepare the machine before touching any model.
- OS: Linux, macOS, or Windows 10/11.
- On Windows use either the prebuilt llama.cpp binaries or WSL2 (Windows Subsystem for Linux — a real Linux kernel running inside Windows).
- Check the version your tooling supports in the documentation; this guide states no version number on purpose.

How to download a model from Hugging Face and run it locally: filters, GGUF and quantization

TL;DR

Requirements and setup

Step 1. Find the right file: filters, GGUF and quantization

Step 2. Download the file and start a local server

Step 3. Verify the model actually answers

Troubleshooting

Summary and next steps

FAQ

Related guides

How to run AI models locally on your PC: a beginner's guide

How to Run an AI Model Locally on Android: Apps and Setup

How to Run a 125B-Parameter LLM on an RTX 4090: Quantization, Offload and 100 Tokens/s