# Jeff 0.8B: a millisecond-fast model and who actually needs it > AI Release · @ai_release1 · https://ai-release.org/guides/jeff-0-8b-lokalno_en.html ## TL;DR - Time to complete: 30 minutes - Difficulty: Intermediate - You will need: a Linux x86_64 or macOS arm64 host, one serving runtime (Ollama, vLLM, LM Studio, llama.cpp, or nexa-bridge in front of them), Python 3.10+, `curl`, and the exact model ID of the Jeff 0.8B checkpoint from its model card ## Requirements and setup ![Illustration: Requirements and setup](https://ai-release.org/guides/img/jeff-0-8b-lokalno-en-1.jpg?v=1793867799) The channel posts do not document Jeff 0.8B itself — no file size, quantization, license, context length, or benchmark numbers. Everything below that depends on the checkpoint is written as a placeholder (``, `.gguf`); read those values from the model card first. The serving commands, ports, and endpoints are standard and verifiable. - **OS:** Linux x86_64 or macOS arm64. Windows works only through WSL2, because vLLM and llama.cpp server builds assume a POSIX shell. - **Python:** 3.10 or newer (`python3 --version`). Needed for the vLLM server and for the measurement script. - **RAM/VRAM:** check the card. As a reference point from our channel: Qwen 3.8 27B weighs 17 GB and runs on a laptop, so a 0.8B checkpoint is a fraction of that — but confirm the exact number instead of guessing. - **Disk:** leave headroom for the weights plus the runtime's cache directory. - **Rights:** no root needed if you install Ollama or llama.cpp into your home directory; vLLM is easiest in a virtualenv. - **Ports:** 8000 (vLLM), 11434 (Ollama), 8080 (llama.cpp server), 1234 (LM Studio). Only one process per port. - **Optional:** nexa-bridge if the machine holding the weights sits behind NAT. Run the pre-flight checklist before touching the model: ```bash python3 --version # expect Python 3.10.x or newer curl --version # expect curl 7.x or 8.x free -g # expect Available >= the model card figure ss -ltnp | grep -E '8000|8080|11434' # expect no output = ports are free ``` - [ ] Python 3.10+ confirmed - [ ] `curl` installed - [ ] Free RAM at or above the model card figure - [ ] Ports 8000/8080/11434 free - [ ] Model ID and license copied from the model card **Example:** `ss -ltnp | grep 11434` prints nothing on a clean host. If it prints `LISTEN 0 4096 127.0.0.1:11434 ... users:(("ollama",pid=2143,fd=3))`, an Ollama instance is already running and you should reuse it instead of starting a second one. ## Step 1. Decide where a sub-second model belongs in your stack ![Illustration: Step 1. Decide where a sub-second model belongs in your stack](https://ai-release.org/guides/img/jeff-0-8b-lokalno-en-2.jpg?v=1793867803) **What to do:** before downloading weights, write down which request path will call the model. A fast small model is not a replacement for your main model — it is a filter, router, or selector in front of it. **Why:** latency budgets are set by the caller, not the model. A 0.8B model that answers in milliseconds is wasted if the answer then waits 4 seconds for a bigger model behind it. **How to check:** you should be able to name the caller (an HTTP handler, a queue worker, an agent loop) and the number of output tokens it needs. If you cannot, stop here. | Runtime | What you get | Pick it when | |---|---|---| | Ollama | Local server with an OpenAI-compatible `/v1` surface | You want the fastest path to a running endpoint | | vLLM | High-throughput OpenAI-compatible server | You need batching and many concurrent requests | | LM Studio | Desktop app with a local server | You are prototyping on a workstation | | llama.cpp | Minimal C++ server for GGUF weights | You have no GPU and want a single binary | | nexa-bridge | OpenAI-compatible Rust proxy with an outbound tunnel in front of vLLM, Ollama, LM Studio, or llama.cpp | The model machine is behind NAT and you cannot forward ports | Typical job shapes for a small, fast model: | Job | Input | Output size | Why it must be fast | |---|---|---|---| | Request router | One user message | 1–5 tokens (a label) | It runs on every request | | PII pre-filter | One prompt | Rewritten prompt | It sits between client and cloud model | | Action selector | State vector from code | One action ID | It runs inside a loop, many times per task | **Example:** a PII filter in front of a cloud model. Our channel covered a self-hosted `guardrails-llm-filter` (Apache-2.0, from Cloud.ru) used as a proxy: in `enforce` mode it replaces personal data and secrets with placeholders, in `detect` mode it only writes to a log. That is exactly the class of work where a 0.8B model earns its latency. **Example:** action selection. TypeSafe AI's JEV picks an action inside code instead of generating text; the open analogue Laya runs in a FastAPI service on the author's own server. A loop that runs dozens of decisions per task cannot wait on a chat model. If you need the wider picture of which model sizes exist before choosing, see [Best AI models and neural networks in 2026](https://ai-release.org/guides/luchshie-nejroseti-i-modeli-2026.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). ## Step 2. Start an OpenAI-compatible endpoint for Jeff 0.8B **What to do:** launch one runtime, bind it to `127.0.0.1`, and give the served model a stable alias (`jeff-0.8b`) so client code never depends on the raw checkpoint path. **Why:** every runtime below exposes the same `/v1/chat/completions` contract, so you can swap engines later without touching application code. The alias is what your code sends as `"model"`. **How to check:** `GET /v1/models` lists your alias. Option A — vLLM: ```bash python3 -m vllm.entrypoints.openai.api_server \ --model \ --served-model-name jeff-0.8b \ --host 127.0.0.1 \ --port 8000 \ --max-model-len 4096 ``` Option B — Ollama: ```bash ollama serve & ollama pull ollama run "ping" # sanity check, not a benchmark ``` Option C — llama.cpp with GGUF weights: ```bash ./llama-server -m .gguf --host 127.0.0.1 --port 8080 -c 4096 ``` Verify the endpoint, replacing the port to match the runtime you started: ```bash curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool ``` Expected output — a JSON object whose `data` array contains your alias: ```json { "object": "list", "data": [ { "id": "jeff-0.8b", "object": "model" } ] } ``` Option D — the machine is behind NAT. Do not open ports. The channel covered `nexa-bridge`, an OpenAI-compatible Rust proxy with an outbound tunnel: the machine holding vLLM, Ollama, LM Studio, or llama.cpp stays behind NAT and the connection goes outward. The repository is named as `NexaEthos/nexa-bridge` on GitHub — confirm the exact clone URL and flag names in its README, then start it in front of the port from Option A or B. **Example:** `ollama pull ` prints progress lines and ends with `success`. If you instead see `Error: model "" not found`, the tag is wrong — copy it verbatim from the model card. For a from-scratch walkthrough of local installs on a workstation, see [How to run AI models locally on your PC: a beginner's guide](https://ai-release.org/guides/kak-zapustit-ii-model-lokalno-na-svoem-kompyutere-gayd.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). ## Step 3. Measure real latency and confirm the endpoint ![Illustration: Step 3. Measure real latency and confirm the endpoint](https://ai-release.org/guides/img/jeff-0-8b-lokalno-en-3.jpg?v=1793867809) **What to do:** send one warm-up request, then 20 identical requests, and record p50 and p95 wall-clock time. Latency claims without a percentile are marketing; p95 is what your users feel. **Why:** the first call is always slow — kernels load, weights page in, CUDA graphs get captured. Measuring a cold call gives you a number nobody will ever see in production. **How to check:** the script below prints one line with two figures. Anything else means the endpoint is not answering. ```python import json, statistics, time, urllib.request URL = "http://127.0.0.1:8000/v1/chat/completions" # 11434 for Ollama MODEL = "jeff-0.8b" def ask(prompt): body = json.dumps({ "model": MODEL, "messages": [{"role": "user", "content": prompt}], "max_tokens": 32, "temperature": 0 }).encode() req = urllib.request.Request(URL, data=body, headers={"Content-Type": "application/json"}) t0 = time.perf_counter() with urllib.request.urlopen(req, timeout=30) as r: payload = json.load(r) return (time.perf_counter() - t0) * 1000, payload["usage"]["completion_tokens"] print(ask("warm-up")) # discard this result runs = [ask("Classify: order 1234 was charged twice.") for _ in range(20)] ms = sorted(lat for lat, _ in runs) print(f"p50={statistics.median(ms):.1f} ms p95={ms[int(len(ms)*0.95)-1]:.1f} ms") ``` Expected on screen — shapes like this, with your own numbers: ``` p50=NN.N ms p95=NN.N ms ``` Success criteria: - [ ] `/v1/models` returns your alias - [ ] The warm-up call returns a completion, not a timeout - [ ] 20/20 measured calls succeeded - [ ] p95 is inside the budget you wrote down in Step 1 - [ ] `completion_tokens` is in the 1–32 range, i.e. you are measuring a short answer, not a long generation **Example:** if your caller budget is a few milliseconds per request and p95 comes back an order of magnitude higher, the model is not the bottleneck — check whether the runtime reloaded weights between calls or whether you are hitting a cold process fork. ## Troubleshooting **Problem. The endpoint refuses connections right after startup.** - Error: `curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Connection refused` - Cause: the server crashed during load (usually out of memory) or bound to a different interface or port. - Solution: check the process and the listening socket, then read the server log tail for the real reason: `ss -ltnp | grep 8000`. If nothing is listening, restart with `--host 127.0.0.1 --port 8000` and watch stdout. **Problem. The client asks for a model that the server does not know.** - Error: `{"object":"error","message":"The model `jeff-0.8b` does not exist","type":"NotFoundError","code":null}` - Cause: the alias passed with `--served-model-name` differs from the `"model"` field in your request, or the Ollama tag was pulled under a different name. - Solution: list what the server actually serves, then copy the `id` value into your code: `curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool`. **Problem. The first request hangs, later ones are fine.** - Error: `curl: (28) Operation timed out after 30001 milliseconds with 0 bytes received` - Cause: cold start — weights are being paged in or the process was evicted between calls. A smaller, faster model does not remove this; it only shortens it. - Solution: keep one process resident, send a warm-up request on deploy, and only then put the endpoint behind a load balancer. If the runtime cannot allocate at all, you will see a memory error such as `Error: model requires more system memory (X GiB) than is available (Y GiB)` — treat it as a sizing problem and read the real requirement from the model card. ## Summary and next steps You now have an OpenAI-compatible endpoint for Jeff 0.8B, a repeatable p50/p95 measurement, and a short list of jobs where a sub-second model pays off. Next, wire that endpoint into one narrow path — a router, a PII pre-filter, or an action selector — and compare its p95 with the larger model you use today. ## FAQ **Who actually needs a 0.8B model instead of a large one?** Callers that run on every request and need a tiny output: request routers that emit one label, PII filters that rewrite a prompt before it leaves your network, and action selectors that run inside a loop. If your task needs long reasoning or open-ended text, a small model is the wrong tool. **Can I serve it from a machine behind NAT without port forwarding?** Yes. nexa-bridge is an OpenAI-compatible Rust proxy with an outbound tunnel that sits in front of vLLM, Ollama, LM Studio, or llama.cpp; the model machine stays behind NAT and the connection goes outward from it. **How do I stop personal data from reaching a cloud model?** Put a self-hosted filter in front of the cloud call. In the setup our channel described, `guardrails-llm-filter` (Apache-2.0) runs as a proxy: `enforce` mode replaces personal data and secrets with placeholders, `detect` mode only logs them. Note the documented risk — if masking fails, the request can leave unprocessed, so that case has to be tracked as an incident. **How can I prove which model produced an answer?** Anthropic's FlashProve generates zkSNARK proofs — cryptographic proofs that an answer came from the declared model. On Llama 4 Maverick with one H100 the proofs are produced at roughly 110 tokens/s, and a proof for a 1,024-token request is about 19 MB. The code is Apache 2.0, but it is a research result, not a finished product.