# How to Deploy a Local LLM on Kubernetes: Step-by-Step Guide > AI Release · @ai_release1 · https://ai-release.org/guides/kak-razvernut-lokalnuyu-ii-model-na-kubernetes-poshagov_en.html ## TL;DR - **Time to complete:** 60–90 minutes (cluster already exists and has a schedulable GPU) - **Difficulty:** Intermediate - **You will need:** a Kubernetes cluster (v1.24+ — check your distribution's docs for the exact supported range), `kubectl` configured against that cluster, one node with an NVIDIA GPU and the NVIDIA GPU Operator (or the NVIDIA device plugin + container toolkit) installed, a StorageClass that supports `ReadWriteOnce` volumes, a model file in GGUF format, and an inference server image with an OpenAI-compatible HTTP API (llama-server from llama.cpp is one option — [this guide](https://ai-release.org/guides/kak-podklyuchit-lokalnuyu-model-k-claude-code-llama-ser.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) covers the same server outside Kubernetes) ## Requirements and setup ![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-razvernut-lokalnuyu-ii-model-na-kubernetes-poshagov-en-1.jpg) - **OS and shell:** Linux or macOS workstation with bash; Windows users can run the same commands from WSL2. - **Cluster:** any conformant distribution (kubeadm, k3s, EKS/GKE/AKS). Managed clusters need a GPU node pool — check the provider docs for the node-pool flag names. - **`kubectl`:** installed and pointing at the right context. - **GPU stack:** NVIDIA driver on the node plus the GPU Operator (or device plugin). The node must advertise the `nvidia.com/gpu` resource. - **Storage:** a default StorageClass, or a named one you will reference explicitly. Model weights are large files — size the volume to your model, not to a guessed number. - **Model file:** a `.gguf` file you are licensed to use, uploaded somewhere the cluster can reach (see Step 2). - **Access:** permission to create namespaces, deployments, services and PVCs in the cluster. - **Free space:** enough on the workstation to hold the model file temporarily plus `kubectl` binaries. Pre-flight checklist — run each line and confirm the result before continuing: - [ ] `kubectl config current-context` prints the cluster you intend to use - [ ] `kubectl get nodes` shows at least one node in `Ready` - [ ] `kubectl get storageclass` shows a class you can use - [ ] `kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'` shows a non-empty GPU count on at least one node - [ ] `kubectl auth can-i create deployments -n default` prints `yes` **Example:** the fourth command above prints `gpu-node-1 1` when the node advertises exactly one GPU, and `gpu-node-1 ` (empty field) when the device plugin is not installed. If you are still deciding whether the workload belongs in a cluster or on a single box, [the self-hosted AI on a home server guide](https://ai-release.org/guides/kak-podklyuchit-ii-k-domashnemu-serveru-poshagovoe-ruko.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) covers the single-machine path; Kubernetes is worth it when you need scheduling, rolling updates, and a stable in-cluster DNS name. ## Step 1. Create the namespace and prove the GPU is schedulable **What to do:** Create a dedicated namespace and run a one-shot Pod that calls `nvidia-smi`. This isolates the AI workload from system pods and proves the GPU is actually injected into containers, not just visible on the host. ```bash kubectl create namespace llm ``` ```yaml # gpu-smoke.yaml apiVersion: v1 kind: Pod metadata: name: gpu-smoke namespace: llm spec: restartPolicy: Never containers: - name: gpu-smoke image: # any image with nvidia-smi; check the toolkit docs for a current tag command: ["nvidia-smi"] resources: limits: nvidia.com/gpu: 1 ``` ```bash kubectl apply -f gpu-smoke.yaml ``` **Why:** Kubernetes schedules GPUs as an extended resource. Requesting `nvidia.com/gpu: 1` in `limits` is what tells the scheduler to place the Pod on a GPU node and what makes the runtime mount the driver libraries. If you skip the limit, the Pod can land on a CPU-only node and the process will fail at model load time instead of scheduling time. **How to check:** Wait for the Pod to finish, then read the logs. ```bash kubectl wait --for=condition=Succeeded pod/gpu-smoke -n llm --timeout=180s kubectl logs gpu-smoke -n llm ``` Success looks like the familiar `nvidia-smi` table — a "GPU Name", "Driver Version" and "Memory-Usage" header printed inside the Pod log. Then clean up: ```bash kubectl delete pod gpu-smoke -n llm ``` **Example:** if `kubectl logs` prints `NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver`, the GPU is not exposed to containers on that node — fix the node's GPU stack before continuing, because every later step depends on it. ## Step 2. Deploy the model server with a volume, GPU limit and Service ![Illustration: Step 2. Deploy the model server with a volume, GPU limit and Service](https://ai-release.org/guides/img/kak-razvernut-lokalnuyu-ii-model-na-kubernetes-poshagov-en-2.jpg) **What to do:** Create three objects: a PersistentVolumeClaim that holds the model file, a Deployment that runs the inference server against that file, and a ClusterIP Service that gives it a stable DNS name. First, make the weights available. Copy the model into the volume from a temporary Pod, or mount the same PVC from a Job that downloads it — the exact `curl`/`wget` source depends on where you host the file. ```yaml # pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: model-weights namespace: llm spec: accessModes: ["ReadWriteOnce"] storageClassName: # must match a class from `kubectl get storageclass` resources: requests: storage: 50Gi # size to your model file, not to a guess ``` ```bash kubectl apply -f pvc.yaml kubectl get pvc -n llm ``` `STATUS` must be `Bound` before you continue — a `Pending` PVC means the StorageClass name is wrong or no provisioner is available. **Example:** copying the weights with a throwaway Pod that mounts the claim at `/models`: ```bash kubectl run weights-loader -n llm --rm -it --restart=Never \ --image= \ --overrides='{"spec":{"containers":[{"name":"weights-loader","image":"","command":["sh","-c","curl -L -o /models/.gguf && ls -lh /models"],"volumeMounts":[{"name":"w","mountPath":"/models"}]}],"volumes":[{"name":"w","persistentVolumeClaim":{"claimName":"model-weights"}}]}}' ``` Now the Deployment: ```yaml # deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: llama-server namespace: llm spec: replicas: 1 selector: matchLabels: {app: llama-server} template: metadata: labels: {app: llama-server} spec: containers: - name: server image: /llama-server: # verify the image name and tag in the project docs args: - "--model" - "/models/.gguf" - "--host" - "0.0.0.0" - "--port" - "8080" ports: - containerPort: 8080 resources: limits: nvidia.com/gpu: 1 memory: 16Gi cpu: "4" requests: memory: 16Gi cpu: "2" readinessProbe: httpGet: {path: /v1/models, port: 8080} initialDelaySeconds: 30 periodSeconds: 10 volumeMounts: - name: weights mountPath: /models volumes: - name: weights persistentVolumeClaim: claimName: model-weights ``` ```yaml # service.yaml apiVersion: v1 kind: Service metadata: name: llama-server namespace: llm spec: selector: {app: llama-server} ports: - port: 8080 targetPort: 8080 type: ClusterIP ``` ```bash kubectl apply -f deployment.yaml -f service.yaml ``` **Why:** `--host 0.0.0.0` is the load-bearing flag. A server bound to `127.0.0.1` inside the container is unreachable from the Service and from port-forwarding, which produces a connection-refused error that looks like a broken Deployment. The `readinessProbe` on `/v1/models` keeps the Pod out of Service endpoints until the model is actually loaded — loading a large file from network storage can take minutes. **Example:** the Service gives you an in-cluster name, `llama-server.llm.svc.cluster.local:8080`, that other workloads can use without knowing the Pod IP. ## Step 3. Verify inference end to end ![Illustration: Step 3. Verify inference end to end](https://ai-release.org/guides/img/kak-razvernut-lokalnuyu-ii-model-na-kubernetes-poshagov-en-3.jpg) **What to do:** Confirm the Pod is Ready, then call the OpenAI-compatible endpoints from your workstation via port-forward. ```bash kubectl get pods -n llm -l app=llama-server kubectl port-forward -n llm svc/llama-server 8080:8080 ``` Leave the port-forward running and use a second terminal: ```bash curl -s http://localhost:8080/v1/models ``` **Why:** `/v1/models` is the OpenAI-compatible contract, so a 200 with a JSON model list proves both that the process is alive and that any OpenAI SDK or tool can talk to it without custom code. Then send a real completion: ```bash curl -s http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"","messages":[{"role":"user","content":"Reply with the single word: ok"}],"max_tokens":8}' ``` **How to check:** You get a JSON body containing `"choices"` with an assistant message. On screen the first call returns a small `{"object":"list","data":[...]}` payload, the second returns `"content":"ok"` (or a similar short reply). That is a working deployment. **Example:** tooling that speaks the OpenAI API can be pointed at `http://llama-server.llm.svc.cluster.local:8080/v1` from inside the cluster, or at `http://localhost:8080/v1` from your laptop while the port-forward is open — [the Claude Code walkthrough](https://ai-release.org/guides/kak-podklyuchit-lokalnuyu-model-k-claude-code-llama-ser.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) shows that client configuration step by step. To publish it to other clusters or users, replace `ClusterIP` with an `Ingress` in front of the Service; the hostname, TLS secret and controller annotations are cluster-specific — check your ingress controller's documentation. ## Troubleshooting **Problem. Pod stays in `Pending` and never starts** - Error: `0/3 nodes are available: 1 Insufficient nvidia.com/gpu, 2 node(s) didn't match Pod's node affinity/selector.` - Cause: the scheduler found no node with a free `nvidia.com/gpu`, or a `nodeSelector`/taint excludes the GPU node. - Solution: check allocatable GPU per node and what is already consuming it, then free it or add a GPU node. ```bash kubectl describe pod -n llm -l app=llama-server | sed -n '/Events/,$p' kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}' ``` **Problem. Container restarts in a loop with `Back-off restarting failed container`** - Error: `error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory` (visible in `kubectl logs`), while `kubectl describe pod` shows `Limits: nvidia.com/gpu: 0`. - Cause: the Pod was admitted without a GPU, so the NVIDIA runtime never injected the driver libraries — usually because the `resources.limits` block is missing or the manifest was applied to a stale object. - Solution: confirm the limit is present, then re-apply and let the Deployment roll a fresh Pod. ```bash kubectl get deployment llama-server -n llm -o jsonpath='{.spec.template.spec.containers[0].resources}' kubectl rollout restart deployment/llama-server -n llm ``` **Problem. `curl` cannot reach the API after port-forwarding** - Error: `curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server` - Cause: either the port-forward died, the Service selector does not match the Pod labels, or the server is bound to `127.0.0.1` inside the container. - Solution: check that the Service has endpoints and that `--host 0.0.0.0` is in the container args. ```bash kubectl get endpoints -n llm llama-server kubectl get pods -n llm -l app=llama-server --show-labels ``` An empty `ENDPOINTS` value means the selector matches no Ready Pod — fix the labels in `deployment.yaml` or the readiness probe first. ## Summary and next steps You now have a local model served by an OpenAI-compatible API inside Kubernetes, reachable through a stable Service name and verifiable with a single `curl`. Next, move the model path and any API keys into a `Secret`/`ConfigMap`, add an Ingress with TLS, and wire the endpoint into the client tool you already use. ## FAQ **Can I run this guide without a GPU?** Yes, but you must remove the `nvidia.com/gpu` limit and expect CPU inference to be much slower. The rest of the manifest is unchanged; verify the server image supports CPU-only builds in its documentation. **How much GPU memory does my model need?** It depends on the model file and the context length you configure. Check the model card and the inference server's docs for the memory requirement formula, then set `resources.limits.memory` and the GPU count accordingly — do not guess a number. **How do I expose the API outside the cluster?** Create an `Ingress` (or a `LoadBalancer` Service) in front of `llama-server` on port `8080`. Hostnames, TLS certificates and controller annotations are environment-specific — check your ingress controller's documentation. **How do I update the model without downtime?** Upload the new `.gguf` file to the PVC, point `--model` at the new filename, and run `kubectl rollout restart deployment/llama-server -n llm`. The readiness probe keeps traffic off the new Pod until it reports Ready, so the old Pod serves requests until then.