← All guides AI Release
RU Subscribe to @ai_release1

How to Deploy a Local LLM on Kubernetes: Step-by-Step Guide

Updated: 21.10.2026 · AI Release · @ai_release1
How to Deploy a Local LLM on Kubernetes: Step-by-Step Guide

TL;DR

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

Pre-flight checklist — run each line and confirm the result before continuing:

Example: the fourth command above prints gpu-node-1 1 when the node advertises exactly one GPU, and gpu-node-1 (empty field) when the device plugin is not installed.

If you are still deciding whether the workload belongs in a cluster or on a single box, the self-hosted AI on a home server guide covers the single-machine path; Kubernetes is worth it when you need scheduling, rolling updates, and a stable in-cluster DNS name.

Step 1. Create the namespace and prove the GPU is schedulable

What to do: Create a dedicated namespace and run a one-shot Pod that calls nvidia-smi. This isolates the AI workload from system pods and proves the GPU is actually injected into containers, not just visible on the host.

kubectl create namespace llm
# gpu-smoke.yaml
apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke
  namespace: llm
spec:
  restartPolicy: Never
  containers:
    - name: gpu-smoke
      image: <cuda-base-image>   # any image with nvidia-smi; check the toolkit docs for a current tag
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
kubectl apply -f gpu-smoke.yaml

Why: Kubernetes schedules GPUs as an extended resource. Requesting nvidia.com/gpu: 1 in limits is what tells the scheduler to place the Pod on a GPU node and what makes the runtime mount the driver libraries. If you skip the limit, the Pod can land on a CPU-only node and the process will fail at model load time instead of scheduling time.

How to check: Wait for the Pod to finish, then read the logs.

kubectl wait --for=condition=Succeeded pod/gpu-smoke -n llm --timeout=180s
kubectl logs gpu-smoke -n llm

Success looks like the familiar nvidia-smi table — a "GPU Name", "Driver Version" and "Memory-Usage" header printed inside the Pod log. Then clean up:

kubectl delete pod gpu-smoke -n llm

Example: if kubectl logs prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, the GPU is not exposed to containers on that node — fix the node's GPU stack before continuing, because every later step depends on it.

Step 2. Deploy the model server with a volume, GPU limit and Service

Illustration: Step 2. Deploy the model server with a volume, GPU limit and Service
Illustration: Step 2. Deploy the model server with a volume, GPU limit and Service

What to do: Create three objects: a PersistentVolumeClaim that holds the model file, a Deployment that runs the inference server against that file, and a ClusterIP Service that gives it a stable DNS name.

First, make the weights available. Copy the model into the volume from a temporary Pod, or mount the same PVC from a Job that downloads it — the exact curl/wget source depends on where you host the file.

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights
  namespace: llm
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: <your-storage-class>   # must match a class from `kubectl get storageclass`
  resources:
    requests:
      storage: 50Gi                          # size to your model file, not to a guess
kubectl apply -f pvc.yaml
kubectl get pvc -n llm

STATUS must be Bound before you continue — a Pending PVC means the StorageClass name is wrong or no provisioner is available.

Example: copying the weights with a throwaway Pod that mounts the claim at /models:

kubectl run weights-loader -n llm --rm -it --restart=Never \
  --image=<image-with-curl> \
  --overrides='{"spec":{"containers":[{"name":"weights-loader","image":"<image-with-curl>","command":["sh","-c","curl -L -o /models/<your-model-file>.gguf <model-url> && ls -lh /models"],"volumeMounts":[{"name":"w","mountPath":"/models"}]}],"volumes":[{"name":"w","persistentVolumeClaim":{"claimName":"model-weights"}}]}}'

Now the Deployment:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-server
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels: {app: llama-server}
  template:
    metadata:
      labels: {app: llama-server}
    spec:
      containers:
        - name: server
          image: <registry>/llama-server:<tag>   # verify the image name and tag in the project docs
          args:
            - "--model"
            - "/models/<your-model-file>.gguf"
            - "--host"
            - "0.0.0.0"
            - "--port"
            - "8080"
          ports:
            - containerPort: 8080
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 16Gi
              cpu: "4"
            requests:
              memory: 16Gi
              cpu: "2"
          readinessProbe:
            httpGet: {path: /v1/models, port: 8080}
            initialDelaySeconds: 30
            periodSeconds: 10
          volumeMounts:
            - name: weights
              mountPath: /models
      volumes:
        - name: weights
          persistentVolumeClaim:
            claimName: model-weights
# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: llama-server
  namespace: llm
spec:
  selector: {app: llama-server}
  ports:
    - port: 8080
      targetPort: 8080
  type: ClusterIP
kubectl apply -f deployment.yaml -f service.yaml

Why: --host 0.0.0.0 is the load-bearing flag. A server bound to 127.0.0.1 inside the container is unreachable from the Service and from port-forwarding, which produces a connection-refused error that looks like a broken Deployment. The readinessProbe on /v1/models keeps the Pod out of Service endpoints until the model is actually loaded — loading a large file from network storage can take minutes.

Example: the Service gives you an in-cluster name, llama-server.llm.svc.cluster.local:8080, that other workloads can use without knowing the Pod IP.

Step 3. Verify inference end to end

Illustration: Step 3. Verify inference end to end
Illustration: Step 3. Verify inference end to end

What to do: Confirm the Pod is Ready, then call the OpenAI-compatible endpoints from your workstation via port-forward.

kubectl get pods -n llm -l app=llama-server
kubectl port-forward -n llm svc/llama-server 8080:8080

Leave the port-forward running and use a second terminal:

curl -s http://localhost:8080/v1/models

Why: /v1/models is the OpenAI-compatible contract, so a 200 with a JSON model list proves both that the process is alive and that any OpenAI SDK or tool can talk to it without custom code. Then send a real completion:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-name-from-/v1/models>","messages":[{"role":"user","content":"Reply with the single word: ok"}],"max_tokens":8}'

How to check: You get a JSON body containing "choices" with an assistant message. On screen the first call returns a small {"object":"list","data":[...]} payload, the second returns "content":"ok" (or a similar short reply). That is a working deployment.

Example: tooling that speaks the OpenAI API can be pointed at http://llama-server.llm.svc.cluster.local:8080/v1 from inside the cluster, or at http://localhost:8080/v1 from your laptop while the port-forward is open — the Claude Code walkthrough shows that client configuration step by step.

To publish it to other clusters or users, replace ClusterIP with an Ingress in front of the Service; the hostname, TLS secret and controller annotations are cluster-specific — check your ingress controller's documentation.

Troubleshooting

Problem. Pod stays in Pending and never starts

kubectl describe pod -n llm -l app=llama-server | sed -n '/Events/,$p'
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Problem. Container restarts in a loop with Back-off restarting failed container

kubectl get deployment llama-server -n llm -o jsonpath='{.spec.template.spec.containers[0].resources}'
kubectl rollout restart deployment/llama-server -n llm

Problem. curl cannot reach the API after port-forwarding

kubectl get endpoints -n llm llama-server
kubectl get pods -n llm -l app=llama-server --show-labels

An empty ENDPOINTS value means the selector matches no Ready Pod — fix the labels in deployment.yaml or the readiness probe first.

Summary and next steps

You now have a local model served by an OpenAI-compatible API inside Kubernetes, reachable through a stable Service name and verifiable with a single curl. Next, move the model path and any API keys into a Secret/ConfigMap, add an Ingress with TLS, and wire the endpoint into the client tool you already use.

FAQ

Can I run this guide without a GPU?

Yes, but you must remove the nvidia.com/gpu limit and expect CPU inference to be much slower. The rest of the manifest is unchanged; verify the server image supports CPU-only builds in its documentation.

How much GPU memory does my model need?

It depends on the model file and the context length you configure. Check the model card and the inference server's docs for the memory requirement formula, then set resources.limits.memory and the GPU count accordingly — do not guess a number.

How do I expose the API outside the cluster?

Create an Ingress (or a LoadBalancer Service) in front of llama-server on port 8080. Hostnames, TLS certificates and controller annotations are environment-specific — check your ingress controller's documentation.

How do I update the model without downtime?

Upload the new .gguf file to the PVC, point --model at the new filename, and run kubectl rollout restart deployment/llama-server -n llm. The readiness probe keeps traffic off the new Pod until it reports Ready, so the old Pod serves requests until then.

🤖 AI summary

Related guides

Step-by-Step Guide: How to Deploy a Self-Hosted Lampline Voice Agent and Import from Dialogflow CX

Пошаговая инструкция по установке Lampline, настройке голосового пайплайна и импорту готового агента из Dialogflow CX в

How to Download and Install AI Apps on Android via RuStore: Step-by-Step Guide

Инструкция по установке ИИ-приложений из RuStore на Android: поиск, скачивание, обход ограничений и настройка.

How to add AI to your home server: self-hosted services guide

Разбираем, как развернуть на домашнем сервере ИИ-сервисы для автоматизации, поиска и управления задачами без облаков и п