Designing AI systems: components, diagrams and common pitfalls
TL;DR
- Time to complete: 90 minutes
- Difficulty: Intermediate
- You need: Python 3 with FastAPI; PostgreSQL; Qdrant (vector database — stores embeddings, the numeric representations of text used for similarity search); S3-compatible storage; vLLM if you serve open weights yourself;
curlandjq
Requirements and setup
- A Linux or macOS host with Docker installed, plus a terminal.
- Qdrant and PostgreSQL running locally. Standard ports:
6333for Qdrant,5432for PostgreSQL. - An S3 bucket for source documents. Use the placeholder
<your-bucket>and replace it with the real bucket name. - A model endpoint. Two verified options from the source material: Mistral Large 4 open weights served with vLLM across eight GPUs in the official example, or a single RTX 4070 Ti SUPER with 16 GB running a 35B model in
Q2_K_XLquantization at 125 tokens/s. - If the model runs behind a hosted API, export the key as
MODEL_API_KEY. Never hardcode it in the repository. - Free disk space equal to at least twice the model weight size. Exact sizes are model-specific — check the documentation on the model card before pulling weights.
Start the two data services:
docker run -d --name qdrant -p 6333:6333 qdrant/qdrant
docker run -d --name pg -p 5432:5432 -e POSTGRES_USER=rag -e POSTGRES_PASSWORD=rag -e POSTGRES_DB=rag postgresHow to check: both commands must return a container ID, and then:
curl -s http://localhost:6333/healthz
pg_isready -h localhost -p 5432Expected output: healthz check passed from Qdrant and localhost:5432 - accepting connections from PostgreSQL.
Step 1. Inventory the components and draw the architecture
What to do: list every layer before writing code, then commit the list as architecture.yaml in the repository root.
Why: downloading and running an LLM on a GPU is only the first step — the model itself does not know the user, the context, or the access rules. A working service is the model plus a harness (the surrounding layer of API, retrieval, tools, limits and monitoring). Every missing layer becomes a production incident, so it has to be named before implementation.
Required layers, taken from the source breakdowns:
| Layer | Component | What breaks without it |
|---|---|---|
| Runtime | LLM on GPU, e.g. Mistral Large 4 via vLLM across 8 GPUs | nothing serves tokens |
| API | FastAPI service, as in the Laya + Gemma 4 example | no interface for clients |
| Retrieval | S3 for documents, PostgreSQL for metadata, Qdrant for the index | the model answers from stale or no data |
| Tools | integrations the model calls to act | the system explains instead of doing |
| Control | request limits and monitoring | silent cost growth and silent failures |
| Agent | planner layer that splits long tasks | tasks overflow the context window |
system: support-rag
runtime:
model: mistral-large-4 # ~1T total parameters, ~52B active per token
engine: vllm
gpus: 8 # count used in the official vLLM example
api:
framework: fastapi
port: 8000
endpoints: [/health, /ask]
retrieval:
documents: s3://<your-bucket>/prod
metadata: postgresql://rag:rag@localhost:5432/rag
vectors: http://localhost:6333/collections/kb
limits:
max_context_tokens: 65535 # value used in the Tiel-Coder-35B config
monitoring:
metrics: [/metrics/requests, /metrics/latency]Example: keep the agent side of the repository in a fixed folder layout so that each agent has one place for instructions, tools and logs — a working pattern is described in ICM in practice: a folder structure that keeps agents manageable.
Also decide at this step whether the project should ship at all. Of 33 corporate AI pilots, only four reach production on average — 88% stay experiments. If you cannot name the owner, the data source and the success metric for each row of the table above, the pilot is in the 88%.
How to check: yq '.retrieval' architecture.yaml prints the three storage components, and grep -c '^ [a-z]' architecture.yaml returns the number of layers you committed to.
Step 2. Build the harness — API, retrieval, limits, monitoring
What to do: create the FastAPI service that wraps the model, queries the vector index, reads document versions from PostgreSQL and exposes limits and metrics.
Why: this is the layer that turns a bare model into a service. Without it every request is unauthenticated, unmeasured and unbounded in length, and there is no way to tell which document version produced an answer.
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
MAX_CONTEXT = int(os.environ["MAX_CONTEXT_TOKENS"])
class Ask(BaseModel):
question: str
top_k: int = 5
@app.get("/health")
def health():
return {"status": "ok", "index_version": os.environ["INDEX_VERSION"]}
@app.post("/ask")
def ask(req: Ask):
if len(req.question) > MAX_CONTEXT:
raise HTTPException(status_code=413, detail="context_limit_exceeded")
# 1. embed the question 2. search Qdrant 3. load document versions from PostgreSQL
# 4. call the model endpoint 5. return the answer together with its source versions
return {"answer": "...", "sources": [], "index_version": os.environ["INDEX_VERSION"]}The index_version field is your own payload value written into Qdrant points and into the PostgreSQL metadata table. It is what makes a silent failure visible later.
export QDRANT_URL=http://localhost:6333
export POSTGRES_DSN=postgresql://rag:rag@localhost:5432/rag
export S3_BUCKET=<your-bucket>
export INDEX_VERSION=2025-09-30
export MAX_CONTEXT_TOKENS=65535
export MODEL_ENDPOINT=http://localhost:8001/v1
uvicorn app:app --host 0.0.0.0 --port 8000For self-hosted weights, start the runtime first so the API has something to call. The official vLLM example spreads Mistral Large 4 over eight GPUs; confirm the exact flags for your model on its card — check the documentation:
vllm serve <model-path> --tensor-parallel-size 8 --port 8001If a 16 GB card is all you have, use a quantized 35B model as in the published Tiel-Coder configuration. The same author splits long tasks through plan.md and a light agent such as Pi instead of a full coding agent — an approach compared with the CLI tools in Claude Code Desktop: setup, limits and how it differs from the CLI.
Design checklist before you call this step done:
- [ ]
/healthreturnsindex_version - [ ] every answer carries the source document versions
- [ ] token limit enforced with HTTP 413
- [ ] tool calls logged with arguments
- [ ] latency and request counters exported
- [ ] credentials read from environment variables only
How to check: curl -s http://localhost:8000/health | jq returns {"status": "ok", "index_version": "2025-09-30"}. Then send an oversized request — python -c "print('a'*70000)" piped into the /ask body — and confirm you get 413 with context_limit_exceeded instead of a model error.
Step 3. Verify consistency, recovery and throughput
What to do: run a version-consistency check across all three storage layers, then a recovery drill, then a throughput measurement.
Why: the dangerous failure mode is silent. The service stays up and keeps answering, but from an old index — one published case answered “14 days” where the current document said “30”. The unit of recovery is not a single database dump: it is a consistent set of document versions, metadata, the index, code versions and model versions.
#!/usr/bin/env bash
set -euo pipefail
EXPECTED=$(psql "$POSTGRES_DSN" -tAc "select max(version) from document_versions")
ACTUAL=$(curl -s -X POST "$QDRANT_URL/collections/kb/points/scroll" \
-H 'content-type: application/json' \
-d '{"limit":1,"with_payload":["index_version"]}' | jq -r '.result.points[0].payload.index_version')
API=$(curl -s http://localhost:8000/health | jq -r '.index_version')
echo "postgres=$EXPECTED qdrant=$ACTUAL api=$API"
[ "$EXPECTED" = "$ACTUAL" ] && [ "$EXPECTED" = "$API" ] \
|| { echo "FAIL index_version=$ACTUAL expected=$EXPECTED"; exit 1; }
echo "OK"Expected output on a healthy system:
postgres=2025-09-30 qdrant=2025-09-30 api=2025-09-30
OKThen measure throughput with a real request:
time curl -s -X POST http://localhost:8000/ask \
-H 'content-type: application/json' \
-d '{"question":"How many days is the return window?","top_k":5}' | jq -r '.answer'Example: on an RTX 4070 Ti SUPER with 16 GB, the published Tiel-Coder-35B configuration reaches 125 tokens/s in Q2_K_XL with MoE contributing +200–450%, MTP +20%, and the Sharp Template cutting inference length by 59%. Use that as a calibration point, not a guarantee — your number depends on the model, quant and prompt.
Finally, run the recovery drill: restore the consistent set into a clean environment and repeat the consistency script. Success looks like OK again, with the same index_version printed by all three layers.
How to check: the script exits with code 0 (echo $? prints 0), the answer text matches the current document, and the request time is within your accepted budget.
Troubleshooting
Problem. The service answers with outdated data
- Error:
FAIL index_version=2025-09-14 expected=2025-09-30 - Cause: only part of the consistent set was restored — the documents and PostgreSQL were refreshed, the Qdrant index was not, so the API kept serving the previous version.
- Solution: reindex from the current S3 documents, write the new
index_versioninto the Qdrant payload, restart the API with the matching value, and re-run the script:
export INDEX_VERSION=2025-09-30
uvicorn app:app --host 0.0.0.0 --port 8000Problem. The API is unreachable after startup
- Error:
curl: (7) Failed to connect to localhost port 8000: Connection refused - Cause: the process crashed on import, usually because an environment variable is missing, or it bound to a different port.
- Solution: check the process and the port, then read the traceback:
ss -ltnp | grep 8000
uvicorn app:app --host 0.0.0.0 --port 8000Problem. The model does not fit on the card
- Error:
torch.OutOfMemoryError: CUDA out of memory. - Cause: the weights are larger than the available VRAM. The Mistral Large 4 example needs eight GPUs; a single 16 GB card cannot hold that model unquantized.
- Solution: switch to a quantized checkpoint such as
Q2_K_XL, or move the runtime to a host with more GPUs — see the model card for supported configurations.
Summary and next steps
You now have an architecture file that names every layer, a FastAPI harness with limits and monitoring, and a consistency check that catches silent index drift. Next, turn the consistency script into a cron or CI job and run the recovery drill on a schedule, not only after an incident.
FAQ
How many components does a minimal AI system need?
Five: a model runtime, an API layer, retrieval over your data, tool integrations and a control layer with limits and monitoring. An agent/planner layer is added when tasks are long enough to overflow the context window.
Do I need a GPU to run an LLM?
Only if you host the weights yourself. Mistral Large 4 open weights are shown in the official vLLM example spread across eight GPUs, while a 35B model in Q2_K_XL runs on one 16 GB card at 125 tokens/s. Otherwise point the API at a hosted endpoint.
Why does my RAG service answer with old data?
Because the index, the document metadata and the API configuration drifted apart. Treat document versions, metadata, the index, code versions and model versions as one consistent set, and verify them with a script before every release.
How do I know an AI pilot is ready for production?
When each component has an owner, a data source and a measurable success criterion. Of 33 corporate pilots, only four reach production on average, and the 88% that stall usually cannot answer those three questions.
🤖 AI summary
- Updated: 2026-11-04
- Topics: ИИ-система, архитектура, RAG, проектирование, продакшен, оценка качества
- Summary: An AI system from task to production: model choice, RAG, memory, evaluation, and the mistakes that keep prototypes from shipping.
- Key facts:
- TL;DR Time to complete: 90 minutes Difficulty: Intermediate You need: Python 3 with FastAPI; PostgreSQL; Qdrant (vector database — stores embeddings, the numeric representations of text used for similarity search); S3-compatible storage; vLLM if you serve open weights yourself; curl and jq Requirements and setup Illustration: Requirements and setup A Linux or macOS host with Docker installed, plus a terminal.
- Qdrant and PostgreSQL running locally.
- Standard ports: 6333 for Qdrant, 5432 for PostgreSQL.
- An S3 bucket for source documents.
- Use the placeholder <your-bucket> and replace it with the real bucket name.
- Two verified options from the source material: Mistral Large 4 open weights served with vLLM across eight GPUs in the official example, or a single RTX 4070 Ti SUPER with 16 GB running a 35B model in Q2_K_XL quantization at 125 tokens/s .