# Designing AI systems: components, diagrams and common pitfalls

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-proektirovat-ii-sistemu_en.html

## TL;DR

- Time to complete: 90 minutes
- Difficulty: Intermediate
- You need: Python 3 with FastAPI; PostgreSQL; Qdrant (vector database — stores embeddings, the numeric representations of text used for similarity search); S3-compatible storage; vLLM if you serve open weights yourself; `curl` and `jq`

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-proektirovat-ii-sistemu-en-1.jpg?v=1793781356)


- A Linux or macOS host with Docker installed, plus a terminal.
- Qdrant and PostgreSQL running locally. Standard ports: `6333` for Qdrant, `5432` for PostgreSQL.
- An S3 bucket for source documents. Use the placeholder `<your-bucket>` and replace it with the real bucket name.
- A model endpoint. Two verified options from the source material: Mistral Large 4 open weights served with vLLM across **eight GPUs** in the official example, or a single **RTX 4070 Ti SUPER with 16 GB** running a 35B model in `Q2_K_XL` quantization at **125 tokens/s**.
- If the model runs behind a hosted API, export the key as `MODEL_API_KEY`. Never hardcode it in the repository.
- Free disk space equal to at least twice the model weight size. Exact sizes are model-specific — **check the documentation** on the model card before pulling weights.

Start the two data services:

```bash
docker run -d --name qdrant -p 6333:6333 qdrant/qdrant
docker run -d --name pg -p 5432:5432 -e POSTGRES_USER=rag -e POSTGRES_PASSWORD=rag -e POSTGRES_DB=rag postgres
```

**How to check:** both commands must return a container ID, and then:

```bash
curl -s http://localhost:6333/healthz
pg_isready -h localhost -p 5432
```

Expected output: `healthz check passed` from Qdrant and `localhost:5432 - accepting connections` from PostgreSQL.

## Step 1. Inventory the components and draw the architecture

**What to do:** list every layer before writing code, then commit the list as `architecture.yaml` in the repository root.

**Why:** downloading and running an LLM on a GPU is only the first step — the model itself does not know the user, the context, or the access rules. A working service is the model plus a harness (the surrounding layer of API, retrieval, tools, limits and monitoring). Every missing layer becomes a production incident, so it has to be named before implementation.

Required layers, taken from the source breakdowns:

| Layer | Component | What breaks without it |
|---|---|---|
| Runtime | LLM on GPU, e.g. Mistral Large 4 via vLLM across 8 GPUs | nothing serves tokens |
| API | FastAPI service, as in the Laya + Gemma 4 example | no interface for clients |
| Retrieval | S3 for documents, PostgreSQL for metadata, Qdrant for the index | the model answers from stale or no data |
| Tools | integrations the model calls to act | the system explains instead of doing |
| Control | request limits and monitoring | silent cost growth and silent failures |
| Agent | planner layer that splits long tasks | tasks overflow the context window |

```yaml
system: support-rag
runtime:
  model: mistral-large-4        # ~1T total parameters, ~52B active per token
  engine: vllm
  gpus: 8                       # count used in the official vLLM example
api:
  framework: fastapi
  port: 8000
  endpoints: [/health, /ask]
retrieval:
  documents: s3://<your-bucket>/prod
  metadata: postgresql://rag:rag@localhost:5432/rag
  vectors: http://localhost:6333/collections/kb
limits:
  max_context_tokens: 65535     # value used in the Tiel-Coder-35B config
monitoring:
  metrics: [/metrics/requests, /metrics/latency]
```

**Example:** keep the agent side of the repository in a fixed folder layout so that each agent has one place for instructions, tools and logs — a working pattern is described in [ICM in practice: a folder structure that keeps agents manageable](https://ai-release.org/guides/icm-praktika-struktura-papok-agenta.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

Also decide at this step whether the project should ship at all. Of **33 corporate AI pilots, only four reach production on average — 88% stay experiments**. If you cannot name the owner, the data source and the success metric for each row of the table above, the pilot is in the 88%.

**How to check:** `yq '.retrieval' architecture.yaml` prints the three storage components, and `grep -c '^  [a-z]' architecture.yaml` returns the number of layers you committed to.

## Step 2. Build the harness — API, retrieval, limits, monitoring
![Illustration: Step 2. Build the harness — API, retrieval, limits, monitoring](https://ai-release.org/guides/img/kak-proektirovat-ii-sistemu-en-2.jpg?v=1793781360)


**What to do:** create the FastAPI service that wraps the model, queries the vector index, reads document versions from PostgreSQL and exposes limits and metrics.

**Why:** this is the layer that turns a bare model into a service. Without it every request is unauthenticated, unmeasured and unbounded in length, and there is no way to tell which document version produced an answer.

```python
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
MAX_CONTEXT = int(os.environ["MAX_CONTEXT_TOKENS"])

class Ask(BaseModel):
    question: str
    top_k: int = 5

@app.get("/health")
def health():
    return {"status": "ok", "index_version": os.environ["INDEX_VERSION"]}

@app.post("/ask")
def ask(req: Ask):
    if len(req.question) > MAX_CONTEXT:
        raise HTTPException(status_code=413, detail="context_limit_exceeded")
    # 1. embed the question  2. search Qdrant  3. load document versions from PostgreSQL
    # 4. call the model endpoint  5. return the answer together with its source versions
    return {"answer": "...", "sources": [], "index_version": os.environ["INDEX_VERSION"]}
```

The `index_version` field is your own payload value written into Qdrant points and into the PostgreSQL metadata table. It is what makes a silent failure visible later.

```bash
export QDRANT_URL=http://localhost:6333
export POSTGRES_DSN=postgresql://rag:rag@localhost:5432/rag
export S3_BUCKET=<your-bucket>
export INDEX_VERSION=2025-09-30
export MAX_CONTEXT_TOKENS=65535
export MODEL_ENDPOINT=http://localhost:8001/v1
uvicorn app:app --host 0.0.0.0 --port 8000
```

For self-hosted weights, start the runtime first so the API has something to call. The official vLLM example spreads Mistral Large 4 over eight GPUs; confirm the exact flags for your model on its card — **check the documentation**:

```bash
vllm serve <model-path> --tensor-parallel-size 8 --port 8001
```

If a 16 GB card is all you have, use a quantized 35B model as in the published Tiel-Coder configuration. The same author splits long tasks through `plan.md` and a light agent such as Pi instead of a full coding agent — an approach compared with the CLI tools in [Claude Code Desktop: setup, limits and how it differs from the CLI](https://ai-release.org/guides/claude-code-desktop-ustanovka.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

Design checklist before you call this step done:

- [ ] `/health` returns `index_version`
- [ ] every answer carries the source document versions
- [ ] token limit enforced with HTTP 413
- [ ] tool calls logged with arguments
- [ ] latency and request counters exported
- [ ] credentials read from environment variables only

**How to check:** `curl -s http://localhost:8000/health | jq` returns `{"status": "ok", "index_version": "2025-09-30"}`. Then send an oversized request — `python -c "print('a'*70000)"` piped into the `/ask` body — and confirm you get `413` with `context_limit_exceeded` instead of a model error.

## Step 3. Verify consistency, recovery and throughput
![Illustration: Step 3. Verify consistency, recovery and throughput](https://ai-release.org/guides/img/kak-proektirovat-ii-sistemu-en-3.jpg?v=1793781364)


**What to do:** run a version-consistency check across all three storage layers, then a recovery drill, then a throughput measurement.

**Why:** the dangerous failure mode is silent. The service stays up and keeps answering, but from an old index — one published case answered “14 days” where the current document said “30”. The unit of recovery is not a single database dump: it is a consistent set of document versions, metadata, the index, code versions and model versions.

```bash
#!/usr/bin/env bash
set -euo pipefail
EXPECTED=$(psql "$POSTGRES_DSN" -tAc "select max(version) from document_versions")
ACTUAL=$(curl -s -X POST "$QDRANT_URL/collections/kb/points/scroll" \
  -H 'content-type: application/json' \
  -d '{"limit":1,"with_payload":["index_version"]}' | jq -r '.result.points[0].payload.index_version')
API=$(curl -s http://localhost:8000/health | jq -r '.index_version')
echo "postgres=$EXPECTED qdrant=$ACTUAL api=$API"
[ "$EXPECTED" = "$ACTUAL" ] && [ "$EXPECTED" = "$API" ] \
  || { echo "FAIL index_version=$ACTUAL expected=$EXPECTED"; exit 1; }
echo "OK"
```

Expected output on a healthy system:

```
postgres=2025-09-30 qdrant=2025-09-30 api=2025-09-30
OK
```

Then measure throughput with a real request:

```bash
time curl -s -X POST http://localhost:8000/ask \
  -H 'content-type: application/json' \
  -d '{"question":"How many days is the return window?","top_k":5}' | jq -r '.answer'
```

**Example:** on an RTX 4070 Ti SUPER with 16 GB, the published Tiel-Coder-35B configuration reaches **125 tokens/s** in `Q2_K_XL` with MoE contributing +200–450%, MTP +20%, and the Sharp Template cutting inference length by 59%. Use that as a calibration point, not a guarantee — your number depends on the model, quant and prompt.

Finally, run the recovery drill: restore the consistent set into a clean environment and repeat the consistency script. Success looks like `OK` again, with the same `index_version` printed by all three layers.

**How to check:** the script exits with code `0` (`echo $?` prints `0`), the answer text matches the current document, and the request time is within your accepted budget.

## Troubleshooting

**Problem. The service answers with outdated data**

- Error: `FAIL index_version=2025-09-14 expected=2025-09-30`
- Cause: only part of the consistent set was restored — the documents and PostgreSQL were refreshed, the Qdrant index was not, so the API kept serving the previous version.
- Solution: reindex from the current S3 documents, write the new `index_version` into the Qdrant payload, restart the API with the matching value, and re-run the script:

```bash
export INDEX_VERSION=2025-09-30
uvicorn app:app --host 0.0.0.0 --port 8000
```

**Problem. The API is unreachable after startup**

- Error: `curl: (7) Failed to connect to localhost port 8000: Connection refused`
- Cause: the process crashed on import, usually because an environment variable is missing, or it bound to a different port.
- Solution: check the process and the port, then read the traceback:

```bash
ss -ltnp | grep 8000
uvicorn app:app --host 0.0.0.0 --port 8000
```

**Problem. The model does not fit on the card**

- Error: `torch.OutOfMemoryError: CUDA out of memory.`
- Cause: the weights are larger than the available VRAM. The Mistral Large 4 example needs eight GPUs; a single 16 GB card cannot hold that model unquantized.
- Solution: switch to a quantized checkpoint such as `Q2_K_XL`, or move the runtime to a host with more GPUs — see the model card for supported configurations.

## Summary and next steps

You now have an architecture file that names every layer, a FastAPI harness with limits and monitoring, and a consistency check that catches silent index drift. Next, turn the consistency script into a cron or CI job and run the recovery drill on a schedule, not only after an incident.

## FAQ

**How many components does a minimal AI system need?**
Five: a model runtime, an API layer, retrieval over your data, tool integrations and a control layer with limits and monitoring. An agent/planner layer is added when tasks are long enough to overflow the context window.

**Do I need a GPU to run an LLM?**
Only if you host the weights yourself. Mistral Large 4 open weights are shown in the official vLLM example spread across eight GPUs, while a 35B model in `Q2_K_XL` runs on one 16 GB card at 125 tokens/s. Otherwise point the API at a hosted endpoint.

**Why does my RAG service answer with old data?**
Because the index, the document metadata and the API configuration drifted apart. Treat document versions, metadata, the index, code versions and model versions as one consistent set, and verify them with a script before every release.

**How do I know an AI pilot is ready for production?**
When each component has an owner, a data source and a measurable success criterion. Of 33 corporate pilots, only four reach production on average, and the 88% that stall usually cannot answer those three questions.