How to Survive an LLM Deprecation: Migrating Your AI Agent to a New Model
This guide is for engineers who run an AI agent — an agent is a program where an LLM (large language model, the text-generating neural network) plans steps, calls tools and keeps memory — and whose provider has just announced that the model behind that agent is being retired. The real problem is not "pick a new model": it is that every system prompt, tool schema, JSON format and RAG step (RAG = the agent first searches your documents, then answers from them) was tuned to the quirks of the old model. The order of steps below works with any provider and any framework, and it lets you finish the migration before the old endpoint starts returning errors.
Requirements and preparation
- A staging environment that mirrors production (same container, same env vars), separate from live traffic.
- API keys for **both** models: the old one (still alive) and the replacement.
- A frozen evaluation set: 50–100 real user requests with the expected answer or a pass/fail criterion for each.
- Version control (Git) and a rollback plan — one env var you can flip back.
- Basic observability: request logs (the raw prompt + raw response), latency and error-rate metrics.
- Python 3.10+ (or Node 18+) with an HTTP client, plus `jq` for reading API responses in the terminal.
- The new model's docs open in a tab: context window (how many tokens — word pieces — it can hold at once), tool-calling support, JSON mode, tokenizer.
- One config file that holds the model name — if you don't have one yet, step 2 creates it.
Step-by-step instructions
1. Find every hardcoded model name. Run a repository-wide search before touching anything:
grep -rn "gpt-4o\|claude-3\|model=" . --include="*.py" --include="*.yaml" --include="*.env*"Expected result: a list of files and line numbers. Why it works: one forgotten string in a Dockerfile or a cron job means half your traffic keeps calling a dead endpoint while your tests are green.
2. Put the model name in one place. Create an adapter — a thin wrapper that all your code calls instead of the provider SDK directly:
# llm_client.py
import os
from openai import OpenAI
MODEL = os.getenv("LLM_MODEL", "old-model-name")
client = OpenAI(base_url=os.getenv("LLM_BASE_URL"))
def chat(messages, tools=None):
return client.chat.completions.create(model=MODEL, messages=messages, tools=tools)Expected result: `grep -rn "old-model-name" .` now returns only `llm_client.py` and `.env`. Why it works: switching models becomes an env-var change, not a code refactor.
3. Freeze a baseline **before** the old model disappears. Run your evaluation set against it and save the raw answers:
python run_eval.py --model old-model-name --out baseline.json
jq 'length' baseline.json # -> 87Expected result: `baseline.json` with 87 records, each containing the request and the old model's answer. Why it works: after the old model is gone you can no longer produce a fair "before" picture, and "the new model feels worse" is not a bug report.
4. Test the replacement on the three things agents actually break on: tool calling, system-prompt obedience, and structured output. Send the same tricky request to the new model three times and check the raw JSON.
curl -s $LLM_BASE_URL/chat/completions -H "Authorization: Bearer $NEW_KEY" \
-d '{"model":"new-model-name","messages":[{"role":"user","content":"Weather in Berlin? Use the tool."}],"tools":[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}]}'Expected result: a `tool_calls` array with `name: "get_weather"` and `arguments: {"city":"Berlin"}`. Why it works: if the new model answers in plain text instead, no amount of prompt polish downstream will save the agent.
5. Fill in a second config and run it in staging:
# config.yaml
llm:
base_url: "https://api.example.com/v1"
model: "new-model-name"
max_tokens: 1024
temperature: 0.2Expected result: staging starts and `/health` reports the new model name. Why it works: keeping both configs side by side makes rollback a one-line revert.
6. Run the regression and compare. Score the new answers with the same script and diff the numbers:
python run_eval.py --model new-model-name --out candidate.json
python compare.py baseline.json candidate.json
# tool-call accuracy: 0.94 -> 0.91
# JSON valid rate: 1.00 -> 0.78Expected result: a per-metric table. Why it works: a drop in one metric (usually JSON validity) points at a specific fix, not a general "new model is bad".
7. Fix prompt drift. Prompt drift = the same prompt produces different behaviour after the model swap, because the new model reads formatting differently. Add one explicit line and one example rather than rewriting everything: "Always answer with a single JSON object. No markdown fences." Then re-run step 6.
Expected result: JSON valid rate back to 0.98–1.00. Why it works: most drift is format, not reasoning.
8. Roll out gradually with a traffic split. Route 5% of requests to the new model, then 25%, then 100%, watching error rate and latency for an hour at each step.
export LLM_TRAFFIC_NEW=5 # then 25, then 100Expected result: error rate stays flat and p95 latency (the time 95% of requests finish within) stays inside your budget. Why it works: a bad model swap hurts a fraction of users, not all of them.
9. Verify the cost and token maths. Cost = input tokens × price per 1M input tokens + output tokens × price per 1M output tokens, using your provider's current price list.
Expected result: a two-number comparison (old vs new) per 1,000 agent runs. Why it works: a cheaper model with longer, chattier answers can be more expensive overall.
10. Clean up and set alerts. Delete the old key, remove the old config, and add alerts on error rate, p95 latency and malformed-JSON count.
Expected result: `grep -rn "old-model-name" .` returns nothing outside Git history. Why it works: dead config is how teams accidentally migrate twice.
Common problems and solutions
- **Symptom:** the agent stops calling tools and replies with plain text like "I will check the weather for you." In the raw response you see `"tool_calls": null` or the field missing. **Fix:** confirm the new model supports tool calling at all; if it does, pass `tool_choice="auto"` explicitly, shorten the tool description to one sentence, and re-test with the `curl` command from step 4. If it still fails, keep the old model as a fallback for tool-calling turns.
- **Symptom:** answers are cut off mid-sentence; the response ends with `"finish_reason": "length"`. **Fix:** raise `max_tokens` in `config.yaml`, or trim the conversation history — older turns eat the context window (the total tokens the model can hold). Check the new model's context size: a model with a smaller window will silently truncate long RAG contexts.
- **Symptom:** your parser crashes with `json.decoder.JSONDecodeError: Expecting value: line 1 column 1`, and the raw text starts with ```` ```json ````. **Fix:** enable the provider's JSON mode if it exists, add "no markdown fences" to the system prompt, and strip fences defensively before parsing. Then re-run `compare.py` and confirm the JSON valid rate.
FAQ
How do I know which replacement model to choose?
Shortlist candidates by the three hard requirements first — tool calling, JSON output, and a context window at least as large as the old one. A broad overview of what is available is in Best AI models and neural networks in 2026.
Will swapping the model break my RAG pipeline?
Only if you also change the embedding model (the network that turns text into vectors for search). Keep the embedding model and vector store untouched during the migration, rerun the evaluation set, and check that retrieved document IDs are identical to the baseline.
Can I keep an old model running myself instead of migrating?
Sometimes, yes — if you have the hardware, a self-hosted model removes the deprecation deadline entirely. Setup basics, including VRAM and quantisation, are covered in How to run AI models locally on your PC: a beginner's guide.
Do I have to rewrite all prompts from scratch?
No. Change the model first, run the regression, and only patch the specific prompts that fail. Rewriting everything at once makes it impossible to tell whether the model or your new prompt caused the change.
How fast can I roll back?
As fast as your env var changes: set `LLM_MODEL` back to the old name and restart. Keep the old key valid until the new model has served 100% of traffic for at least a week.
🤖 AI summary
- Updated: 2026-10-15
- Topics: снятие модели с продакшена, модель устарела что делать, миграция на новую LLM, deprecation API модели, версионирование промптов
- Summary: Пошаговый план миграции ИИ-агента, когда провайдер отключает модель: аудит промптов, прогон тестов, роутинг запросов и подстраховка на локальной LLM.
- Key facts:
- This guide is for engineers who run an AI agent — an agent is a program where an LLM (large language model, the text-generating neural network) plans steps, calls tools and keeps memory — and whose provider has just announced that the model behind that agent is being retired.
- The real problem is not "pick a new model": it is that every system prompt, tool schema, JSON format and RAG step (RAG = the agent first searches your documents, then answers from them) was tuned to the quirks of the old model.
- The order of steps below works with any provider and any framework, and it lets you finish the migration before the old endpoint starts returning errors.
- Requirements and preparation Illustration: Requirements and preparation A staging environment that mirrors production (same container, same env vars), separate from live traffic.
- API keys for both models: the old one (still alive) and the replacement.
- A frozen evaluation set: 50–100 real user requests with the expected answer or a pass/fail criterion for each.