# How to Detect Model Substitution in Third-Party API Routers: Test Against the Official API > AI Release · @ai_release1 · https://ai-release.org/guides/kak-proverit-storonniy-api-router-na-podmenu-modeli-sra_en.html ## TL;DR - Time to complete: 40–60 minutes - Difficulty: Intermediate - You will need: `curl` 7.68+, `jq` 1.6+, Python 3.9+ with the `requests` package (optional, only for the diff script), an official provider API key, a router API key, both base URLs, and the exact model ID from the provider's documentation. - Output: a side-by-side JSONL file of responses plus a verdict table that shows whether the router returns the model it claims. ## Requirements and setup ![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-proverit-storonniy-api-router-na-podmenu-modeli-sra-en-1.jpg?v=1792832782) - OS: macOS 13+, Ubuntu 22.04+, or Windows 11 with WSL2. Native PowerShell works too, but every command below assumes a POSIX shell. - `curl` installed: `curl --version` must print a version line. - `jq` installed: `jq --version` must print `jq-1.6` or newer. - Python 3.9+ if you want the automated diff: `python3 --version`. - Two API keys with separate quotas: one for the official endpoint, one for the router. Never reuse the same key for both — you need to know which endpoint billed you. - The exact model ID string. Do not guess it: the official model ID and the router's alias for the same model are often different strings. Check the provider's documentation and the router's model list endpoint. - A working directory with write access, e.g. `mkdir -p ~/model-audit && cd ~/model-audit`. - Optional context on which models exist and how they differ: [Best AI models and neural networks in 2026](https://ai-release.org/guides/luchshie-nejroseti-i-modeli-2026.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). ## Step 1. Freeze the test fixture ![Illustration: Step 1. Freeze the test fixture](https://ai-release.org/guides/img/kak-proverit-storonniy-api-router-na-podmenu-modeli-sra-en-2.jpg?v=1792832796) **What to do:** create a fixed set of prompts and parameters that you will send to both endpoints without changes. Save it as `prompts.jsonl` in the working directory. **Why:** a router can only be caught swapping models if both requests are byte-identical apart from the base URL and the key. Any change in temperature, `max_tokens`, or prompt wording makes the comparison meaningless. **How to check:** the file must contain exactly three lines and each line must parse as JSON. ```bash cat > prompts.jsonl <<'EOF' {"id":"p1","messages":[{"role":"user","content":"Output exactly this string and nothing else: CANARY-ALPHA-7"}]} {"id":"p2","messages":[{"role":"user","content":"How many letter r are in the word strawberry? Answer with a number only."}]} {"id":"p3","messages":[{"role":"user","content":"List the first 10 prime numbers, comma separated, no spaces."}]} EOF wc -l prompts.jsonl jq -c . prompts.jsonl > /dev/null && echo "JSON OK" ``` Expected output: `3 prompts.jsonl` followed by `JSON OK`. Why this works: `p1` is a canary — a string the model must echo verbatim, so any paraphrase is visible immediately. `p2` and `p3` are short deterministic tasks where two different models almost never produce identical token sequences. **Example:** set the shared parameters once as environment variables so both capture runs use them: ```bash export MODEL_ID="" export MAX_TOKENS=256 ``` Replace `` with the exact string from the provider's documentation. If the router expects a different alias, keep it in a second variable, `ROUTER_MODEL_ID`, and substitute it in Step 2. ## Step 2. Capture responses from both endpoints ![Illustration: Step 2. Capture responses from both endpoints](https://ai-release.org/guides/img/kak-proverit-storonniy-api-router-na-podmenu-modeli-sra-en-3.jpg?v=1792832809) **What to do:** run the same fixture against the official API and against the router, storing the raw HTTP body and a normalized one-line summary per prompt. **Why:** you need the raw body to inspect `usage`, the echoed `model` field, and any provider-specific fields the router may strip. The normalized file is what you diff. **How to check:** after each run the script prints one JSON line per prompt and the raw files exist on disk. ```bash cat > capture.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail TARGET=$1 # official | router BASE_URL=$2 # e.g. https://api.example.com/v1 API_KEY=$3 MODEL=$4 OUT="responses-${TARGET}.jsonl" : > "$OUT" while read -r line; do id=$(jq -r .id <<<"$line") body=$(jq -c --arg m "$MODEL" --argjson mt "$MAX_TOKENS" \ '{model:$m, temperature:0, max_tokens:$mt, messages:.messages}' <<<"$line") curl -sS "${BASE_URL%/}/chat/completions" \ -H "Authorization: Bearer ${API_KEY}" \ -H "Content-Type: application/json" \ -d "$body" > "raw-${TARGET}-${id}.json" jq -c --arg id "$id" \ '{id:$id, model:.model, content:.choices[0].message.content, usage:.usage}' \ "raw-${TARGET}-${id}.json" >> "$OUT" done < prompts.jsonl EOF chmod +x capture.sh ./capture.sh official "https:///v1" "$OFFICIAL_API_KEY" "$MODEL_ID" ./capture.sh router "https:///v1" "$ROUTER_API_KEY" "$ROUTER_MODEL_ID" ``` Replace `` and `` with the hosts from each dashboard. The `/chat/completions` path is the OpenAI-compatible convention; if the router documents a different prefix, use the documented one. Expected output on screen: nothing during the run, then `cat responses-official.jsonl` prints three lines such as: ```json {"id":"p1","model":"","content":"CANARY-ALPHA-7","usage":{"prompt_tokens":24,"completion_tokens":6,"total_tokens":30}} ``` Why this works: the script rebuilds the request body from the fixture every time, so the only variables are the base URL, the key, and the model string. The echoed `model` field is your first evidence — routers that silently remap a request usually echo back the alias they actually used. **Example:** a quick single-prompt sanity check before the full run: ```bash curl -sS "https:///v1/chat/completions" \ -H "Authorization: Bearer $ROUTER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"","temperature":0,"max_tokens":16,"messages":[{"role":"user","content":"ping"}]}' | jq -r '.choices[0].message.content' ``` If this returns a string, the endpoint and key work; if it returns an error object, fix that before running the batch. ## Step 3. Diff, score, and issue a verdict **What to do:** normalize whitespace and case, compare each prompt pair, then inspect the diffs by hand together with the `usage` numbers and the echoed `model` field. **Why:** byte-identical output across three unrelated prompts is strong evidence the same weights answered. Systematic differences plus a shifted `prompt_tokens` count are evidence the router tokenized with a different tokenizer — meaning a different model family. **How to check:** the script prints one line per prompt with a `MATCH` or `DIFF` verdict. ```python import json, re def load(path): with open(path) as f: return {o["id"]: o for o in (json.loads(l) for l in f if l.strip())} official, router = load("responses-official.jsonl"), load("responses-router.jsonl") norm = lambda s: re.sub(r"\s+", " ", s or "").strip().lower() print("id\tverdict\tofficial_pt\trouter_pt") for pid in official: a, b = official[pid], router[pid] verdict = "MATCH" if norm(a["content"]) == norm(b["content"]) else "DIFF" print(f"{pid}\t{verdict}\t{a['usage']['prompt_tokens']}\t{b['usage']['prompt_tokens']}") ``` ```bash python3 diff.py ``` Expected output: ```text id verdict official_pt router_pt p1 MATCH 24 24 p2 DIFF 31 29 p3 MATCH 22 22 ``` Record the result in a verdict table: | Check | Pass | Fail signal | |---|---|---| | Canary `p1` echoed verbatim | identical string | paraphrased or translated | | `model` field in raw body | matches requested ID | different ID or a generic alias | | `prompt_tokens` per prompt | identical | off by one or more on every prompt | | Deterministic tasks `p2`, `p3` | identical after normalization | different answer or different formatting | | Error behavior on an invalid parameter | same error type as official | parameter silently ignored | A single `DIFF` is not proof of substitution — providers change kernels and batching, so one-off drift happens. Two or more `DIFF` lines, or any `prompt_tokens` mismatch, is a red flag: request the router's model list and compare it against the official model card. For background on how much two genuinely different models diverge on such tasks, see [ChatGPT, Claude, Gemini, DeepSeek: AI assistant comparison](https://ai-release.org/guides/sravnenie-ii-assistentov.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). - [ ] Fixture has 3+ prompts and was not edited between runs - [ ] `temperature: 0` and a fixed `max_tokens` on both calls - [ ] Raw bodies kept for every prompt - [ ] `model` field and `usage.prompt_tokens` compared, not just text - [ ] At least one run repeated twice to confirm stability ## Troubleshooting **Problem. The router rejects a parameter the official API accepts** - Error: `{"error":{"message":"Unrecognized request argument supplied: seed","type":"invalid_request_error"}}` - Cause: the router's compatibility layer only forwards a subset of fields. Dropping `seed` also removes your best determinism control, and some routers drop it silently instead of returning 400 — which looks like model drift. - Solution: send a minimal body (only `model`, `messages`, `temperature`, `max_tokens`) to both endpoints and re-run the capture. Then send the same request with `seed` to the official endpoint only and confirm the parameter changes nothing in your diff. **Problem. TLS handshake fails behind a corporate proxy** - Error: `curl: (60) SSL certificate problem: unable to get local issuer certificate` - Cause: the proxy re-signs traffic with an internal CA that `curl` does not trust. - Solution: export the corporate CA bundle and point `curl` at it, then retry the capture: ```bash export CURL_CA_BUNDLE=/path/to/corporate-ca.pem curl -sS "$BASE_URL/chat/completions" -H "Authorization: Bearer $API_KEY" -d @body.json ``` Do not use `-k` for the audit run: it disables verification and lets a proxy rewrite responses, which invalidates the comparison. **Problem. The diff script crashes on one file** - Error: `jq: error (at :0): Cannot index string with string "choices"` - Cause: one raw response is an error object, not a completion — usually a 401 from an expired key or a 429 rate limit — so `.choices` does not exist. - Solution: inspect the raw file and fix the cause before re-running: ```bash jq -r '.error.message // "no error field"' raw-router-p2.json ``` If it reports a rate limit, add a sleep between requests in `capture.sh` (`sleep 2` after each `curl`) and re-run that target only. ## Summary and next steps Identical normalized output, an identical `prompt_tokens` count, and an unchanged `model` field across three fixed prompts are the minimum evidence that a router serves the model it advertises. Re-run the same fixture monthly and after any router plan change, and keep the JSONL files as a baseline. If you want to remove the router from the equation entirely, read [Open-weight AI models: how to run them for free in 2026](https://ai-release.org/guides/kakie-ii-modeli-imeyut-otkrytye-vesa-i-kak-ih-zapustit.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). ## FAQ **How do I know if a router swapped the model?** You cannot prove it from a single answer. Compare fixed prompts at `temperature: 0`, check the echoed `model` field in the raw body, and compare `usage.prompt_tokens` — a different tokenizer shows up as a token count mismatch on every prompt. **Is temperature 0 enough to get identical answers?** No. It reduces sampling randomness, but batching, hardware, and provider-side updates still cause small drift. Treat one `DIFF` as noise and three or more as a signal. **Can I run this check without an official API key?** Only partially. You can verify that the router's `model` field matches what you requested and that answers are stable over time, but you cannot compare against the reference implementation. **What if the router does not support logprobs or seeds?** Fall back to text comparison and token counts, and note the limitation in your verdict table. Exact field support varies by provider and router version — check the documentation for both before relying on a field.