How to Detect Model Substitution in Third-Party API Routers: Test Against the Official API

TL;DR

• Time to complete: 40–60 minutes
• Difficulty: Intermediate
• You will need: curl 7.68+, jq 1.6+, Python 3.9+ with the requests package (optional, only for the diff script), an official provider API key, a router API key, both base URLs, and the exact model ID from the provider's documentation.
• Output: a side-by-side JSONL file of responses plus a verdict table that shows whether the router returns the model it claims.

Requirements and setup

• OS: macOS 13+, Ubuntu 22.04+, or Windows 11 with WSL2. Native PowerShell works too, but every command below assumes a POSIX shell.
• curl installed: curl --version must print a version line.
• jq installed: jq --version must print jq-1.6 or newer.
• Python 3.9+ if you want the automated diff: python3 --version.
• Two API keys with separate quotas: one for the official endpoint, one for the router. Never reuse the same key for both — you need to know which endpoint billed you.
• The exact model ID string. Do not guess it: the official model ID and the router's alias for the same model are often different strings. Check the provider's documentation and the router's model list endpoint.
• A working directory with write access, e.g. mkdir -p ~/model-audit && cd ~/model-audit.
• Optional context on which models exist and how they differ: Best AI models and neural networks in 2026.

Step 1. Freeze the test fixture

What to do: create a fixed set of prompts and parameters that you will send to both endpoints without changes. Save it as prompts.jsonl in the working directory.

Why: a router can only be caught swapping models if both requests are byte-identical apart from the base URL and the key. Any change in temperature, max_tokens, or prompt wording makes the comparison meaningless.

How to check: the file must contain exactly three lines and each line must parse as JSON.

cat > prompts.jsonl <<'EOF'
{"id":"p1","messages":[{"role":"user","content":"Output exactly this string and nothing else: CANARY-ALPHA-7"}]}
{"id":"p2","messages":[{"role":"user","content":"How many letter r are in the word strawberry? Answer with a number only."}]}
{"id":"p3","messages":[{"role":"user","content":"List the first 10 prime numbers, comma separated, no spaces."}]}
EOF
wc -l prompts.jsonl
jq -c . prompts.jsonl > /dev/null && echo "JSON OK"

Expected output: 3 prompts.jsonl followed by JSON OK.

Why this works: p1 is a canary — a string the model must echo verbatim, so any paraphrase is visible immediately. p2 and p3 are short deterministic tasks where two different models almost never produce identical token sequences.

Example: set the shared parameters once as environment variables so both capture runs use them:

export MODEL_ID="<official-model-id>"
export MAX_TOKENS=256

Replace <official-model-id> with the exact string from the provider's documentation. If the router expects a different alias, keep it in a second variable, ROUTER_MODEL_ID, and substitute it in Step 2.

Step 2. Capture responses from both endpoints

What to do: run the same fixture against the official API and against the router, storing the raw HTTP body and a normalized one-line summary per prompt.

Why: you need the raw body to inspect usage, the echoed model field, and any provider-specific fields the router may strip. The normalized file is what you diff.

How to check: after each run the script prints one JSON line per prompt and the raw files exist on disk.

cat > capture.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
TARGET=$1        # official | router
BASE_URL=$2      # e.g. https://api.example.com/v1
API_KEY=$3
MODEL=$4
OUT="responses-${TARGET}.jsonl"
: > "$OUT"
while read -r line; do
  id=$(jq -r .id <<<"$line")
  body=$(jq -c --arg m "$MODEL" --argjson mt "$MAX_TOKENS" \
    '{model:$m, temperature:0, max_tokens:$mt, messages:.messages}' <<<"$line")
  curl -sS "${BASE_URL%/}/chat/completions" \
    -H "Authorization: Bearer ${API_KEY}" \
    -H "Content-Type: application/json" \
    -d "$body" > "raw-${TARGET}-${id}.json"
  jq -c --arg id "$id" \
    '{id:$id, model:.model, content:.choices[0].message.content, usage:.usage}' \
    "raw-${TARGET}-${id}.json" >> "$OUT"
done < prompts.jsonl
EOF
chmod +x capture.sh
./capture.sh official "https://<official-base-url>/v1" "$OFFICIAL_API_KEY" "$MODEL_ID"
./capture.sh router   "https://<router-base-url>/v1"   "$ROUTER_API_KEY"   "$ROUTER_MODEL_ID"

Replace <official-base-url> and <router-base-url> with the hosts from each dashboard. The /chat/completions path is the OpenAI-compatible convention; if the router documents a different prefix, use the documented one.

Expected output on screen: nothing during the run, then cat responses-official.jsonl prints three lines such as:

{"id":"p1","model":"<official-model-id>","content":"CANARY-ALPHA-7","usage":{"prompt_tokens":24,"completion_tokens":6,"total_tokens":30}}

Why this works: the script rebuilds the request body from the fixture every time, so the only variables are the base URL, the key, and the model string. The echoed model field is your first evidence — routers that silently remap a request usually echo back the alias they actually used.

Example: a quick single-prompt sanity check before the full run:

curl -sS "https://<router-base-url>/v1/chat/completions" \
  -H "Authorization: Bearer $ROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"<router-model-alias>","temperature":0,"max_tokens":16,"messages":[{"role":"user","content":"ping"}]}' | jq -r '.choices[0].message.content'

If this returns a string, the endpoint and key work; if it returns an error object, fix that before running the batch.

Step 3. Diff, score, and issue a verdict

What to do: normalize whitespace and case, compare each prompt pair, then inspect the diffs by hand together with the usage numbers and the echoed model field.

Why: byte-identical output across three unrelated prompts is strong evidence the same weights answered. Systematic differences plus a shifted prompt_tokens count are evidence the router tokenized with a different tokenizer — meaning a different model family.

How to check: the script prints one line per prompt with a MATCH or DIFF verdict.

import json, re

def load(path):
    with open(path) as f:
        return {o["id"]: o for o in (json.loads(l) for l in f if l.strip())}

official, router = load("responses-official.jsonl"), load("responses-router.jsonl")
norm = lambda s: re.sub(r"\s+", " ", s or "").strip().lower()

print("id\tverdict\tofficial_pt\trouter_pt")
for pid in official:
    a, b = official[pid], router[pid]
    verdict = "MATCH" if norm(a["content"]) == norm(b["content"]) else "DIFF"
    print(f"{pid}\t{verdict}\t{a['usage']['prompt_tokens']}\t{b['usage']['prompt_tokens']}")

python3 diff.py

Expected output:

id      verdict official_pt router_pt
p1      MATCH   24          24
p2      DIFF    31          29
p3      MATCH   22          22

Record the result in a verdict table:

A single DIFF is not proof of substitution — providers change kernels and batching, so one-off drift happens. Two or more DIFF lines, or any prompt_tokens mismatch, is a red flag: request the router's model list and compare it against the official model card. For background on how much two genuinely different models diverge on such tasks, see ChatGPT, Claude, Gemini, DeepSeek: AI assistant comparison.

• [ ] Fixture has 3+ prompts and was not edited between runs
• [ ] temperature: 0 and a fixed max_tokens on both calls
• [ ] Raw bodies kept for every prompt
• [ ] model field and usage.prompt_tokens compared, not just text
• [ ] At least one run repeated twice to confirm stability

Troubleshooting

Problem. The router rejects a parameter the official API accepts

• Error: {"error":{"message":"Unrecognized request argument supplied: seed","type":"invalid_request_error"}}
• Cause: the router's compatibility layer only forwards a subset of fields. Dropping seed also removes your best determinism control, and some routers drop it silently instead of returning 400 — which looks like model drift.
• Solution: send a minimal body (only model, messages, temperature, max_tokens) to both endpoints and re-run the capture. Then send the same request with seed to the official endpoint only and confirm the parameter changes nothing in your diff.

Problem. TLS handshake fails behind a corporate proxy

• Error: curl: (60) SSL certificate problem: unable to get local issuer certificate
• Cause: the proxy re-signs traffic with an internal CA that curl does not trust.
• Solution: export the corporate CA bundle and point curl at it, then retry the capture:

export CURL_CA_BUNDLE=/path/to/corporate-ca.pem
curl -sS "$BASE_URL/chat/completions" -H "Authorization: Bearer $API_KEY" -d @body.json

Do not use -k for the audit run: it disables verification and lets a proxy rewrite responses, which invalidates the comparison.

Problem. The diff script crashes on one file

• Error: jq: error (at <stdin>:0): Cannot index string with string "choices"
• Cause: one raw response is an error object, not a completion — usually a 401 from an expired key or a 429 rate limit — so .choices does not exist.
• Solution: inspect the raw file and fix the cause before re-running:

jq -r '.error.message // "no error field"' raw-router-p2.json

If it reports a rate limit, add a sleep between requests in capture.sh (sleep 2 after each curl) and re-run that target only.

Summary and next steps

Identical normalized output, an identical prompt_tokens count, and an unchanged model field across three fixed prompts are the minimum evidence that a router serves the model it advertises. Re-run the same fixture monthly and after any router plan change, and keep the JSONL files as a baseline. If you want to remove the router from the equation entirely, read Open-weight AI models: how to run them for free in 2026.

FAQ

How do I know if a router swapped the model?

You cannot prove it from a single answer. Compare fixed prompts at temperature: 0, check the echoed model field in the raw body, and compare usage.prompt_tokens — a different tokenizer shows up as a token count mismatch on every prompt.

Is temperature 0 enough to get identical answers?

No. It reduces sampling randomness, but batching, hardware, and provider-side updates still cause small drift. Treat one DIFF as noise and three or more as a signal.

Can I run this check without an official API key?

Only partially. You can verify that the router's model field matches what you requested and that answers are stable over time, but you cannot compare against the reference implementation.

What if the router does not support logprobs or seeds?

Fall back to text comparison and token counts, and note the limitation in your verdict table. Exact field support varies by provider and router version — check the documentation for both before relying on a field.

🤖 AI summary

• Updated: 2026-10-24
• Topics: проверка api роутера, подмена модели api, как проверить api на подмену, сторонний api gpt сравнение, дешёвый api роутер отзывы
• Summary: Методика проверки дешёвых API-роутеров: контрольные промпты, сверка отпечатков модели и выявление подмены на GPT, Claude или Gemini.
• Key facts:
  • TL;DR Time to complete: 40–60 minutes Difficulty: Intermediate You will need: curl 7.68+, jq 1.6+, Python 3.9+ with the requests package (optional, only for the diff script), an official provider API key, a router API key, both base URLs, and the exact model ID from the provider& x27;s documentation.
  • Output: a side-by-side JSONL file of responses plus a verdict table that shows whether the router returns the model it claims.
  • Requirements and setup Illustration: Requirements and setup OS: macOS 13+, Ubuntu 22.04+, or Windows 11 with WSL2.
  • Native PowerShell works too, but every command below assumes a POSIX shell.
  • curl installed: curl --version must print a version line.
  • jq installed: jq --version must print jq-1.6 or newer.

Related guides

Best AI models and neural networks in 2026

Обзор главных ИИ-моделей года — GPT, Claude, Gemini, DeepSeek и другие: возможности, тесты и ссылки на наши обзоры.

ChatGPT, Claude, Gemini, DeepSeek: AI assistant comparison

Прямое сравнение популярных ИИ-ассистентов: качество ответов, цена, лимиты и для кого что лучше.

How to build an AI shopping assistant on a local model

Практический гайд по созданию локального ИИ-ассистента для магазина на 9B-модели без облачных API.