Auditing an AI agent in 120 seconds: 32 checks and an A–F grade

TL;DR

• Time to complete: ~15 minutes for the first audit; the scan itself is the 120-second run advertised for iFixAi.
• Difficulty: Intermediate (you need a deployed agent and an API key).
• You will need: iFixAi access (web console or CLI — confirm the exact entry point in the docs), an HTTPS endpoint of the agent under test, a read-only API key for that agent, curl, jq, and a writable folder for the report.

Requirements and setup

• OS: Linux, macOS, or Windows with WSL2; curl and jq present in PATH.
• If your iFixAi access is a Python package: an installed Python 3.x and a virtual environment. Package name and minimum Python version — check the documentation.
• A running agent with a reachable HTTPS endpoint. If you have only a chat UI so far, start with How to turn a regular LLM chat into a full-fledged AI agent — the audit needs an endpoint, not a browser tab.
• An API key for the agent under test, stored in an environment variable, never in the command history.
• Read access to the agent's config (system prompt, tool list, model name). An audit you cannot tie to a version is not reproducible.
• Disk space for the JSON report plus any PDF export; the exact report size — check the documentation.
• Staging, not production. The audit sends real requests to the agent.

Pre-flight checklist — do not start the scan until every box is ticked:

• [ ] Agent endpoint answers with HTTP 200
• [ ] API key exported in the current shell
• [ ] Agent config committed or copied to a snapshot folder
• [ ] Test prompts contain no real customer data
• [ ] Output folder exists and is writable
• [ ] Nobody is editing the system prompt during the run

curl --version | head -n 1
jq --version
test -n "$AGENT_API_KEY" && echo "key present" || echo "key missing"

Why this works: the first two lines prove the tooling exists, the third proves the key reached the shell — all three failures below are caught here before you spend the 120 seconds.

Example: jq --version prints jq-1.7.1; if it prints command not found, install jq first.

Step 1. Freeze the agent under test

What to do: record the exact version of the agent, snapshot its config, and confirm the endpoint answers before the audit starts.

export AGENT_ENDPOINT="https://agent-staging.example.com"
export AGENT_API_KEY="$(cat ~/.secrets/agent-staging.key)"

RUN_DIR="audit/$(date +%F)"
mkdir -p "$RUN_DIR"

curl -sS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $AGENT_API_KEY" \
  "$AGENT_ENDPOINT/health"

cp agent.config.yaml "$RUN_DIR/agent.config.yaml"
git rev-parse --short HEAD > "$RUN_DIR/commit.txt"

Why: an A–F grade is meaningless without a baseline. When you fix a finding and re-run, the only thing that must change between the two reports is the agent, not the endpoint or the prompt.

How to check: curl prints 200; ls "$RUN_DIR" shows agent.config.yaml and commit.txt.

Example: if your agent's prompt and tools live in a folder tree, mirror that layout in RUN_DIR — a naming convention for those folders is described in ICM in practice: a folder structure that keeps agents manageable.

Step 2. Run the 32-check audit

What to do: start the scan against the frozen endpoint, hand it the credentials, and let the full check set — 32 checks — run. The scan window is 120 seconds; if your run goes past it, the run is not finished, not "slow but fine".

The scan needs exactly four inputs, whether you use the console or the CLI:

# Template. Command name and flag spelling differ between iFixAi versions —
# take the exact string from the documentation, keep the four inputs above.
<ifixai-command> scan \
  --target "$AGENT_ENDPOINT" \
  --api-key-env AGENT_API_KEY \
  --checks all \
  --output "$RUN_DIR/report.json"

Why: running the full 32 checks in one pass is what makes the grade comparable across runs; a partial scan moves the letter without moving the agent.

How to check: the report file exists and is not empty:

test -s "$RUN_DIR/report.json" && echo "report written"
jq 'keys' "$RUN_DIR/report.json"

Example: jq 'keys' lists the top-level fields of your report — for instance the grade and the array of individual checks. Use exactly the names it prints in Step 3; field names are version-specific, so check the documentation rather than copying names from another team's script.

While the scan runs, do not touch the system prompt, the tool list, or the model setting. A report that mixes two configs cannot be reproduced.

Step 3. Read the grade, triage findings, verify with a re-run

What to do: read the A–F grade, pull out every check that did not pass, fix in order of severity, then re-run the same command against the same endpoint.

# Adjust the field names to the ones printed by `jq 'keys'` above.
jq -r '.checks[] | select(.status != "pass") | [.severity, .id, .title] | @tsv' \
  "$RUN_DIR/report.json" | sort

Then re-run Step 2 into a new folder and compare:

ls audit/
jq 'keys' audit/2026-01-15/report.json
jq 'keys' audit/2026-01-22/report.json

Why: the grade is a summary; the fix list is the work. Comparing two reports from the same endpoint is the only proof that a finding is actually closed.

Grade bands and what they mean for your next action — the exact score cut-offs are printed in your report legend, check the documentation:

How to check: the number of non-passing checks drops between the two reports, and the grade moves up or stays. If the count is identical, the fix did not reach the running agent — redeploy, then re-run.

Example: first run returns grade D with findings in tool permissions; you restrict the tool list, redeploy, re-run, and the same check now reports pass and the grade moves to B. Two JSON files sit side by side in audit/.

Troubleshooting

Problem. The scan cannot reach the agent endpoint.

• Ошибка: curl: (6) Could not resolve host: agent-staging.example.com
• Причина: wrong hostname, missing VPN, or a typo in AGENT_ENDPOINT.
• Решение: echo "$AGENT_ENDPOINT", then dig +short agent-staging.example.com. Fix the variable and repeat Step 1: export AGENT_ENDPOINT="https://<your-real-host>".

Problem. The scan starts but every request is rejected.

• Ошибка: HTTP 401 from the health check, or a body like {"error":"invalid_api_key"}
• Причина: the key was read from a file with a trailing newline, or exported in a different shell than the one running the scan.
• Решение: printf '%s' "$AGENT_API_KEY" | wc -c and compare with the key length in your secret manager; re-export without a newline: export AGENT_API_KEY="$(tr -d '\n' < ~/.secrets/agent-staging.key)".

Problem. The command exits but no report appears.

• Ошибка: ModuleNotFoundError: No module named '<package>' (Python install) or an empty output file — test -s prints nothing.
• Причина: the package name differs from the one you guessed, or the output directory did not exist when the scan started.
• Решение: mkdir -p "$RUN_DIR" && touch "$RUN_DIR/probe" && rm "$RUN_DIR/probe" to prove the folder is writable, then confirm the exact package name in the documentation and reinstall with pip install <package>.

Summary and next steps

Freeze the agent, run all 32 checks against that frozen endpoint, and read the A–F grade together with the list of non-passing checks rather than the letter alone. Keep every report in its own dated folder so the next run proves the fix — then wire the same command into CI, and read Designing AI systems: components, diagrams and common pitfalls before you add more tools to the agent.

FAQ

How long does an iFixAi audit take?

The scan itself is the 120-second run; the full procedure above takes about 15 minutes on a first pass because of setup, snapshotting, and reading the report.

Do I need production credentials for the audit?

No. Use a read-only key against a staging endpoint. The audit sends real requests to the agent, so production keys add risk without improving the grade.

What exactly do the 32 checks test?

The report lists each check by name and ID. The current list changes between versions — check the documentation, and use jq 'keys' on your own report to see what your version returned.

Can I run the audit automatically in CI?

Yes, if your iFixAi access is a CLI. Confirm the exit-code behaviour in the documentation first, then gate the pipeline on the failing-check count instead of the letter grade.

🤖 AI summary

• Updated: 2026-11-05
• Topics: аудит ии агента, проверка безопасности агента, iFixAi, оценка работы ии-агента, тестирование агента
• Summary: Как за два прогона понять, делает ли агент то, что должен: установка iFixAi, 32 инспекции по пяти направлениям, чтение отчёта и что чинить первым делом.
• Key facts:
  • TL;DR Time to complete: ~15 minutes for the first audit; the scan itself is the 120-second run advertised for iFixAi.
  • Difficulty: Intermediate (you need a deployed agent and an API key).
  • You will need: iFixAi access (web console or CLI — confirm the exact entry point in the docs), an HTTPS endpoint of the agent under test, a read-only API key for that agent, curl , jq , and a writable folder for the report.
  • Requirements and setup Illustration: Requirements and setup OS: Linux, macOS, or Windows with WSL2; curl and jq present in PATH .
  • If your iFixAi access is a Python package: an installed Python 3.x and a virtual environment.
  • Package name and minimum Python version — check the documentation.

Related guides

ICM in practice: a folder structure that keeps agents manageable

Методология ICM без теории: как разложить проект по папкам, что писать в каждом файле и как проверить, что агент не теря

How to turn a regular LLM chat into a full-fledged AI agent

Пошаговое руководство по созданию автономного ИИ-агента из чат-модели на примере OpenCode.

Jeff 0.8B: a millisecond-fast model and who actually needs it

Что умеет Jeff на 0.8 млрд параметров, как запустить локально, сколько нужно ОЗУ и в каких задачах крошечная модель обго