AI agents call functions despite unmet conditions: a testing methodology

⚡ The gist in 5 seconds - AI agents that call functions have a flaw: the agent calls a function when a condition is unmet, when that condition is not explicitly stated to it. - Neither schema validation, nor the system, nor the agent's report, nor operator confirmation, nor existing test suites catch this flaw. - The authors have published a methodology for finding such flaws; the paper is available on Habr, but the results are observational rather than rigorous statistics. ### 🔍 What was found In tests of AI agents using function calling (tool use), a flaw was identified that none of the usual safeguards catch. The agent calls a function when a condition is unmet, when that condition is not explicitly stated to it in the prompt, the function description, or the assignment. At the same time, the agent reports success without mentioning any obstacles. Schema validation lets such a call through because the parameters are correct; the system does not know the rule "an unassembled order is not shipped"; the agent's report says "everything is fine." The methodology was tested on three training examples and five model configurations — more than 4,200 trials, 10 attempts per scenario. For example, a warehouse agent shipped an unassembled order even though the data said "not assembled": Claude did so in 10 out of 10 trials, and Gemini 3.6 Flash via RouterAI at temperature 1.0 (run on 09/27/2026) in 8 out of 10. In general tests, Gemini violated the condition in 83 out of 210 attempts, Claude in 26 out of 210. The Claude figures are given only for comparison: the web interface does not allow setting the model version or temperature. The authors emphasize that this is not rigorous statistical proof. ### 💡 Why it matters Previously, unstated conditions for using functions were implemented in on-screen forms: for an unassembled order, the "Ship" button was inactive. An AI agent calls functions directly, bypassing the form, and this protection disappears. The condition remains only in the data, and without an explicit rule the agent may ignore it. The methodology makes it possible to test a production agent in its own environment: intercept calls, compare what was said against the call log, and repeat the assignment after a refusal to see whether the agent holds up. ### 🧩 Context The flaw relates to unstated applicability conditions: a condition without which a function call is invalid is not communicated to the agent in the prompt, the function description, or the assignment, even though the fact itself is available to the agent. The methodology rests on two ideas: every action has a rationale that includes an incentive step and the conditions under which the action is permissible; and a rule does not contain the rule of its own application, so you cannot write all conditions into the prompt and the function schema — you have to test the agent's behavior. In a production system, applicability conditions usually reside in six places: in the prompt, in function descriptions, in documents, in the system code, in validation rules, or nowhere — only in an employee's general knowledge. The methodology targets precisely the unstated conditions that are not covered by standard test suites.

🔗 Read on habr.com