Provenance vs. Fabrication: How to Keep a Made-Up Value Out of a Real Call

⚡ The Core of the Debate

Jang-woo opened the thread: confirmations, schemas, and guardrails look at a value only after it has already arrived, and only at the value itself — an account number typed by a human and one invented by a model are the same string. As long as the model and the agent are treated as one entity, the question becomes "how much can you trust the model," and the answer leads to a stronger model or a stricter prompt. Participants discuss whether the source of a value and its executor can be separated at all. The trigger is not hypothetical: the post cites the OpenAI and Georgia Tech paper "Why Language Models Hallucinate," in which a model was asked three times for a birth date in DD-MM format and gave three different wrong answers.

🗣 What Participants Say

SalutefaceAAC calls the principle "values come from sources" a provenance chain and agrees that agentic frameworks mostly skip it. According to him, structured outputs and JSON mode force the model to produce values within a schema, but don't track where those values came from: "your 'check the source' step is the missing piece." He also references the OWASP LLM Top 10 — in the quoted text, the mention cuts off at the word "excessi."

Timothy-rogers supports the idea that an agent's task should be defined outside the model, but offers a different framing: "a stability problem rather than a problem of determinative control." The goal is a stably defined executive role for the agent, because the model by its nature redefines boundaries unpredictably.

Jang-woo responds in agreement: the model can indeed redefine boundaries at any moment, and on the model's side this can't be fixed by engineering. It becomes critical at tool selection — having redefined the user's task, the model will pick a different tool, and with it a different checklist. The defense, in his view, is more conversation and confirmations before the tool is locked in.

In response to the sources schema, Jang-woo clarifies: declaring allowed sources per argument is close to what the specification proposes, but it's dangerous to trust the model to define provenance. If the source tag comes from the model's output or from a framework that believes its claims, a fabricated value will get a clean tag. That's why, as he writes, the specification doesn't have the model supply provenance at all.

orke-ai separates provenance from freshness: a pointer may locate the correct source at validation time, but by the time the tool executes, the source may have changed. He proposes binding the validated value to a version or hash and requiring re-verification at dispatch — this closes the time-of-check/time-of-use gap without making the model judge its own evidence. He notes he works in this field, but the text cuts off.

Jang-woo replies that provenance and freshness are different tasks, but they also have different owners. Conditions declared by the user — typed values, defined scope, pre-approved settings — remain on the user's responsibility: they don't drift on their own, and if they changed, it means the user changed them, and that's a new declaration.

⚖️ Where They Agree, Where They Diverge

There is a common denominator: a value without a source should never reach a real call, and a single glance at the string isn't enough for that. All five replies in the thread agree on this, including the post's author. Nobody disputes either that current schemas and guardrails check form, not origin.

The real point of conflict is who owns the source and its changes. Jang-woo insists the model should not supply provenance, and that drift in user-declared conditions is not a system problem. orke-ai adds a dimension missing from the original framing: a validated value can go stale between check and call. Timothy-rogers steers the debate toward the stability of the agent's role. No consensus was reached in the thread on who is responsible for a source changing between validation and the call.

🧪 What to Check Yourself

• Look at your agent's argument schema: is there a source field for each argument, and which arguments come only from user input.
• Repeat the test from the post: ask the model for a date in DD-MM format three times in a row with the same prompt and compare the answers.
• Check who stamps the provenance tag. If it comes from the model's output, a test with a deliberately fabricated value will show a clean tag on the fabrication.
• Reproduce the check-vs-use gap: validate a value, change the source, call the tool — and see whether re-verification happens.
• Compare against the OWASP LLM Top 10 and what JSON mode provides: it constrains the form of values, but not their origin.

🔗 Sources

Hugging Face Forum — How to keep what the model makes up out of the agent

Disclaimer: this is a summary of a discussion, not facts verified by the editors. Opinions belong to the participants; figures belong to those who cited them.