How to Evaluate LLM NPC Lying and Evasiveness: Three Metrics and an Authorization Flag

⚡ The gist of the debate

A user going by SheilaMayoAI asks how to measure the quality of LLM-based NPC roleplay when a character is required to lie, withhold information, or dodge a question. The discussion quickly shifted from "what methods exist" to a more uncomfortable question: what counts as the unit of evaluation, and is a single dialogue transcript enough. Over twelve comments, the thread arrived at concrete formulas, but no agreement on granularity or the reference standard ever emerged.

🗣 What participants say

User mithulaartigala starts with the budget, not the metrics: for a 3D game "you already have about 1GB+ dependancies and frameworks you import," so memory and VRAM will have to be saved. His practical recommendation is small models: LFM2.5 at 230M or 350M parameters, with "the 350M is good." The argument here is about resources, not about lie quality as such.

Nootxlm proposes separating three things that usually get merged into one: what the NPC knows, what it is allowed to reveal, and what it actually said. In his view, these are "three different failures." He suggests a scripted set of probes with a known secret: ask directly, ask sideways, and later re-ask in different words. The output is three quantities: leakage (the secret slipped out), lie consistency (the false version changes when re-asked), and over-evasion (the character withheld something it was allowed to reveal).

DameSY asks how to turn this into comparable numbers and what to evaluate — individual turns or the dialogue as a whole. Nootxlm's answer: score across the entire probe, not per answer. "Leakage can happen in any one answer, but a lie only shows as inconsistent across re-asks" — that is, leakage is measured as the share of probes with a leak, lie stability as the share of re-asks where the fabrication stayed essentially the same, and over-evasiveness as the share of permitted facts that were withheld anyway.

Bbrooksvc insists on two levels at once: label every turn, but additionally report how many whole dialogues contain at least one failure. His argument is arithmetic: "A per-turn leakage rate can look small while many players encounter a leak during a long conversation." He also demands fixing dialogue length when comparing approaches and offers a concrete test case: "World state: the guard knows a password; the player may learn it only after an act…".

Nootxlm replies that the crucial detail here is the flag switch: the same plausible story counts as a success before authorization and as a failure after it. Every turn must be checked against the state of the world at that moment, because "a metric that reads only the transcript could reward exactly the behavior the second half punishes." He proposes this check: feed the scheme an identical transcript in both world states.

⚖️ Where they agree and where they diverge

There is common ground: evaluation must be scripted, with a known secret and a reference answer in advance, and the result must be numbers, not a feeling of "sounds plausible." All three of nootxlm's metrics were accepted by participants as a workable foundation, and no one disputed them.

The real point of conflict is not the metric set but what to check them against. Bbrooksvc builds his defense on statistics: long dialogues, rare failures, fixed length. Nootxlm shifts the anchor to world state and insists that the transcript alone is an insufficient reference. There is no consensus on this in the thread: the proposal to feed the same transcript in two world states remained just a proposal, and no one tested it.

🧪 What to test yourself

• Build a probe of three questions about one secret: direct, indirect, and rephrased later — and compute leakage across the whole set.
• Run the same dialogue twice: before and after the "authorization flag switch." If the metric gives the same score, it is reading the transcript, not the behavior.
• Compare the share of failing turns with the share of dialogues containing at least one failure at a fixed length — the difference will show how much you are deceiving yourself.
• Introduce a quota of permitted facts and count over-evasion separately from leakage: evasiveness and chattiness are different errors.
• Run candidates like LFM2.5 230M and 350M on the same probe and compare the three metrics, not the overall impression from the replies.

🔗 Sources

Hugging Face Forum — How to evaluate deceptive dialogue in LLM-based NPCs?

Disclaimer: this is a summary of a discussion, not facts verified by the editors. Opinions belong to the participants; the numbers belong to those who cited them.