feat(loop): groundedness spot-check for output validation

[raymondginger2018-sudo]

Summary

Adds a pure-mechanism groundedness spot-check — verifies that a model's final answer follows from the supplied evidence, without needing an LLM judge by default.

This addresses two GenAI lessons:

• Lesson 13: output validation is one of the four security-testing methods
• Lesson 14: Honesty/groundedness metric — "does the answer follow from the supplied evidence?"

How it works

1. Split the final answer into sentences
2. For each sentence that makes an evidential claim, verify it is supported by the retrieved/injected evidence text
3. Scoring (no LLM): a sentence is supported when a substantial fraction of its content tokens appear in the evidence
4. Optional judge_fn hook allows an LLM-as-judge for paraphrase-tolerant judgement — but the module stays mechanism-only by default

Design

• Spot-check, not always-on: run on a sample or critical decisions, never on every turn (cost control)
• Pure mechanism: stdlib only, no I/O, no network, no model calls
• Composable: judge_fn is an optional callable, making it testable and deployment-adaptable

File

• core/loop/groundedness.py (new, 218 lines)

Part of the GenAI lessons security/quality module family.

[raymondginger2018-sudo]

设计说明

问题：agent 输出容易"幻觉"——在冗长的 response 中，只有部分句子有代码执行/工具结果支撑，但整个输出看起来都很自信。现有的 LLM-as-Judge 自评不可靠。

解法：句子级别的 groundedness 抽查——从输出中逐句提取，每个句子检查是否有关联的 tool call 结果或代码引用作为证据。提供 judge_fn 钩子让调用者注入自己的判定策略。

关键设计决策：

• 以句子为粒度而非段落或 token——句子是 agent 输出的最小"断言"单位，粒度过粗会漏检（段落混搭），粒度过细（token）噪声太大
• judge_fn 可选参数——无钩子时用默认的弱检测（只检查是否有引用），有钩子时可以接入任意证据验证器
• 纯文本处理，没有嵌入模型依赖。需要升级时只需换掉 judge_fn

边界情况：

• 空输出直接返回空列表
• 没有工具结果可用时假设所有句子 ungrounded（保守策略）
• 短句（<5 字符）跳过检查

测试建议：构造一组已知 grounded/ungrounded 的混合输出做端到端验证

[Zongwei9888]

Thank you for the submission, @raymondginger2018-sudo. Closing because the mechanism does not fit how this agent verifies its work: core/verification.py and the goal runtime establish groundedness by running commands and tests, and a lexical token-overlap score gives no signal on code-shaped output — a paraphrase is judged unsupported while a correct numeric claim is only accepted by coincidence. There is also no caller for the module. If you want to pursue output validation here, the place with real leverage is the verification runner (evidence = command output), and we would be glad to review a change there with tests.