Repository navigation
conformance: wire the Phase 2–2.8 scenario bodies as their features land #444
Description
Summary
Split out from #9. The M1 conformance kit has shipped: the scenario
declaration (tests/conformance/suite.py — CONFORMANCE_SCENARIOS +
KNOWN_LIMITATIONS), the LangGraph conformance run, the customer-facing
pytest plugin (donkey_kit/conformance/*, #191), the contract fixtures, and
the nightly matrix (nightly-matrix.yml). This epic tracks the remaining,
still-open work: wiring the scenario bodies for the Phase 2–2.8 scenarios,
each of which is gated on the feature it exercises.
The scenario names already exist in CONFORMANCE_SCENARIOS; what's missing is
each scenario's executable body plus its LangGraph pass and any asserted
KNOWN_LIMITATIONS exemption. Never a silent skip (§8.1).
Motivation
Working instruction #5: build the conformance kit before the second adapter —
done. But a declared scenario with no body is not coverage. Each Phase 2
feature (tools discovery, scanner/publish, adapter descriptor) must land its
matching conformance scenario in the same slice, or the kit's credibility
erodes into a list of names. This epic keeps that lockstep visible.
Scope — scenario bodies to wire, grouped by the feature that gates them
- Tools (BG §2.7, gated on tools: donkey.tools.discover() returns governed MCP tools as native framework tools #221) —
single_tool_call,
multi_tool_multi_server,tool_filtering,governed_filter_excludes - Governance drift (M2.5) —
governance_resolve_drift,
governance_target_switch - Publication (M2.7, scanner/publish scanner: donkey scan derives a deterministic scan model of agents, tools and MCP servers #217/scanner: donkey publish registers the manifest with Exchange / Agent Registry #218) —
publication_verify_drift,publication_idempotent - Descriptor / auto-detection (M2.8, BG §2.8) —
descriptor_auto_stable,
descriptor_matches_framework,auto_vs_live_agree,
dynamic_tools_detected,asset_type_detection
Each item wires the body, adds the LangGraph pass, and records any framework
exemption in KNOWN_LIMITATIONS with a stated cause.
Cross-surface impact
- core/llm/registry/tools/integrations: touches
tools/andintegrations/
test coverage as those features land; no production-code change in this epic
itself (the feature issues own that). - TypeScript parity (Phase 5, typescript: shared conformance scenarios across Python and TypeScript #243): the shared-scenario TS port is a
separate Phase 5 issue; not affected here beyond keeping the scenario list
the future source of truth. - Provisioning CLI:
donkey publish/ scanner scenarios (publication_*)
exercise CLI behavior; wired when scanner: donkey scan derives a deterministic scan model of agents, tools and MCP servers #217/scanner: donkey publish registers the manifest with Exchange / Agent Registry #218 land, not here. - Nextra docs site: the README
KNOWN_LIMITATIONSexemption table grows as
scenarios wire — keep it in sync (ddk-docs-sync). - docs/verified-apis.md: several scenarios (tools discovery) are
blocked-on-verification; wiring waits on the matching row flip, not this epic.
Out of scope
- The M1 kit already delivered (closed as Epic: Testing — the adapter conformance kit, contract fixtures, nightly matrix #9).
- The customer-facing plugin's own scenario set (testing: the pytest plugin and the conformance suite customers run against their own agent #191, shipped).
--donkey-mode serverplugin path (testing: --donkey-mode server path for the conformance plugin #322) — its own issue.- TypeScript shared-scenario parity (typescript: shared conformance scenarios across Python and TypeScript #243, Phase 5).
Depends on
- tools: donkey.tools.discover() returns governed MCP tools as native framework tools #221 — tools discovery (gates the four tool scenarios), blocked-on-verification
- scanner: donkey scan derives a deterministic scan model of agents, tools and MCP servers #217 / scanner: donkey publish registers the manifest with Exchange / Agent Registry #218 — scanner scan/publish (gates the publication scenarios)
Related
- Epic: Testing — the adapter conformance kit, contract fixtures, nightly matrix #9 — the M1 conformance-kit epic this splits out of (to be closed as delivered)
- testing: --donkey-mode server path for the conformance plugin #322 — server-path conformance plugin
- typescript: shared conformance scenarios across Python and TypeScript #243 — TypeScript shared conformance scenarios (Phase 5)
Activity