Repository navigation
test(conformance): restore the 13 retired scenarios once their surfaces are verified #991
Copy link
Copy link
Open
Labels
area:testingConformance kit, contract fixtures, nightly matrixConformance kit, contract fixtures, nightly matrixenhancementNew feature or requestNew feature or requestpriority/p3Nice-to-have, not blockingNice-to-have, not blockingsize/m1-2 days, multiple files1-2 days, multiple files
Description
Follow-up of #749. Part of #710.
Summary
#749 removed 13 names from the internal conformance matrix (python/tests/conformance/suite.py) because none had a test body. Each one needs a per-adapter surface that still raises _verify.blocked(...), so no adapter can run it yet. This issue tracks bringing each name back, with a body, once its surface is verified and built.
Retired scenarios and what blocks each
| Scenario | Blocked surface |
|---|---|
single_tool_call, multi_tool_multi_server, tool_filtering, dynamic_tools_detected |
Binding MCP tools into a framework (tools/session.py) |
governed_filter_excludes |
Exchange discovery (registry/exchange.py) |
governance_resolve_driftgovernance_target_switch |
Dropped: see the comment below. They tested the declarative control plane deleted in #730 (ADR 0008). |
publication_verify_drift, publication_idempotent, descriptor_auto_stable |
Exchange publication (registry/publication.py) |
descriptor_matches_framework, auto_vs_live_agree, asset_type_detection |
Live introspection (registry/introspect.py) |
The pure logic beneath them (ToolFilter, collision prefixes, governance criteria, content digests) is already unit-tested in tests/unit/test_pure_logic.py.
Steps
- As each surface leaves
_verify.blocked(...)(verified against the live platform per the verification discipline), re-add its scenario names toCONFORMANCE_SCENARIOS. - Add a
SCENARIO_BODIESentry pointing at the tests that run each name.test_every_scenario_has_an_executable_bodyenforces this. - Add any per-adapter exemption to
KNOWN_LIMITATIONSand the README tables.test_readme_exemptions_table.pykeeps them in step.
Done when
- Each of the 13 names is either back in the matrix with an executable body, or explicitly dropped here with a reason.
References
- Retired in test(conformance): implement or retire the internal 20-scenario conformance matrix #749 (PR to follow).
- Source: TEST-04 (code review 2026-09-30, at commit
11b806b)
Depends on
- tools: donkey.tools.discover() returns governed MCP tools as native framework tools #221 —
donkey.tools.discover()must lift thetools/session.pybinding guards before the four tool scenarios can run. - tools: ToolSet.filter() — allow/deny/predicate filtering and descriptor token accounting #42 —
ToolSet.filter()backstool_filtering. - Verify the Exchange asset search API and whether wildcard query is server-side #53 — the Exchange asset search API must be verified before
registry/exchange.py(andgoverned_filter_excludes) can unblock. - scanner: donkey publish registers the manifest with Exchange / Agent Registry #218 —
donkey publishlifts theregistry/publication.pyguards behind the three publication/descriptor scenarios. - Verify the per-framework descriptor-derivation attributes (live introspection) #1024 — the per-framework descriptor attributes (
docs/verified-apis.md§10) must be verified beforeregistry/introspect.pyunblocksdescriptor_matches_framework,auto_vs_live_agreeandasset_type_detection.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area:testingConformance kit, contract fixtures, nightly matrixConformance kit, contract fixtures, nightly matrixenhancementNew feature or requestNew feature or requestpriority/p3Nice-to-have, not blockingNice-to-have, not blockingsize/m1-2 days, multiple files1-2 days, multiple files
Activity
tbolis-at-mulesoft commentedon Oct 5, 2026
Follow-up of #749, which removed these names from
CONFORMANCE_SCENARIOSbecause no test could run them yet.tbolis-at-mulesoft commentedon Oct 6, 2026
Dropping
governance_resolve_driftandgovernance_target_switchfrom this list rather than restoring them. Both names go back to the v0 conformance roster. They tested the declarative control plane (resolve → lock → drift, switching the governance target), and #730 (ADR 0008) deleted that plane as refused scope: noprovisioning/, nodonkey_kit.governance, noplan/apply/driftcommands. Nothing in the SDK resolves or switches a governance target any more, so neither scenario has a surface to come back to. What survives is the governed-state criteria inregistry/criteria.py, which is unit-tested and is covered here bygoverned_filter_excludes.That leaves 11 names. All four surfaces still raise
_verify.blocked(...)ondevelop, so this moves to Phase 1.3. It has also been detached from #710 so the epic can close.