Skip to content

test(conformance): restore the 13 retired scenarios once their surfaces are verified #991

Open

Description

@tbolis-at-mulesoft

Follow-up of #749. Part of #710.

Summary

#749 removed 13 names from the internal conformance matrix (python/tests/conformance/suite.py) because none had a test body. Each one needs a per-adapter surface that still raises _verify.blocked(...), so no adapter can run it yet. This issue tracks bringing each name back, with a body, once its surface is verified and built.

Retired scenarios and what blocks each

Scenario Blocked surface
single_tool_call, multi_tool_multi_server, tool_filtering, dynamic_tools_detected Binding MCP tools into a framework (tools/session.py)
governed_filter_excludes Exchange discovery (registry/exchange.py)
governance_resolve_drift, governance_target_switch Dropped: see the comment below. They tested the declarative control plane deleted in #730 (ADR 0008).
publication_verify_drift, publication_idempotent, descriptor_auto_stable Exchange publication (registry/publication.py)
descriptor_matches_framework, auto_vs_live_agree, asset_type_detection Live introspection (registry/introspect.py)

The pure logic beneath them (ToolFilter, collision prefixes, governance criteria, content digests) is already unit-tested in tests/unit/test_pure_logic.py.

Steps

  • As each surface leaves _verify.blocked(...) (verified against the live platform per the verification discipline), re-add its scenario names to CONFORMANCE_SCENARIOS.
  • Add a SCENARIO_BODIES entry pointing at the tests that run each name. test_every_scenario_has_an_executable_body enforces this.
  • Add any per-adapter exemption to KNOWN_LIMITATIONS and the README tables. test_readme_exemptions_table.py keeps them in step.

Done when

  • Each of the 13 names is either back in the matrix with an executable body, or explicitly dropped here with a reason.

References

Depends on

Activity

  1. added
    enhancementNew feature or request
    area:testingConformance kit, contract fixtures, nightly matrix
    size/m1-2 days, multiple files
    priority/p3Nice-to-have, not blocking
    on Oct 5, 2026
  2. tbolis-at-mulesoft commented on Oct 5, 2026

    @tbolis-at-mulesoft
    CollaboratorAuthor

    Follow-up of #749, which removed these names from CONFORMANCE_SCENARIOS because no test could run them yet.

  3. tbolis-at-mulesoft commented on Oct 6, 2026

    @tbolis-at-mulesoft
    CollaboratorAuthor

    Dropping governance_resolve_drift and governance_target_switch from this list rather than restoring them. Both names go back to the v0 conformance roster. They tested the declarative control plane (resolve → lock → drift, switching the governance target), and #730 (ADR 0008) deleted that plane as refused scope: no provisioning/, no donkey_kit.governance, no plan/apply/drift commands. Nothing in the SDK resolves or switches a governance target any more, so neither scenario has a surface to come back to. What survives is the governed-state criteria in registry/criteria.py, which is unit-tested and is covered here by governed_filter_excludes.

    That leaves 11 names. All four surfaces still raise _verify.blocked(...) on develop, so this moves to Phase 1.3. It has also been detached from #710 so the epic can close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:testingConformance kit, contract fixtures, nightly matrixenhancementNew feature or requestpriority/p3Nice-to-have, not blockingsize/m1-2 days, multiple files

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions