feat: implement RFC 0009 phase 2 external middleware services

[pimlock]

Description

Implement RFC 0009 Phase 2: external supervisor middleware services and operational registration on top of the proto contract established in Phase 1.

This phase should turn the in-process middleware seam into a pluggable supervisor/gateway capability while preserving the same request, response, policy, failure, and audit semantics.

Context

Parent feature issue: #1733

Phase 2 should build on the Phase 1 openshell.middleware.v1 contract and address the pieces intentionally deferred from the first slice:

• External gRPC middleware service invocation from the supervisor.
• Gateway registration, discovery, and validation for middleware implementations.
• Authentication and transport security for supervisor-to-middleware communication.
• Runtime and CLI management flows for configured middleware where appropriate.
• Operational health, timeout, retry, and backpressure behavior for external middleware.
• Compatibility rules for existing policies and supervisors without configured middleware.

Open design points to resolve before or during implementation:

• Whether HttpRequest/post_credentials is part of Phase 2 or remains deferred.
• Whether openshell/sigv4 belongs in Phase 2 or needs a separate streaming/header-signing design issue.
• The exact trust and deployment boundary between gateway-owned middleware, supervisor-local middleware, and user-supplied middleware.

Definition of Done

• Supervisor can invoke external middleware over the RFC 0009 gRPC contract.
• Gateway/supervisor configuration can register and validate external middleware implementations.
• External middleware invocation preserves Phase 1 chain ordering, fail-open/fail-closed behavior, metadata/finding accumulation, and safe mutation rules.
• Transport/auth requirements are enforced and documented for supervisor-to-middleware calls.
• Runtime observability reports external middleware allow, deny, transform, failure, timeout, and finding outcomes without logging raw sensitive payload data.
• Integration tests cover healthy external middleware, deny short-circuiting, transformations before credential injection, timeout/failure behavior, and compatibility with no configured middleware.

[github-actions]

This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

[pimlock]

Closing this broad phase issue because the phase split no longer matches the implementation or how we want to track middleware evolution.

PR #2027 delivered the first usable supervisor middleware version, including statically registered operator-run gRPC services, gateway registration and validation, ordered execution, failure behavior, and observability. The shipped scope is now captured in #2010.

Remaining work should be tracked as focused capabilities rather than one phase:

• feat: detect and resolve middleware manifest drift with digest pinning and gateway refresh #2217 tracks manifest drift detection and refresh.
• feat: add operator-controlled binding policies to supervisor middleware #2282 tracks operator-controlled binding policies.
• feat: support provider-profile and typed middleware attachment selectors #2283 tracks provider-profile and typed attachment selectors.
• feat: expose supervisor middleware registrations through Helm values #2286 tracks Helm registration configuration.
• feat(sandbox): introduce middleware layer for L7 proxy request transformations #1694 tracks the credential-visible request transformation use case.
• Authenticated transport, runtime management, additional deployment models, and new protocol or operation hooks should each get their own issue when their design is ready.

The parent #1733 remains the umbrella for supervisor middleware.