Skip to content

refusals: detect the F5 Guardrails block, which returns HTTP 200 #1177

Open

Description

@tbolis-at-mulesoft

Summary

Omni Gateway v1.14.0 ships the
F5 Guardrails policy,
backed by F5 AI Guardrails (CalypsoAI). Unlike every guardrail the SDK types today, it blocks with
HTTP 200
and a native content-filter response in the caller's format (OpenAI, Anthropic or
Gemini). The SDK currently reports an F5 block as a successful completion.

Motivation

BG §2.1/§2.2. A block that looks like success is worse than an untyped one. The agent goes on to use
the content-filter stub as if it were a real answer, and no refusal handler (donkey.on()) or span
ever records it.

Current behavior (verified)

  • python/src/donkey_kit/core/errors.py:706-715 (the classify() docstring) and
    docs/verified-apis.md:469: only non-2xx responses are classified. A 2xx never reaches
    classify().
  • python/src/donkey_kit/core/_refusal_body.py:220: the content-safety vendor table is consulted
    only on the 403 path.
  • rg content_filter python/src only matches the Bedrock rejection fixture. Nothing inspects a
    2xx for a filtered completion.

Proposal

  • Verify first: capture a blocked 200 on
    Donkey-Development-Kit/donkey-development-kit-provisioning#61, non-streaming and
    streaming. Record the body (for example finish_reason: "content_filter"), the
    observability/reason headers, and the scan-ID header. Add the docs/verified-apis.md row in this
    issue.
  • Add a 2xx detection hook in the shared transport. It keys on the policy's verdict header, not on
    body text, so a genuine model content_filter stop isn't misread. It raises a typed refusal with
    vendor "F5 AI Guardrails", the failed F5 scanner names as categories, and the scan ID as an
    attribute.
  • Streaming: the policy buffers and rewrites the stream, so the hook must also work on stream
    completion.
  • Don't raise on flagged (audit) or redacted; at most record them on the span.
  • Type the 503 fail-closed (none_response, redaction_failed) as a guardrail outage, not a
    content block.

Cross-surface impact

  • core: a new 2xx hook in transport, plus _refusal_body.py, _wire.py, errors.py, and
    telemetry.py (verdict and scan-ID span attributes).
  • llm: the OpenAI client's streaming and non-streaming paths.
  • integrations: every adapter goes through the shared transport. The conformance kit
    (python/tests/conformance/suite.py) needs a 2xx-refusal case so all frameworks are checked at
    once.
  • registry / tools: not affected.
  • TypeScript parity (planned): flag. A 2xx-refusal hook is a new transport contract the TS
    port must copy.
  • CLI: donkey mock needs to be able to serve a 200 content-filter fixture.
  • Nextra docs site: errors.mdx. Explain that this refusal arrives on a 200.
  • docs/verified-apis.md: a new row with a new discriminator class (2xx refusal).

Out of scope

  • Other federated guardrail vendors. Each has its own independent issue.
  • F5 audit-mode reporting beyond span attributes.

Depends on

  • Donkey-Development-Kit/donkey-development-kit-provisioning#61: the live proxy to capture
    against.

Related

Acceptance criteria

  • A live-captured F5 block (non-streaming and streaming) raises a typed refusal instead of
    returning a completion.
  • A genuine model content_filter stop without the F5 verdict header does not raise.
  • flagged / redacted outcomes don't raise. The 503 is not typed as a content block.
  • The conformance kit covers the 2xx refusal across all adapters.
  • Fixtures, the docs/verified-apis.md row, and errors.mdx updated.

Activity

  1. added
    enhancementNew feature or request
    blocked-on-verificationCannot be closed by writing code (build plan §0.3)
    priority/p2Meaningful improvement, ship soon
    size/l3+ days, cross-layer or verification sign-off
    area:skeletonDonkeyAsyncClient, config, transport hooks, Donkey facade (1.1)
    area:refusalsclassify(), the typed error taxonomy, handlers (1.2, 2.1, 2.2)
    on Oct 19, 2026
  2. tbolis-at-mulesoft commented on Oct 20, 2026

    @tbolis-at-mulesoft
    CollaboratorAuthor

    Expected shape for f5-guardrails-policy 1.0.0. This is not a live capture; the live capture in this issue still has to confirm it.

    • Block: HTTP 200, confirming this issue's premise. The body is the caller's native refusal format and carries a text message:

      • OpenAI: chat.completion with choices[0].finish_reason: "content_filter" and "model": "f5-guardrails".
      • Anthropic: stop_reason: "refusal".
      • Gemini: finishReason: "SAFETY".

      The message looks like Your request was blocked because it matched: <guardrail names> (phase request, scan <id>). Some public docs describe a 403 block; that was the shape of earlier versions, and 1.0.0 returns 200.

    • Headers: the prefix is x-llm-proxy-f5-calypsoai-guardrail-, not a plain f5:

      • -action: allow, flag, redact or reject;
      • -phase;
      • -reason: the display names of the scanners that fired, which become the categories;
      • -ids: the scanner IDs;
      • -scan-id: the F5 scan ID.

      Key the 2xx hook on -action: reject.

    • A blocked call is never streamed. The block is always non-stream JSON, even when the request had stream: true. The hook therefore has to recognise a JSON body that arrives on a call the client expected to stream.

    • flag and redact let the call continue: redact rewrites the body with F5's masked text. Neither should raise. With applyRedaction: false, a redact outcome becomes a 200 block.

    • Fail-closed: 503 with body {"error":{"message":"F5 AI Guardrails call failed","type":"service_error","code":"<reason>"}}. The reasons are auth_failed, bad_request, not_found, upstream_error, connection_error and parse_error. A missing verdict returns "F5 AI Guardrails returned no verdict" with none_response. These are outages, not blocks.

    • Redaction can also fail closed, with -reason: redaction_failed. That is a 200 block, not a 503. This corrects the scope line above, which lists redaction_failed as a 503.

    When this lands, it should also emit decision=refuse and the provider and categories span attributes from #1183.

  3. tbolis-at-mulesoft commented on Oct 20, 2026

    @tbolis-at-mulesoft
    CollaboratorAuthor

    The live capture is now tracked separately in #1186 (Verification milestone), the F5 sibling of #1184. This issue keeps the implementation: the 2xx refusal hook, the typing and the conformance coverage.

  4. tbolis-at-mulesoft commented on Oct 20, 2026

    @tbolis-at-mulesoft
    CollaboratorAuthor

    #1183 (PR #1187) defines a single scheme for refusals that come from an external provider. Set `provider="f5-guardrails"`. The F5 block arrives as HTTP 200, and `_span_decision` returns `allow` for every 2xx. The typing here also has to make the span record `decision=refuse`; otherwise these attributes never reach the span.

    • Exception fields: provider (a stable kebab-case slug) and categories (list[str], the provider's flagged filter names).
    • Span attributes: donkey.policy.provider and donkey.policy.categories, from the constants DONKEY_POLICY_PROVIDER and DONKEY_POLICY_CATEGORIES in core/telemetry.py.
    • No transport change needed. _policy_detail in transport/policy.py keys on the attribute, not the class. Any PolicyViolation that sets provider gets both span attributes automatically.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:refusalsclassify(), the typed error taxonomy, handlers (1.2, 2.1, 2.2)area:skeletonDonkeyAsyncClient, config, transport hooks, Donkey facade (1.1)blocked-on-verificationCannot be closed by writing code (build plan §0.3)enhancementNew feature or requestpriority/p2Meaningful improvement, ship soonsize/l3+ days, cross-layer or verification sign-off

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions