Repository navigation
refusals: detect the F5 Guardrails block, which returns HTTP 200 #1177
Copy link
Copy link
Open
Labels
area:refusalsclassify(), the typed error taxonomy, handlers (1.2, 2.1, 2.2)classify(), the typed error taxonomy, handlers (1.2, 2.1, 2.2)area:skeletonDonkeyAsyncClient, config, transport hooks, Donkey facade (1.1)DonkeyAsyncClient, config, transport hooks, Donkey facade (1.1)blocked-on-verificationCannot be closed by writing code (build plan §0.3)Cannot be closed by writing code (build plan §0.3)enhancementNew feature or requestNew feature or requestpriority/p2Meaningful improvement, ship soonMeaningful improvement, ship soonsize/l3+ days, cross-layer or verification sign-off3+ days, cross-layer or verification sign-off
Description
Summary
Omni Gateway v1.14.0 ships the
F5 Guardrails policy,
backed by F5 AI Guardrails (CalypsoAI). Unlike every guardrail the SDK types today, it blocks with
HTTP 200 and a native content-filter response in the caller's format (OpenAI, Anthropic or
Gemini). The SDK currently reports an F5 block as a successful completion.
Motivation
BG §2.1/§2.2. A block that looks like success is worse than an untyped one. The agent goes on to use
the content-filter stub as if it were a real answer, and no refusal handler (donkey.on()) or span
ever records it.
Current behavior (verified)
python/src/donkey_kit/core/errors.py:706-715(theclassify()docstring) and
docs/verified-apis.md:469: only non-2xx responses are classified. A2xxnever reaches
classify().python/src/donkey_kit/core/_refusal_body.py:220: the content-safety vendor table is consulted
only on the403path.rg content_filter python/srconly matches the Bedrock rejection fixture. Nothing inspects a
2xxfor a filtered completion.
Proposal
- Verify first: capture a blocked
200on
Donkey-Development-Kit/donkey-development-kit-provisioning#61, non-streaming and
streaming. Record the body (for examplefinish_reason: "content_filter"), the
observability/reason headers, and the scan-ID header. Add thedocs/verified-apis.mdrow in this
issue. - Add a
2xxdetection hook in the shared transport. It keys on the policy's verdict header, not on
body text, so a genuine modelcontent_filterstop isn't misread. It raises a typed refusal with
vendor "F5 AI Guardrails", the failed F5 scanner names as categories, and the scan ID as an
attribute. - Streaming: the policy buffers and rewrites the stream, so the hook must also work on stream
completion. - Don't raise on
flagged(audit) orredacted; at most record them on the span. - Type the
503fail-closed (none_response,redaction_failed) as a guardrail outage, not a
content block.
Cross-surface impact
- core: a new
2xxhook in transport, plus_refusal_body.py,_wire.py,errors.py, and
telemetry.py(verdict and scan-ID span attributes). - llm: the OpenAI client's streaming and non-streaming paths.
- integrations: every adapter goes through the shared transport. The conformance kit
(python/tests/conformance/suite.py) needs a2xx-refusal case so all frameworks are checked at
once. - registry / tools: not affected.
- TypeScript parity (planned): flag. A
2xx-refusal hook is a new transport contract the TS
port must copy. - CLI:
donkey mockneeds to be able to serve a200content-filter fixture. - Nextra docs site:
errors.mdx. Explain that this refusal arrives on a200. - docs/verified-apis.md: a new row with a new discriminator class (
2xxrefusal).
Out of scope
- Other federated guardrail vendors. Each has its own independent issue.
- F5 audit-mode reporting beyond span attributes.
Depends on
- Donkey-Development-Kit/donkey-development-kit-provisioning#61: the live proxy to capture
against.
Related
- verify: federated guardrail verdict shapes for Akamai, CrowdStrike and Google Armor #305 / refusals: type the Akamai federated guardrail verdict #313: earlier combined federated-guardrail tracking. This issue owns F5 independently.
- upstream: guardrail-verdict error contract is under-documented #251: upstream guardrail-verdict contract gap. It's worth asking the gateway team why F5 uses
200when the other guardrails use403. - Acceptance: Donkey-Development-Kit/donkey-development-kit-acceptance#69.
Acceptance criteria
- A live-captured F5 block (non-streaming and streaming) raises a typed refusal instead of
returning a completion. - A genuine model
content_filterstop without the F5 verdict header does not raise. -
flagged/redactedoutcomes don't raise. The503is not typed as a content block. - The conformance kit covers the
2xxrefusal across all adapters. - Fixtures, the
docs/verified-apis.mdrow, anderrors.mdxupdated.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area:refusalsclassify(), the typed error taxonomy, handlers (1.2, 2.1, 2.2)classify(), the typed error taxonomy, handlers (1.2, 2.1, 2.2)area:skeletonDonkeyAsyncClient, config, transport hooks, Donkey facade (1.1)DonkeyAsyncClient, config, transport hooks, Donkey facade (1.1)blocked-on-verificationCannot be closed by writing code (build plan §0.3)Cannot be closed by writing code (build plan §0.3)enhancementNew feature or requestNew feature or requestpriority/p2Meaningful improvement, ship soonMeaningful improvement, ship soonsize/l3+ days, cross-layer or verification sign-off3+ days, cross-layer or verification sign-off
Activity
tbolis-at-mulesoft commentedon Oct 20, 2026
Expected shape for
f5-guardrails-policy1.0.0. This is not a live capture; the live capture in this issue still has to confirm it.Block: HTTP
200, confirming this issue's premise. The body is the caller's native refusal format and carries a text message:chat.completionwithchoices[0].finish_reason: "content_filter"and"model": "f5-guardrails".stop_reason: "refusal".finishReason: "SAFETY".The message looks like
Your request was blocked because it matched: <guardrail names> (phase request, scan <id>).Some public docs describe a403block; that was the shape of earlier versions, and 1.0.0 returns200.Headers: the prefix is
x-llm-proxy-f5-calypsoai-guardrail-, not a plainf5:-action:allow,flag,redactorreject;-phase;-reason: the display names of the scanners that fired, which become thecategories;-ids: the scanner IDs;-scan-id: the F5 scan ID.Key the
2xxhook on-action: reject.A blocked call is never streamed. The block is always non-stream JSON, even when the request had
stream: true. The hook therefore has to recognise a JSON body that arrives on a call the client expected to stream.flagandredactlet the call continue:redactrewrites the body with F5's masked text. Neither should raise. WithapplyRedaction: false, aredactoutcome becomes a200block.Fail-closed:
503with body{"error":{"message":"F5 AI Guardrails call failed","type":"service_error","code":"<reason>"}}. The reasons areauth_failed,bad_request,not_found,upstream_error,connection_errorandparse_error. A missing verdict returns"F5 AI Guardrails returned no verdict"withnone_response. These are outages, not blocks.Redaction can also fail closed, with
-reason: redaction_failed. That is a200block, not a503. This corrects the scope line above, which listsredaction_failedas a503.When this lands, it should also emit
decision=refuseand the provider and categories span attributes from #1183.tbolis-at-mulesoft commentedon Oct 20, 2026
The live capture is now tracked separately in #1186 (Verification milestone), the F5 sibling of #1184. This issue keeps the implementation: the
2xxrefusal hook, the typing and the conformance coverage.tbolis-at-mulesoft commentedon Oct 20, 2026
#1183 (PR #1187) defines a single scheme for refusals that come from an external provider. Set `provider="f5-guardrails"`. The F5 block arrives as HTTP 200, and `_span_decision` returns `allow` for every 2xx. The typing here also has to make the span record `decision=refuse`; otherwise these attributes never reach the span.
provider(a stable kebab-case slug) andcategories(list[str], the provider's flagged filter names).donkey.policy.provideranddonkey.policy.categories, from the constantsDONKEY_POLICY_PROVIDERandDONKEY_POLICY_CATEGORIESincore/telemetry.py._policy_detailintransport/policy.pykeys on the attribute, not the class. AnyPolicyViolationthat setsprovidergets both span attributes automatically.feat(telemetry): report the content-safety provider and categories (#…