docs(main): ATR-only baseline on Garak + PINT (per #66)

[eeee2345]

@hiskudin — as you asked in #66, here it is as a PR rather than a comment: the reproducible ATR-only baseline on Garak and PINT.

benchmarks/atr-baseline.md contains:

• Garak — in-scope recall 74.4% (1,321/1,775 across 21 families), with the two out-of-scope families (snowball, harmbench) called out and excluded from the headline number.
• PINT — precision 99.7% / recall 63.6% / F1 77.7% (TP=287, FP=1, TN=398, FN=164); PINT is the only corpus here with a real measured precision, since it carries benign samples.
• Reproduce — pinned to agent-threat-rules commit 1831d0d5, with the exact scripts to regenerate both reports.

The Defender-side column is intentionally left blank — this is the ATR half of a like-for-like table, for your team to fill in on the same corpora, per your offer to review on your side.

Happy to adjust the file's location or format, or to align on the tool-result-focused subset you raised in #66 as a starting point for a bundled pack.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Pull request overview

Adds a reproducible benchmark note capturing an ATR-only baseline on two corpora (Garak and Lakera PINT) to support the ongoing discussion in issue #66 about potential ATR rule bundling and like-for-like comparison against Defender.

Changes:

• Documented pinned ATR baseline results for Garak (recall) and PINT (precision/recall/F1), including scope caveats.
• Added step-by-step reproduction commands pinned to a specific agent-threat-rules commit for repeatability.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hiskudin]

Defender column (same corpora as this baseline)

Ran locally against the same Garak + PINT inputs pinned in this PR (agent-threat-rules@1831d0d5 corpora). Criterion: @stackone/defender defendToolResult with blockHighRisk: true → count as positive when allowed === false.

Engine	PINT P	PINT R	PINT F1	PINT confusion	Garak in-scope R (1,775)	Garak overall R (3,475)
ATR (this PR)	99.7%	63.6%	77.7%	TP=287 FP=1 TN=398 FN=164	74.4% (1,321/1,775)	38.2% (1,329/3,475)
Defender Tier 1	100%	12.9%	22.8%	TP=58 FP=0 TN=399 FN=393	33.5% (595/1,775)	17.1% (595/3,475)
Defender Tier 1+2	82.1%	87.6%	84.8%	TP=395 FP=86 TN=313 FN=56	76.8% (1,363/1,775)	41.7% (1,448/3,475)

Notes for reading this honestly:

• Reporting both T1 and T1+T2 — a single “Defender” cell would be misleading (T1 is a small curated regex pack; T2 carries most recall on these chat/jailbreak sets).
• ATR still leads on PINT precision; Defender T1+T2 leads on PINT recall/F1 and is roughly tied with ATR on Garak in-scope recall.
• These corpora are jailbreak/PINT-shaped, not tool-result traffic — still want an FP check on tool-like benign data before deciding on an opt-in ATR pack (ATR rule-pack as a Tier 1 supplement (462 patterns, MIT) + cross-benchmark offer #66).

Happy to fold this into benchmarks/atr-baseline.md once this PR lands (or as a follow-up docs commit).

[eeee2345]

@hiskudin — thank you for running it rather than taking the table at face value, and for reporting T1 and T1+T2 separately. A single "Defender" cell would have been the easier and worse choice.

Two things I owe you, both about the ATR column you compared against.

The PINT row was mislabelled, and I have corrected it in this PR. I described that corpus as "Lakera's public Prompt Injection Test set". It is not. It is a self-built 850-sample corpus in PINT's format, assembled from deepset/prompt-injections and Lakera/gandalf_ignore_instructions; Lakera's actual benchmark is private and roughly five times larger. The measurements are unchanged — the label was wrong. It also carries a scope caveat worth knowing before drawing a conclusion from the precision figure: only a minority of the ruleset fires on that corpus at all, and a single rule accounts for most of the detections. So "ATR leads on PINT precision" is true of this corpus and says less about the ruleset as a whole than the number suggests.

The ATR figures you compared against are pinned to an older commit, and the newer ones moved more than I expected. Everything in this document is 1831d0d5 (v3.5.7, 714 rules), which is the right way to do it — but the upstream table has since been re-measured on 3.5.11, and the all-families garak recall is materially higher than the 38.2% recorded here. That is the row your "roughly tied on Garak in-scope recall" reading rests on, so it may not survive a re-run. I have made the pinning explicit in the doc rather than quietly refreshing the numbers, because a comparison across two different ATR versions would be worse than a stale one.

If you want the comparison to stand as a decision input for #66, the clean version is both engines re-run on one newer commit. I am happy to produce the ATR column for whichever commit you pick, and to have the result be whatever it is — including it going against ATR.

On folding this into benchmarks/atr-baseline.md: yes, please, and your table belongs in it rather than beside it. Your point about these corpora being jailbreak-shaped rather than tool-result traffic is the one I would most want kept in the text — an FP check on tool-like benign data is a different measurement and this document does not stand in for it.

[hiskudin]

@eeee2345 — thanks for this, and sorry for the slow reply. Both corrections are exactly the kind that make the comparison worth trusting, so let's redo it cleanly rather than land stale numbers.

Plan:

• Relabel the 850-sample set as what it is — a PINT-format set from deepset/prompt-injections + Lakera/gandalf_ignore_instructions — and keep your caveat that only a minority of ATR fires on it (one rule doing most of the work), so that precision figure stays corpus-specific.
• Re-run both engines on ATR v4.0.0 (464548b4). To keep it genuinely apples-to-apples I'll run both through the same harness on the same rows rather than splicing two separate runs together — check out v4.0.0, run your Garak + PINT-format harness for the ATR column, then score Defender (Tier-1 and Tier-1+2 separately) on those exact sample sets, same positive criterion (defendToolResult with blockHighRisk: true → positive when allowed === false). I'll send you the ATR column to sanity-check before it goes in the doc — result either direction. (If the 3.x→4.0 spec churn makes a 3.5.x point the fairer comparison, say the word and I'll pin there instead.)

Your "these corpora are jailbreak-shaped, not tool-result traffic, and this doc doesn't stand in for a tool-result FP check" point stays prominent in the text — I'll keep that measurement where it belongs, as part of scoping the opt-in Tier-1 pack rather than here.

Once that's in, I'll fold it into benchmarks/atr-baseline.md (your table in the text, per your note), land it, and close #75 with credit — then we can pick up the opt-in Tier-1 pack itself (codegen'd from a pinned ATR release, scoped to the tool-result-relevant families), which is where the tool-result benign FP check comes in.

I'll get the Defender numbers on v4.0.0 turned around within a week.

[hiskudin]

@eeee2345 — re-ran it on ATR v4.0.0 (464548b4) and published @stackone/defender@0.8.2, both on the relabelled corpora. Numbers below — please sanity-check the ATR row against what you'd get, since I produced it from your own harness rather than taking a table at face value.

Engine	PINT-format P	R	F1	confusion	Garak in-scope R (1,775)	Garak overall R
ATR v4.0.0	100%	65.4%	79.1%	TP=295 FP=0 TN=399 FN=156	80.5% (1,429)	57.2% (1,987/3,475)
Defender T1	100%	12.9%	22.8%	TP=58 FP=0 TN=399 FN=393	33.3% (591)	17.0%
Defender T1+2	81.9%	86.0%	83.9%	TP=388 FP=86 TN=313 FN=63	76.5% (1,357)	41.4%

How I ran it:

• ATR: your Garak + PINT harnesses unmodified at 464548b4 (785 rules). Garak in-scope = 21 families, excluding snowball (1,500) + harmbench (200) per your convention; overall = all 23. PINT-format = the relabelled 850-sample deepset + gandalf set.
• Defender: defendToolResult on published 0.8.2, blockHighRisk: true, positive ⇔ allowed === false; onnxruntime-node pinned to 1.21.0; T1 and T1+2 separately.

Your v3.5.7→v4.0.0 warning was right — Garak in-scope rose 74.4%→80.5%, so ATR now leads Defender T1+2 there (80.5 vs 76.5); the old "roughly tied" reading is gone. On PINT-format, Defender T1+2 leads on F1 (83.9 vs 79.1) by buying +20 pts recall for 86 benign FPs, where ATR has 0.

Footnotes I'll carry in the doc so no cell gets over-read:

• Garak measures no precision (100% adversarial); the only FP evidence here is PINT-format's 399 benign.
• ATR's Garak cell is regex-only (evaluate()); its PINT cell adds the tier-2.5 embedding layer — different engine per cell.
• ATR's Garak harness scores each prompt on two channels (llm_input + a re-typed mcp_exchange) and counts a hit on either; from the code, the mcp_exchange re-typing looks to admit all rules regardless of source-type scoping. Defender is scored on its single tool-result surface — so the Garak column is each tool on its native harness, not identical-input head-to-head. Flag if I've misread that.
• These corpora are jailbreak-shaped, not tool-result traffic; the tool-result benign FP check is the separate measurement for the Tier-1-pack decision, kept out of this doc.

If the ATR row matches on your side (or you spot a config I got wrong), I'll fold it into benchmarks/atr-baseline.md with your table in the text, land it, and close #75 with credit — then we can scope the opt-in Tier-1 pack.

[eeee2345]

@hiskudin — I re-ran both benchmarks on your exact commit rather than checking your table against a different build, and also ran one ablation that answers your second footnote directly. Everything below is at 464548b4 / v4.0.0 / 785 rules.

All three ATR cells reproduce.

• PINT-format: TP=295, FP=0, TN=399, FN=156 — R=65.4%, P=100%, F1=79.1%. Identical to yours.
• Garak overall: 1,987/3,475 = 57.2%.
• Garak in-scope: dropping snowball (1,500) and harmbench (200) leaves 21 families, 1,775 samples, 1,429 detected = 80.5% — exactly your split.

Your footnote about the PINT cell using a different engine is literally true but numerically inert. I re-ran the PINT harness at your commit with the Tier-2.5 embedding stage disabled: the confusion matrix is identical, TP=295 FP=0 TN=399 FN=156. The embedding contributes exactly one match, and it lands on a sample regex rules already catch. So the PINT cell is reproducible as a pure regex result, and in that respect it is comparable to the Garak cells. Where the embedding does distort something is our per-rule block — tier2.5-embedding-match is listed there next to the ATR rule ids, so the fired count reads 62 when 61 rules actually fired (61 + 724 never-firing = 785 loaded). Our own README gets this wrong today, saying 63 of 784 rules fire; I'll fix that on our side.

Three caveats the PINT column still needs. All about ATR, not about your method.

It isn't Lakera's PINT. Our README says so in that row's footnote: not the official benchmark — that corpus is private and roughly 5x larger. Ours is 850 samples assembled in PINT's format from deepset/prompt-injections (660) and Lakera/gandalf_ignore_instructions (190). Anyone reading "PINT" in a published table will assume the official one.

The 100% precision is in-sample, in two distinct ways. Eight rules reference this corpus at your commit. Five were mined from its false negatives — ATR-2026-02002's description says "Recovered from the PINT false-negative corpus", ATR-2026-02232's closes with "Mined from PINT false negatives", and 02005, 02230, 02231 similarly. The sharper two are 02026 and 02500, which record their zero-false-positive verification against this corpus' benign half — 02500 states "0 / 399 PINT benign prompts" explicitly, 02026 counts them inside an 830-sample PINT + skill-benchmark gate. Either way, the benign set the 100% precision is measured on is a set rules were verified against. (Grep case-insensitively and you'll get nine; the ninth matches "Pinterest".)

And the column is concentrated. ATR-2026-00001 accounts for 226 of the 295 true positives, 76.6%. On provenance: that 226 comes from the per-rule block of our eval report, not from the confusion matrix — the matrix only gives you the 295.

On the channel question — the mechanism you spotted is real, and the naming is defensible; what doesn't carry over is the consequence.

Two channels is right. The second is a tool_response event, and ATR maps that event type to the source type mcp_exchange (src/engine.ts), so if that's what you meant, you named it correctly.

The part that doesn't follow is admitting every rule regardless of scoping. That needs an event whose type is itself mcp_exchange, because that type has no entry in the event-type→source-type map and so is filtered against nothing. That shape is wideRaw(), and it lives in the eval-harness shape set the PINT row uses — not in the Garak path. A tool_response event is filtered normally; it just admits mcp_exchange-scoped rules.

So the practical consequence is: our two rows aren't measured at the same width — PINT through the canonical shape set including that admit-all shape plus scanSkill(), Garak through two production channels only. They shouldn't be read as comparable to each other. That asymmetry is ours, and it's the thing I'd most want stated if this table gets cited.

On your footnotes, individually rather than in a block:

• "Garak measures no precision" — keep exactly as is, it's the most important one.
• "jailbreak-shaped, not tool-result traffic" — keep; it's what should drive the Tier-1 decision.
• "different engine per cell" — I'd soften it given the ablation above: the tier-2.5 layer runs on the PINT cell but changes no cell of the matrix.
• The two-channel one — I'd move it to the PINT row and drop the "admits all rules" clause from the Garak description, per the above.

For the Tier-1 pack scoping: these corpora are jailbreak-shaped rather than tool-result traffic, so the separate tool-result benign FP check you're keeping out of this doc is the one that should drive that decision. Point me at whatever corpus you use for it and I'll run it on our side, so you get a second independent number rather than one from our harness.

[hiskudin]

Thanks for the thorough re-run — that verification is what makes this citable. I've updated atr-baseline.md to the v4.0.0 figures straight from the committed reports at 464548b4 (garak-full-report.json / pint-eval-report.json), including the real per-family breakdown, and folded in the caveats you raised: the in-sample PINT precision (8 rules reference the corpus), the ATR-2026-00001 concentration (226/295), the different-width note between the two rows, and the Tier-2.5-embedding-is-inert point. Kept "Garak measures recall only" and "jailbreak-shaped, not tool-result traffic" as-is. Merging now.

Separately, for the Tier-1-pack decision: I'll point you at our tool-result benign corpus for the FP run — that's the number that should drive it, since these corpora are jailbreak-shaped.