Skip to content

Security detection rules: Datadog alerts on suspicious activity by one agent or person (CHOO-4505) - #2135

Draft
petr-sandbox wants to merge 8 commits into
security/event-streamfrom
security/detection-rules
Draft

petr-sandbox wants to merge 8 commits into
security/event-streamfrom
security/detection-rules

Conversation

@petr-sandbox

Copy link
Copy Markdown
Collaborator

What this changes

Thirteen Datadog log monitors, tagged category:security, that alert on suspicious activity by a single agent or person, plus the per-actor log events some of them need.

Stacked PR, draft. It is based on #2134 (security event stream) and merges #2125 (alert routing). Until those land, the diff here includes #2125's changes. Review only the last two commits, which are this PR's own. When #2134 merges, the base moves to main.

Rule Signal Grouped by Fires above P
Agent posting a burst of messages security.agent.message_sent agent 300 in 5m 3
Agent reading many rooms security.agent.room_read, distinct rooms agent 10 in 10m 2 (pages)
One actor adding many room members room.member_added actor user or acting agent 10 in 10m 3
One actor moving many files security.attachment.uploaded / downloaded agent, actor 50 in 10m 3
Agent asking for many service tokens security.service_token.issued agent 30 in 10m 3
Switch Trust blocked or redacted a message trust.message_blocked / redacted agent, actor any 3
Agent given more power agent.auto_approve_enabled, agent.can_manage_agents_granted agent any 3
Platform operator acted in a workspace platform.access actor any 4
API key revealed api_key.revealed actor any 4
Sign-ins failing from one address security.auth.sign_in_failed client IP 20 in 10m 3
Sign-ins failing for one account the same account 20 in 10m 3
One caller tripping the refusal breaker security.refusal_breaker.tripped caller 5 in 10m 3
Agent or person suspended agent.suspended / user.suspended target any 4

Each alert names the actor, has alert and recovery text, and points to the suspend and freeze controls. The thresholds are starting values, to be revisited after two weeks of production baseline.

New log-only events. They carry ids and counts only and write nothing to the database:

  • security.agent.message_sent, written in the router once the row commits;
  • security.agent.room_read and security.agent.rooms_listed;
  • security.attachment.uploaded / downloaded, on the agent and Console paths;
  • security.service_token.issued.

Message sends are the highest volume: about 5 lines/s for 1,000 busy agents at the load test's realistic rate. If that proves too much, the fallback is per-agent, per-minute aggregation.

Other changes:

  • check_monitors.py's SECURITY_SIGNALS now requires a category:security tag on any monitor that reads the security stream.
  • test_monitors.py pins each rule and its routing, and checks that every event or action a rule names exists.
  • test_datadog_artifacts.py skips log monitors when it checks metric names.

Not yet proven in Datadog: two-facet grouping, the distinct-room count and {{[@facet].name}} follow Datadog's documented syntax but haven't run there. After the first real apply, check that each monitor imports, and trigger one on purpose.

How it was tested

  • Core at -n 3: 3175 passed, covering observability/, delivery/, bridges/agent/, trust/, connections/, chats routes, security events and logging config.
  • deploy/observability pytest: 80 passed.
  • An apply.sh dry run renders all 13 rules. Only "an agent is reading many rooms" routes to the pager as well as the security channel.
  • just check and just typecheck are clean. No gateway route or response shape changed.

🤖 Generated with Claude Code

petr-sandbox and others added 8 commits November 18, 2026 13:31
…(CHOO-4509)

Every monitor now carries a category:ops or category:security tag.
apply.sh sends ops monitors to NOTIFY and security monitors to
NOTIFY_SECURITY, adding NOTIFY_SECURITY_PAGE at priority 1 or 2. The three
credential and sign-in monitors are tagged security.

apply.sh checks monitors.json first (check_monitors.py), requires all three
handle variables, fills placeholders in jq, and refuses to send anything
that still carries one. DRY_RUN=1 now works without Datadog keys and prints
each payload and its route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
observability-apply.yml applies deploy/observability to Datadog on a push
to main that touches it, from the observability-prod environment's secrets
and variables; workflow_dispatch defaults to a dry run. PR CI's
observability job now runs check_monitors.py, and its pytest run picks up
test_monitors.py, neither needing secrets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README now explains the category tags and who each routes to, the three
handle variables, the dry run, the observability-prod environment and its
secrets and variables, and that the security on-call owner still has to be
named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bearer-route refusal alert moves from P3 to P2, so it reaches the
security pager as well as the security channel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apply.sh now fills placeholders in jq, and reads the signal board's file
name before the default-service guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-actor detection needs a line naming the agent or person for each send,
room read, file transfer and service token, and none existed: metrics carry
no ids and per-agent routes are kept out of the access log. Add log-only
security events at the choke points: the router once an agent's message
commits, the read_context and list_all_rooms operations, the agent and
Console attachment paths, and the service broker's issue. Ids and counts
only, no database writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Thirteen log monitors over the security event stream, each grouped by the
agent, person, address or caller it watches and tagged category:security:
message bursts, room fan-out reads (P2, pages), membership sprees, bulk
files, service-token spikes, Switch Trust blocks and redactions, agent
escalation, operator access, key reveals, sign-in failures by address and
by account, refusal-breaker trips and suspensions. Each message names the
actor and ends with how to suspend them.

check_monitors.py now treats every security-stream field as a security
signal and does not mistake a facet variable in a message for a handle.
The tests pin each rule's priority, shape and routing, and check that the
events and actions the rules name are ones Core writes. The Datadog
artifacts test no longer reads a log alert's `switch.*` log attributes as
metric names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant