Repository navigation
Security detection rules: Datadog alerts on suspicious activity by one agent or person (CHOO-4505) - #2135
Draft
petr-sandbox wants to merge 8 commits into
Draft
Security detection rules: Datadog alerts on suspicious activity by one agent or person (CHOO-4505)#2135petr-sandbox wants to merge 8 commits into
petr-sandbox wants to merge 8 commits into
Conversation
…(CHOO-4509) Every monitor now carries a category:ops or category:security tag. apply.sh sends ops monitors to NOTIFY and security monitors to NOTIFY_SECURITY, adding NOTIFY_SECURITY_PAGE at priority 1 or 2. The three credential and sign-in monitors are tagged security. apply.sh checks monitors.json first (check_monitors.py), requires all three handle variables, fills placeholders in jq, and refuses to send anything that still carries one. DRY_RUN=1 now works without Datadog keys and prints each payload and its route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
observability-apply.yml applies deploy/observability to Datadog on a push to main that touches it, from the observability-prod environment's secrets and variables; workflow_dispatch defaults to a dry run. PR CI's observability job now runs check_monitors.py, and its pytest run picks up test_monitors.py, neither needing secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README now explains the category tags and who each routes to, the three handle variables, the dry run, the observability-prod environment and its secrets and variables, and that the security on-call owner still has to be named. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bearer-route refusal alert moves from P3 to P2, so it reaches the security pager as well as the security channel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apply.sh now fills placeholders in jq, and reads the signal board's file name before the default-service guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-actor detection needs a line naming the agent or person for each send, room read, file transfer and service token, and none existed: metrics carry no ids and per-agent routes are kept out of the access log. Add log-only security events at the choke points: the router once an agent's message commits, the read_context and list_all_rooms operations, the agent and Console attachment paths, and the service broker's issue. Ids and counts only, no database writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Thirteen log monitors over the security event stream, each grouped by the agent, person, address or caller it watches and tagged category:security: message bursts, room fan-out reads (P2, pages), membership sprees, bulk files, service-token spikes, Switch Trust blocks and redactions, agent escalation, operator access, key reveals, sign-in failures by address and by account, refusal-breaker trips and suspensions. Each message names the actor and ends with how to suspend them. check_monitors.py now treats every security-stream field as a security signal and does not mistake a facet variable in a message for a handle. The tests pin each rule's priority, shape and routing, and check that the events and actions the rules name are ones Core writes. The Datadog artifacts test no longer reads a log alert's `switch.*` log attributes as metric names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Thirteen Datadog log monitors, tagged
category:security, that alert on suspicious activity by a single agent or person, plus the per-actor log events some of them need.Stacked PR, draft. It is based on #2134 (security event stream) and merges #2125 (alert routing). Until those land, the diff here includes #2125's changes. Review only the last two commits, which are this PR's own. When #2134 merges, the base moves to main.
security.agent.message_sentsecurity.agent.room_read, distinct roomsroom.member_addedsecurity.attachment.uploaded/downloadedsecurity.service_token.issuedtrust.message_blocked/redactedagent.auto_approve_enabled,agent.can_manage_agents_grantedplatform.accessapi_key.revealedsecurity.auth.sign_in_failedsecurity.refusal_breaker.trippedagent.suspended/user.suspendedEach alert names the actor, has alert and recovery text, and points to the suspend and freeze controls. The thresholds are starting values, to be revisited after two weeks of production baseline.
New log-only events. They carry ids and counts only and write nothing to the database:
security.agent.message_sent, written in the router once the row commits;security.agent.room_readandsecurity.agent.rooms_listed;security.attachment.uploaded/downloaded, on the agent and Console paths;security.service_token.issued.Message sends are the highest volume: about 5 lines/s for 1,000 busy agents at the load test's realistic rate. If that proves too much, the fallback is per-agent, per-minute aggregation.
Other changes:
check_monitors.py'sSECURITY_SIGNALSnow requires acategory:securitytag on any monitor that reads the security stream.test_monitors.pypins each rule and its routing, and checks that every event or action a rule names exists.test_datadog_artifacts.pyskips log monitors when it checks metric names.Not yet proven in Datadog: two-facet grouping, the distinct-room count and
{{[@facet].name}}follow Datadog's documented syntax but haven't run there. After the first real apply, check that each monitor imports, and trigger one on purpose.How it was tested
-n 3: 3175 passed, coveringobservability/,delivery/,bridges/agent/,trust/,connections/, chats routes, security events and logging config.deploy/observabilitypytest: 80 passed.apply.shdry run renders all 13 rules. Only "an agent is reading many rooms" routes to the pager as well as the security channel.just checkandjust typecheckare clean. No gateway route or response shape changed.🤖 Generated with Claude Code