Skip to content

Security event stream: mirror audit writes to the logs, audit what was missing, resolve real client IPs (CHOO-4503, CHOO-4504) - #2134

Open
petr-sandbox wants to merge 8 commits into
mainfrom
security/event-stream
Open

petr-sandbox wants to merge 8 commits into
mainfrom
security/event-stream

Conversation

@petr-sandbox

Copy link
Copy Markdown
Collaborator

What this changes

Audit events now reach the log pipeline, so they can be alerted on. Several security-relevant changes that left no record before are now audited.

Security event stream (core/switch_core/security_events.py)

  • Every audit write also writes one INFO line on the switch_core.security logger:
    • switch.event is security.<action>;
    • it carries the scope, tenant, organization, actor, target, agent, room and audit-row ids;
    • details go under security_details.*, flattened to one level.
  • A line is written only when the outermost transaction commits. A rolled-back change, a rolled-back savepoint, or a session closed without committing leaves no line, just as it leaves no row.
  • The security logger stays at INFO even when the log level is raised.
  • Details whose key names an email address are left out of the line. The audit row keeps them.
  • Metrics still carry no tenant, agent or room ids. Per-actor detection reads these lines instead.
  • The Helm chart now defaults to JSON logs. The DEBUG line that logged agent message content now logs ids and length only.

New audit actions, written in the same transaction as the change:

  • Rooms:
    • room.member_added / room.member_removed, at the one place a membership row is written;
    • room.created / archived / unarchived / deleted.
  • Agents and controllers:
    • agent.registered / agent.reregistered;
    • controller.enrolled / controller.revoked;
    • agent.can_manage_agents_granted / _revoked, and agent.auto_approve_enabled, on the gateway, MCP and template paths.
  • Sign-ins: user.signed_in (password and OIDC) in the server audit table, with method and client IP, plus organization.directory_sign_in.
  • Trust: trust.message_blocked / trust.message_redacted, with policy, outcome, room and finding categories, never the detected text.
  • On deep write paths with many callers, the actor comes from the identity authentication bound to the request.

Log-only events (switch.scope=log), which have no transaction or would fire too often for a row each:

  • security.auth.sign_in_failed. The account is named by id only when it exists, never by the email that was typed.
  • security.budget.exceeded, once per budget per period.
  • security.refusal_breaker.tripped.
  • security.trust.verdict, for alerted and errored.

Real client addresses

  • uvicorn now resolves X-Forwarded-For from trusted proxies only. They are set by the new chart value switchCore.forwardedAllowIps (FORWARDED_ALLOW_IPS).
  • It defaults to 127.0.0.1. The chart refuses "*", and switch-core warns if it gets it.
  • A deployment must set this to the source CIDRs its ingress or load balancer connects from. Until it does, logged client IPs and per-address login limits see the load balancer's address.

No migration: action has no constraint.

How it was tested

  • Core against real Postgres at -n 3, after rebasing on main: 10162 passed across gateway, trust, bridges, db, delivery and the new tests. 4 tests broken by the rebase were fixed and now pass, together with all of observability/ (360 passed). The full suite passed before the follow-ups: 14977 passed.
  • New tests:
    • test_security_events.py: commit, rollback and savepoint behaviour, and email exclusion;
    • test_room_audit_events.py;
    • management/test_audit_events.py;
    • gateway/test_sign_in_events.py;
    • test_main_forwarded_ips.py;
    • Trust row tests.
  • just check and just typecheck are clean. scripts/check_chart_decision.py passes, and the chart render and lint pass.
  • tooling-e2e: 78 passed. The surface snapshot is current; no routes changed.

🤖 Generated with Claude Code

petr-sandbox and others added 8 commits November 18, 2026 13:28
Audit events lived only in Postgres, where nothing could alert on them.
Each write through record_audit_event, record_organization_audit_event and
record_server_audit_event now also writes one INFO line on the
`switch_core.security` logger, with `switch.event` = `security.<action>`
and the tenant/organization, actor, target, agent and room ids as
`switch.*` fields and the details as `security_details.*`.

The line is written only after the transaction holding the audit row
commits: pending lines ride on the session and its after_commit,
after_soft_rollback and after_transaction_end events decide them, so a
rolled-back change, or a rolled-back savepoint around it, leaves no line.
The logger is held at INFO whatever SWITCH_LOG_LEVEL says.

Also: the Helm chart now defaults to LOG_FORMAT=json (the server's own
default stays text, for terminals and local compose); the agent message
endpoint no longer logs message content at DEBUG, only its length; and
docs/old/observability.md describes the stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New audit actions, each written in the transaction of the change it
records (and so mirrored to the security event stream on commit):

- room.member_added / room.member_removed, at the one place a
  client_rooms row is written or deleted (RoomStore.add_client /
  remove_client), for agents, Console members and bridged platform
  users; bridge and system clients are not recorded. Bridge member sync
  goes through the same path, so a first sync of a large channel writes
  one event per person it adds.
- room.created, room.archived, room.unarchived, room.deleted in
  RoomService (and the unarchive-by-posting path in chats).
- agent.registered / agent.reregistered in AgentCore registration.
- controller.enrolled (console and enrollment code) and
  controller.revoked (in _revoke, which every revocation goes through).
- agent.can_manage_agents_granted / _revoked from the gateway route and
  template runs.
- agent.auto_approve_enabled when a managed agent is created with, or
  changed to, auto_approve.

Where many callers share a deep write path, the actor is taken from the
request's bound identity (request_actor: the signed-in person or the
calling agent's owner, plus the calling agent) rather than threaded
through every caller.

No migration: audit_events.action carries no constraint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A successful password or OIDC sign-in is a server audit event
(user.signed_in, with method and client IP); finishing the WorkOS
organization sign-in is an organization audit event
(organization.directory_sign_in). A refused sign-in on any of them is a
log-only security event, security.auth.sign_in_failed, with method,
reason and client IP, naming the account by id only when it exists and
never by the email typed: a failure has no tenant and no trustworthy
account, and a row per attempt would let anyone grow an append-only
table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three log-only security events, with switch.scope=log:

- security.budget.exceeded when a usage budget stops an agent, once per
  budget period per guard rather than on every refused message.
  require_within now takes the room.
- security.refusal_breaker.tripped, naming the caller by a short prefix
  of its hashed credential and the client IP.
- security.trust.verdict for a Switch Trust verdict other than ok: the
  policy, the room, the outcome and each finding's category, detector
  and severity, never the detected text. The check runs before the
  send's transaction opens, and during an outage every message is
  errored, so these are not rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The invitation audit event's details carry the invitee's address. The
row keeps it; the log line, which reaches Datadog, leaves out any detail
whose key names an email.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uvicorn now runs with proxy_headers on and forwarded_allow_ips from the
new FORWARDED_ALLOW_IPS setting (the variable uvicorn itself reads),
defaulting to 127.0.0.1. X-Forwarded-For is believed only from those
addresses, so the access log, the sign-in events' client_ip and the
refusal breaker report the real client behind an ingress once it is set
to the ingress's source CIDR(s). "*" is warned about at startup.

The Helm chart exposes it as switchCore.forwardedAllowIps (default
127.0.0.1) and refuses "*", which would let any caller forge its
address. docs/old/observability.md says what to set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Switch Trust verdict of blocked or redacted is now a workspace audit
row (trust.message_blocked / trust.message_redacted) with the policy id
and name, outcome, room, sender role, sending agent and each finding's
category, detector and severity; never the detected text or the
message. alerted and errored stay log-only security.trust.verdict lines.

The row is written by check_message itself, so every caller is covered
(agent posts, cards and attachments in AgentCore, inbound bridged
messages and attachments in CollaborationCore, Console chats), on a
short session of its own: the check runs before the send's transaction
opens and a block has no send. It is committed before the verdict is
returned, so if the write fails its error propagates and nothing,
blocked or redacted, is delivered. check_message and check_attachments
now take the session factory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The password sign-in now counts attempts against the login limits, so
the sign-in event tests give it their settings; the unlock test picks
its audit row by action, now that the sign-in is audited too; and the
SSO callback test's request carries a client address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant