Operations runbook #13

Context

Pixel will be run by volunteers. The common operational tasks should be written down so anyone with access can do them safely, without needing the original author around.

Goal

A runbook (for example docs/runbook.md) covering day-to-day operations and incidents for dev and prod.

Procedures to cover

• Access lists: add or remove a member, friend or admin in each environment, and roll out the change
• Moderation:
  • find a user's actions in the logs by Discord ID
  • block a user (Moderation: block list for banned users #6)
• Secrets:
  • rotate the Discord bot token
  • rotate other secrets (Sentry DSN and so on)
• Deploys:
  • roll back to the previous version
  • check what's currently deployed
• Incidents:
  • the bot is offline or not responding
  • duplicate replies
  • SpaceAPI is down
• Sentry: where alerts go, and how to triage them
• Privacy: handle a data access or deletion request (Privacy notice and data retention (GDPR) #7)
• Disaster recovery: recreate an environment from scratch

Acceptance criteria

• Each procedure has been followed successfully by someone other than its author (in dev)
• Linked from the README
• Kept in sync with infrastructure changes. Note this in AGENTS.md

Open questions

• Who is on the hook when the bot is down? Is there a contact list?