Skip to content

feat(skills): add /self-diagnose for 24h introspection - #396

Merged
sonichi merged 2 commits into
mainfrom
feat/self-diagnose-skill
Apr 17, 2026
Merged

sonichi merged 2 commits into
mainfrom
feat/self-diagnose-skill

Conversation

@sonichi

@sonichi sonichi commented Apr 16, 2026

Copy link
Copy Markdown
Owner

Summary

  • New skill skills/self-diagnose/ — invokable via /self-diagnose [--since 24h].
  • scripts/gather.sh pulls logs, git, PR state, build log tail, pending questions, cold-review log, health output, and quota into a /tmp/sutando-diagnose-<ts>/ directory. ~80 lines, no new deps.
  • SKILL.md describes the three-heading report format: What's been happening / What's broken / What I'd do next. Output lands at notes/diagnose-YYYY-MM-DD-HHMM.md.

Motivation

Health checks pass when services are up. They don't catch transport-layer bugs that only show up inside the logs. We need a second signal that reads the logs and thinks about them.

Proof point

First test run (see notes/diagnose-2026-04-16-1204.md) surfaced a bodhi state-machine FATAL at voice-session.ts:705 — Invalid transition: CLOSED → RECONNECTING. The health-check was passing at the same moment. That's the class of bug this skill is for.

Not in this PR

  • Automated cadence (a proactive-loop integration that runs /self-diagnose once a day and DMs findings). Deferred — wire it up once the report format is stable.
  • The report writing is done by Claude via the SKILL.md instructions — no model invocation is baked into the gather script.

Test plan

  • bash skills/self-diagnose/scripts/gather.sh 24h produces a non-empty output dir with all expected files.
  • Manually ran the full skill flow; output at notes/diagnose-2026-04-16-1204.md.
  • Surfaced at least one bug the existing health-check missed.

🤖 Generated with Claude Code

Reads logs, git log, build_log.md tail, pending-questions, cold-review
log, health-check output, PR state, and recent Discord activity into a
/tmp working dir, then synthesizes a structured note at
notes/diagnose-YYYY-MM-DD-HHMM.md covering:
- What's been happening
- What's broken
- What I'd do next

First test run surfaced a bodhi state-machine FATAL at
voice-session.ts:705 ('Invalid transition: CLOSED → RECONNECTING')
that the passing health-check did not flag — the motivating proof
that passive health pings miss real transport-layer bugs that only
show up when you read the log.

Usage: /self-diagnose [--since 24h]

Scope:
- SKILL.md: trigger, usage, expected report shape
- scripts/gather.sh: bounded collection of logs, PR state, git, health,
  quota. ~80 lines of bash, no external deps beyond gh/python3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sonichi

sonichi commented Apr 16, 2026

Copy link
Copy Markdown
Owner Author

Mini cross-review: LGTM with one real bug.

The skill structure is clean, proof point is concrete (voice-session.ts:705 FATAL), and deferring the cron wiring until the format stabilizes is good discipline. Ship-shape overall.

One actual correctness issue: section 7 in gather.sh uses -newer "$OUT/meta.txt" to find recent result files:

find "$REPO/results" -maxdepth 1 -type f -name "*.txt" -newer "$OUT/meta.txt" 2>/dev/null | head -20 > "$OUT/results-recent-paths.txt" || true

meta.txt is created at the start of gather.sh via echo ... > "$OUT/meta.txt", so -newer meta.txt means "created after gather started" — which captures roughly nothing (gather takes seconds to run). The intent is presumably "files created in the last $WINDOW," but this matches essentially no files. I'd expect results-recent-paths.txt to be empty in every real run.

Two ways to fix:

# (a) Use a mmin/mtime arg derived from SECONDS_AGO
find "$REPO/results" -maxdepth 1 -type f -name "*.txt" -mmin -$((SECONDS_AGO/60)) 2>/dev/null | head -20 > "$OUT/results-recent-paths.txt" || true

or

# (b) Touch a reference file with SINCE_EPOCH mtime and compare against that
touch -d "@$SINCE_EPOCH" "$OUT/.since-ref" 2>/dev/null || touch -t "$(date -r $SINCE_EPOCH +%Y%m%d%H%M.%S)" "$OUT/.since-ref"
find "$REPO/results" -maxdepth 1 -type f -name "*.txt" -newer "$OUT/.since-ref" 2>/dev/null | head -20 > "$OUT/results-recent-paths.txt" || true

(a) is 1-line and cleaner; (b) matches the rest of the file's style if you prefer.

Minor-minor: the voice-agent signal regex Error: (trailing space) is broad — common to catch unrelated lines from generic try/catch logs. If it's noisy in practice, could tighten to Error: (Transport|Stream|Connect|Session) or similar. Non-blocking, eyeball after first few real runs.

Would love to point this at the 24h window right now and see if it surfaces the same dm-fallback flood + Telegram zombie I caught this morning — both pre-PR-#395 timeframe, both exactly the "health-check passes but something is wrong" class.

Validates: output dir created, expected files present, meta.txt fields,
git-log non-empty when commits exist in window, stdout returns valid
path, invalid window rejects, 3d window format works.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sonichi

sonichi commented Apr 16, 2026

Copy link
Copy Markdown
Owner Author

Nice test addition — 14 cases, good coverage for the happy path + invalid window + 3d format. 🎯

Just flagging: the original correctness issue from my earlier review (find -newer "$OUT/meta.txt" in section 7) is still present on d1491bb. The new tests check file existence for meta.txt, git-log.txt, build_log-tail.md, etc., but don't check whether results-recent-paths.txt actually captures files within $WINDOW — so the bug would pass this suite silently.

Quick test that would catch it (add to test-gather.sh):

# Test N: results-recent-paths picks up files mtime-newer than SINCE_EPOCH
touch -t "$(date -v -1H +%Y%m%d%H%M.%S 2>/dev/null || date -d '1 hour ago' +%Y%m%d%H%M.%S)" /tmp/sutando-test-recent-result.txt
cp /tmp/sutando-test-recent-result.txt "$(pwd)/results/" 2>/dev/null
OUT=$(bash skills/self-diagnose/scripts/gather.sh 24h 2>/dev/null)
grep -q "sutando-test-recent-result" "$OUT/results-recent-paths.txt" && pass "results-recent-paths captures 1h-old file in 24h window" || fail "results-recent-paths empty (likely meta.txt -newer bug)"
rm -f "$(pwd)/results/sutando-test-recent-result.txt" /tmp/sutando-test-recent-result.txt

Happy to push a patch on the branch if useful — fix + the test together — since it's stacked on your work. Or leave it for a follow-up.

@sonichi
sonichi merged commit a88f755 into main Apr 17, 2026
@sonichi
sonichi deleted the feat/self-diagnose-skill branch April 17, 2026 00:44
sonichi pushed a commit that referenced this pull request Apr 17, 2026
Followup to #396. The find in section 7 used `-newer "$OUT/meta.txt"`, but
meta.txt is created at the start of gather.sh. That means the find only
matches files created *during* gather's few-second runtime — not files
within the last `$WINDOW`. `results-recent-paths.txt` was empty in every
real run.

Switch to `-mmin -$((SECONDS_AGO/60))` so "recent" actually means recent.

Also add test #8 to test-gather.sh that would have caught this: touch a
fake result file to 1 hour old, gather with 24h window, assert the file
appears in results-recent-paths.txt. Full suite now 15 cases, all pass.

macOS BSD date + linux GNU date both handled for the backdate step.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sonichi added a commit that referenced this pull request Apr 17, 2026
Followup to #396. The find in section 7 used `-newer "$OUT/meta.txt"`, but
meta.txt is created at the start of gather.sh. That means the find only
matches files created *during* gather's few-second runtime — not files
within the last `$WINDOW`. `results-recent-paths.txt` was empty in every
real run.

Switch to `-mmin -$((SECONDS_AGO/60))` so "recent" actually means recent.

Also add test #8 to test-gather.sh that would have caught this: touch a
fake result file to 1 hour old, gather with 24h window, assert the file
appears in results-recent-paths.txt. Full suite now 15 cases, all pass.

macOS BSD date + linux GNU date both handled for the backdate step.

Co-authored-by: Chi <wangchi@Chis-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
liususan091219 pushed a commit to liususan091219/sutando-fork that referenced this pull request May 28, 2026
* feat(skills): add /self-diagnose for 24h introspection

Reads logs, git log, build_log.md tail, pending-questions, cold-review
log, health-check output, PR state, and recent Discord activity into a
/tmp working dir, then synthesizes a structured note at
notes/diagnose-YYYY-MM-DD-HHMM.md covering:
- What's been happening
- What's broken
- What I'd do next

First test run surfaced a bodhi state-machine FATAL at
voice-session.ts:705 ('Invalid transition: CLOSED → RECONNECTING')
that the passing health-check did not flag — the motivating proof
that passive health pings miss real transport-layer bugs that only
show up when you read the log.

Usage: /self-diagnose [--since 24h]

Scope:
- SKILL.md: trigger, usage, expected report shape
- scripts/gather.sh: bounded collection of logs, PR state, git, health,
  quota. ~80 lines of bash, no external deps beyond gh/python3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(self-diagnose): add gather.sh smoke tests (14 cases)

Validates: output dir created, expected files present, meta.txt fields,
git-log non-empty when commits exist in window, stdout returns valid
path, invalid window rejects, 3d window format works.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
liususan091219 pushed a commit to liususan091219/sutando-fork that referenced this pull request May 28, 2026
…chi#401)

Followup to sonichi#396. The find in section 7 used `-newer "$OUT/meta.txt"`, but
meta.txt is created at the start of gather.sh. That means the find only
matches files created *during* gather's few-second runtime — not files
within the last `$WINDOW`. `results-recent-paths.txt` was empty in every
real run.

Switch to `-mmin -$((SECONDS_AGO/60))` so "recent" actually means recent.

Also add test sonichi#8 to test-gather.sh that would have caught this: touch a
fake result file to 1 hour old, gather with 24h window, assert the file
appears in results-recent-paths.txt. Full suite now 15 cases, all pass.

macOS BSD date + linux GNU date both handled for the backdate step.

Co-authored-by: Chi <wangchi@Chis-Mac-mini.local>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant