Skip to content

chore(gramjs): add gap-fill recovery script + runbook - #84

Merged
d4d0h merged 2 commits into
mainfrom
chore/commit-gap-fill-tool
Apr 17, 2026
Merged

chore(gramjs): add gap-fill recovery script + runbook#84
d4d0h merged 2 commits into
mainfrom
chore/commit-gap-fill-tool

Conversation

@d4d0h

@d4d0h d4d0h commented Apr 17, 2026

Copy link
Copy Markdown
Owner

Summary

Commits the one-shot `gap-fill` tool that was used successfully on 2026-04-17 to recover a ~7-day gap in the archive after a silent ping-loop zombie (the same incident that motivated the watchdog in #72).

  • New file: `gramjs/src/gap-fill.ts` — for each dialog, fetches recent Telegram history newest→oldest and ingests only messages with `sent_at > GAP_START_TS`. Safe to re-run (all inserts use `ON CONFLICT DO UPDATE`).
  • `npm run gap-fill` script added to `gramjs/package.json`.
  • `docs/gap-fill.md` — short runbook: when to use, prerequisites (stop listener, set `BACKFILL_ALLOWED=true`, compute `GAP_START_TS` from the `stats` tool's `latest_message_at`), expected runtime, and the safety rule that the listener must be stopped first.

Why not roll this into #78?

Cleanups applied

The script's docstring + error message referenced a stale Fly app name `tg-reader-main` (should have been `tg-reader` in our instance, or `` more generally). Both replaced with the placeholder form to match Phase 1 of #83.

Test plan

  • `npx tsc --noEmit` clean
  • Already exercised in production on 2026-04-17 — recovered 1,615 messages across 54 chats in ~90 min without errors

🤖 Generated with Claude Code

d4d0h added 2 commits April 17, 2026 16:34
@d4d0h
feat(mcp): unify listy output shape with opaque cursors (#76)
5482b3f 
Closes #76.

Every paginated MCP tool now returns the same shape:

  { messages: [...],
    page: { has_more, next_cursor, total? } }

Callers pass `page.next_cursor` back as `cursor` on the next call.
Cursors are opaque base64(JSON) blobs; direction (before vs after)
is implicit in the tool contract.

Before this PR each listy tool had its own shape:

  search  → { results, total, next_before_id, next_before_sent_at }
  history → { results,        next_after_id,  next_after_sent_at  }
  thread  → { chat_id, message_id, messages,  next_after_id       }

Consistency matters most for AI callers — an LLM that learned the
search shape used to trip on history / thread. One canonical layout
means one mental model.

Changes:
- helpers.ts: encodeCursor / decodeCursor with strict validation
- index.ts MCP dispatchers: search, history, thread, digest, recent
  all return the canonical shape. REST /search keeps its old shape
  (dashboard consumes it) — the MCP layer translates.
- mcp-tools.ts: search / history / thread input schemas replace the
  old before_id+before_sent_at / after_id+after_sent_at pairs with
  a single opaque `cursor` param; descriptions reference page.next_cursor.
- Server `instructions` string updated in step so the LLM sees the
  same contract as the schemas.

digest stays chat-grouped; each chat entry now carries a trivial
page: { has_more, next_cursor: null } where has_more = true iff the
messages array hit the per_chat cap.

Out of scope (follow-ups):
- External docs (docs/agents.md, docs/tool-reference.md, etc.) still
  reference the old cursor names — cleanup PR.
- Pagination on `chats` and `contacts`/`address_book` — separate PR.
- `include_total: false` opt-out on search — separate PR.

Verified by subagent review: cursor round-trip is correct across FTS
and B-tree search paths; has_more off-by-one on exact-cap pages is
pre-existing (same logic in the original REST handler). tsc clean.
@d4d0h
feat(gramjs): add gap-fill recovery script for zombie-state gaps
9804ce1 
One-shot tool that, per dialog, fetches recent Telegram history
newest→oldest and ingests messages with sent_at > GAP_START_TS.
Used successfully on 2026-04-17 to recover a ~7-day gap after a
silent ping-loop zombie (fixed in #72 watchdog).

Prerequisite: stop the listener first (flyctl machine stop) to
avoid double-ingest race. BACKFILL_ALLOWED=true env guard.

Also adds `npm run gap-fill` and a short runbook at docs/gap-fill.md.
@github-actions

github-actions Bot commented Apr 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @d4d0h's task —— View job

PR #84 Code Review

Reviewing the gap-fill recovery script and related changes:

Todo List

  • Read and analyze all changed files
  • Check against project conventions from CLAUDE.md
  • Review for security issues
  • Review for correctness and code quality
  • Provide actionable feedback

Review Summary

This PR introduces a well-designed gap-fill tool with appropriate safety measures. The code follows project conventions and includes comprehensive documentation. Here are the key findings:

🔒 Security Issues

  • None identified - The script properly handles authentication and includes necessary guards

✅ Code Quality & Correctness

Strengths:

  • Follows all project conventions from CLAUDE.md:
    • ✅ TelegramClient properly initialized with floodSleepThreshold: 300 and realistic device info (lines 233-239)
    • ✅ Telegram IDs handled as strings throughout (e.g., line 176: tg_message_id: String(raw.id))
    • ✅ Unix epoch seconds preserved for timestamps (line 191: sent_at: raw.date)
    • ✅ Proper anti-ban measures with randomized sleep intervals (line 214)
    • ✅ Serial processing of dialogs (one at a time)
  • Excellent safety measures:
    • BACKFILL_ALLOWED=true guard prevents accidental runs
    • AUTH_KEY_DUPLICATED detection with helpful error message
    • Graceful FLOOD_WAIT handling with script termination
    • ON CONFLICT DO UPDATE makes it safe to re-run
  • Well-structured with clear separation of concerns

🔍 Minor Issues & Suggestions

  1. Hardcoded default timestamp (line 37):

    const GAP_START_TS = parseInt(process.env['GAP_START_TS'] ?? '1774871233', 10);

    The hardcoded fallback 1774871233 should never be used in practice. Consider making it required or using a more obvious sentinel value.

  2. Type safety enhancement (line 188):

    reply_to_message_id: raw.replyTo?.replyToMsgId != null ? String(raw.replyTo.replyToMsgId) : undefined,

    Good null checking, but could be more explicit about the bigInt conversion.

📋 Documentation Quality

The docs/gap-fill.md runbook is excellent:

  • Clear prerequisites and safety warnings
  • Step-by-step instructions
  • Expected behavior and runtime estimates
  • Proper emphasis on stopping the listener first

🧪 Testing

  • TypeScript compilation verified (npx tsc --noEmit)
  • Production-tested successfully on 2026-04-17

✨ Nice to Have

Consider adding a confirmation prompt before execution that shows:

  • Current GAP_START_TS value
  • Estimated time range being recovered
  • Final "proceed?" confirmation

This could prevent misconfigurations where wrong timestamps are used.

Verdict

✅ Approved - This is production-ready code that follows all conventions and includes appropriate safety measures. The minor suggestions above are non-blocking improvements.

@d4d0h
d4d0h merged commit e2d9dd2 into main Apr 17, 2026
1 check passed
@d4d0h
d4d0h deleted the chore/commit-gap-fill-tool branch April 17, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d4d0h