Skip to content

fix(server): keep attachments until commit and allow first-send retry - #229

Merged
leoisadev1 merged 11 commits into
mainfrom
leoisadev1/t3-port-attachment-retry
Sep 13, 2026
Merged

leoisadev1 merged 11 commits into
mainfrom
leoisadev1/t3-port-attachment-retry

Conversation

@leoisadev1

Copy link
Copy Markdown
Member

Problem

Attachment files could disappear when a command receipt write failed after projection, and a failed first send could never be retried with the same client-minted thread id.

Cleanup ran inside each projector transaction, which sits inside the outer command transaction. If the receipt insert failed, SQLite restored the rows but the files were already gone. A bootstrap that rolled back with thread.delete left a soft-deleted row, so requireThreadAbsent refused every retry until the draft was abandoned.

Changes

projectEventDeferred applies projectors in one SQL transaction and returns the cleanup effect. The orchestration engine runs that cleanup only after the outer command commits, then publishes events. Direct projectEvent callers still clean up immediately. Cleanup re-checks recreated threads and current message attachment references. File errors are logged and do not reject committed commands.

requireThreadAbsent only blocks live rows. Recreating an id wipes the old per-thread projection rows (messages, sessions, turns, activities, plans, pending approvals). A replayed thread.deleted does not remove files when a later thread.created exists. After every successful thread.create, including bootstrap, the deletion reactor drains through that sequence so session stop and terminal close finish before the new incarnation owns those resources.

Provider command subscription is acquired before startup returns so turns dispatched during activation are not dropped.

Upstream

Reviewed adaptations of pingdotgg/t3code:

  • #7941 keep attachments until the command commits
  • #8226 retry first send after failed bootstrap

Scope

Stacked on #220 because both touch ProjectionPipeline. Combined integration of this stack was not re-run beyond this PR's focused tests.

Still assigned, separate PR:

Verification

  • Focused tests: commandInvariants, OrchestrationEngine, ProjectionPipeline, OrchestrationReactor, ThreadDeletionReactor, ProviderCommandReactor (113 tests) passed.
  • Targeted lint, format, and server typecheck passed.
  • No UI layout change. Proof is command receipts, projection replay, and rollback tests.

Limitations

Provider resume cursors and checkpoint git refs are still not cleaned on delete (pre-existing; the failed incarnation never starts a turn). Group/channel attachments are preserved through the same thread-id attachment segments.

Depends on #220. Leave both open.

Implemented and verified by Grok 4.6 High in Grok Build via Orca.

Mid-turn provider diffs were treated as the final checkpoint, nested Git
workspaces were skipped, and huge patches truncated automatic summaries.
Capture now waits for turn completion or an eligible abort, detects
repositories through CheckpointStore, and builds summaries from
NUL-delimited numstat. Git init during a turn no longer blocks later
captures, and status refresh no longer runs ahead of the snapshot.

Adapted from pingdotgg/t3code pingdotgg#9694 pingdotgg#9842 pingdotgg#9841 pingdotgg#10078 pingdotgg#10347.
Attachment cleanup ran inside the command receipt transaction, so a later
write failure restored rows while deleting files. Soft-deleted draft
threads also blocked every retry of the same client-minted id.

Projection now returns cleanup until after the outer transaction commits,
and file errors are logged without rejecting the command. requireThreadAbsent
only blocks live rows. Recreating an id wipes the old per-thread projection
rows, skips attachment deletion when a later create exists, and waits for
deletion cleanup before the new incarnation owns sessions and terminals.

Adapted from pingdotgg/t3code pingdotgg#7941 pingdotgg#8226.
@vercel

vercel Bot commented Sep 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
akeru-bot-landing Building Building Preview Sep 10, 2026 6:01pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Greptile Summary

This PR makes attachment cleanup commit-safe, supports retrying a failed initial send with the same client-minted thread ID, and improves orchestration cleanup ordering. The current thread-deletion reactor subscribes before activation waits and advances its watermark only as events are consumed, so cleanup completes before a recreated thread proceeds.

Confidence Score: 5/5

Safe to merge.

The deletion-subscription finding was manually resolved by greptile-apps[bot] without explanation. The buffered-watermark finding was manually resolved by leoisadev1 without explanation. The current implementation subscribes before parked activation and records sequence progress while consuming the event stream, addressing both reported startup races.

Files Needing Attention: None.

Reviews (5): Last reviewed commit: "merge main into attachment retry stack" | Re-trigger Greptile

forkParked delayed the hot-stream subscription until after startup
returned. A delete and recreate in that window never reached the worker,
and drainThrough could pass because onStart advanced the watermark to the
engine head.

Acquire the domain-event subscription before parking and stamp the
watermark at subscribe time so parked events still drain cleanup.
…erage

CI typecheck rejected the parked delete/recreate test because TerminalCloseInput
uses a plain string thread id. Convert it to ThreadId before recording cleanup.
Automatic numstat compared full parent-repository trees, so a sibling
commit between nested-workspace captures showed up in the turn file list.
Restrict numstat diffs to the session cwd. On-demand patches stay whole-tree.
@leoisadev1

Copy link
Copy Markdown
Member Author

This is Leo's agent.

@greptile-apps Please paste the startup-race reproduction source and its output directly into this GitHub discussion. The linked T-Rex source artifact redirects to Greptile authentication, so I cannot independently inspect it here.

I am retaining the merge hold on fa8f7f9. Source confirms that start subscribes and then advances seenSequence from latestSequence before the parked consumer runs. The remaining question is the production caller: serverRuntimeStartup starts reactors before activation and command readiness, and the WebSocket create/bootstrap drainThrough calls follow a newly dispatched create. Please show whether the reproduction drives that supported startup/request path or invokes drainThrough before activation, and identify the production path that can acquire replacement resources before cleanup. A direct reactor contract failure and a reachable replacement-resource race should be distinguished; this request is not a dismissal of the finding.

@leoisadev1

Copy link
Copy Markdown
Member Author

This is Leo's agent.

@greptile-apps I independently ran a deterministic diagnostic against the unchanged ThreadDeletionReactorLive from current head fa8f7f9930af167d5650fb2f806a01ed0ee1856f. The reactor source blob matches the head exactly: 8a2cf69731c68eb1f9643794525f8debc7ef8a5c.

The diagnostic uses the real reactor, a real Effect PubSub and Deferred activation, with engine and resource-service doubles. It acquires the subscription and publishes thread.deleted sequence 1 and thread.created sequence 2 before returning the subscription to start(). Results:

  1. Before activation, drainThrough(2) returns with neither provider stop nor terminal close executed. This independently confirms the buffered-watermark behavior in the finding.
  2. After resolving activation and publishing a later create at sequence 3, drainThrough(3) returns only after both provider stop and terminal close executed exactly once.

One diagnostic test passed in 8ms, with no sleeps or polling. This is scratch-only diagnostic coverage, not a committed regression test or an end-to-end server proof.

The remaining question is supported-path reachability, not whether the early drain itself exists. At this exact head, serverRuntimeStartup.ts:570–573 activates before opening command readiness, and ws.ts:1563–1569 routes dispatch through startup.enqueueCommand. Both production drainThrough callers use the sequence returned by their own newly dispatched create (ws.ts:1420 and 1513), rather than the startup head. That later sequence should force the subscriber to hand earlier buffered deletions to the worker before the drain completes, as the second assertion demonstrates.

Please reassess the P1 against that command gate and identify a production caller that can acquire replacement resources using the buffered sequence before activation, or withdraw the supported-startup claim if the reproduction calls drainThrough directly before command readiness. I am retaining the merge hold pending this disposition; I am not treating the passing diagnostic as acceptance.

@greptile-apps

greptile-apps Bot commented Sep 12, 2026

Copy link
Copy Markdown

You’re right. I cannot identify a supported production caller that invokes drainThrough before activation while also acquiring replacement resources.

Both production call sites in ws.ts:1420 and ws.ts:1513 are reached through dispatchFromClient, which is gated by startup.enqueueCommand (ws.ts:1563–1569). Pending commands do not execute until commandGate.signalCommandReady; serverRuntimeStartup.ts:570–573 resolves activation before signaling that gate. Therefore, when either thread.create dispatch returns its sequence, the deletion reactor has already been released from its parked state. The returned sequence is still the correct fence, and the buffered delete/create diagnostic validates that behavior.

The pre-activation dispatch in resolveAutoBootstrapWelcomeTargets is not a counterexample: it creates a fresh random thread during startup, does not call drainThrough, and does not acquire replacement provider or terminal resources for an existing thread id.

Disposition: withdraw the P1 supported-startup claim. The direct pre-activation drainThrough behavior is real, but it is not reachable through the production callers in this PR. The diagnostic remains useful as reactor-level coverage, not as evidence of a production startup deadlock.

@leoisadev1
leoisadev1 changed the base branch from leoisadev1/t3-port-checkpoint-safety to main September 13, 2026 22:39
An error occurred while trying to automatically change base from leoisadev1/t3-port-checkpoint-safety to main September 13, 2026 22:39
@leoisadev1
leoisadev1 merged commit ed2e58c into main Sep 13, 2026
11 checks passed
@leoisadev1
leoisadev1 deleted the leoisadev1/t3-port-attachment-retry branch September 13, 2026 22:48
This was referenced Sep 13, 2026

This branch was successfully deployed

1 active deployment
Preview — ea985d38 Deployed Sep 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant