Skip to content

ci(fork): upstream sync repairs itself in rounds and continues a red sync - #163

Merged
Pawl-Patrol merged 5 commits into
forkfrom
feat/upstream-sync-self-heal
Oct 16, 2026
Merged

Pawl-Patrol merged 5 commits into
forkfrom
feat/upstream-sync-self-heal

Conversation

@Pawl-Patrol

@Pawl-Patrol Pawl-Patrol commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #162.

Problem

The upstream sync keeps ending as red drafts (#146–#149, #157, #158). Since 10 Oct only 2 of 7 scheduled runs merged themselves (#153, #154). The others needed Paul to step in. The sync artifacts show four causes:

  1. The Copilot CLI ends the session while verify runs. verify takes 20–25 min, and the server's related tests alone take 10–17 min. The CLI moves a command to the background after 600 s, and when the model then waits, the CLI gives up after another 600 s (Timed out after 600s waiting for background tasks … Set COPILOT_TASK_WAIT_TIMEOUT_SECONDS). The session ends with no summary. This happened on 12, 13, 14 and 15 Oct; chore: merge upstream/main (e536bfd33) #157's "Verification is still running." is this case.
  2. Budget runs out. On days with 26–52 conflicts, gpt-5.6-sol used up the 300 credits in 7–9 min (10, 11, 12 and 16 Oct; Session limits: 278/300 AI credits used). chore: merge upstream/main (cfafb4bb54) #158 stopped half done. It even left a lockfile that doesn't match catalogs, which is why its copilot-setup-steps check failed. That is a symptom, not a separate environment problem.
  3. No feedback loop. There is one Copilot session, then the fixed verify, then a draft. The red check output never reaches the agent.
  4. Red days pile up. Each run starts again from fork, so the backlog grows: 38 → 47 → 52 conflicts from 10 to 12 Oct, and 7 → 26 from 15 to 16 Oct. That lasted until someone merged by hand (chore: merge upstream/main (58d253231d) #150 from the …-manual branch).

The failures in #157/#158 were real breakages (CoordinatorUpdates, ThreadDecisions, typecheck), not flaky tests.

Fix

  • Rounds: the Copilot step now loops up to rounds sessions (default 3). After each one it commits what's left, including unresolved markers, and runs the fixed verify. While verify is red, the next session starts with checks.md, guards.md, audit.md and the failing lines of checks.log. A session that writes blocked ends the loop. The final verify step reuses the last result in the loop and doesn't spend another 25 min.
  • COPILOT_TASK_WAIT_TIMEOUT_SECONDS=3600. The prompt also tells the agent to iterate with targeted typecheck and test runs, and to run the full verify at most once.
  • Weekly instead of daily: the sync runs on Mondays at 04:00 UTC (06:00 Berlin in summer, 05:00 in winter), so each run can spend more.
  • Budget: max_ai_credits now applies per session, and the default goes from 300 to 1500. Each session is capped at 80 min, and the job timeout goes from 150 to 355 min (GitHub-hosted jobs stop at 360). That leaves room for 3 rounds per run, each up to 80 min of Copilot plus up to 30 min of verify. The loop doesn't start a round that couldn't finish in time.
  • Follow-up runs: a red run that Copilot didn't mark blocked dispatches the next run (new attempt input, up to MAX_ATTEMPTS = 3). That run continues from the draft PR the first one just opened. This gives up to 9 rounds a week without splitting the job. In the worst case that's 3 × 3 × 1500 credits = $135 a week, roughly what 7 daily runs at 3 × 600 could have cost before.
  • Model: the default is gpt-6.1-sol, which has been GA in Copilot since 29 Sep. The org's model list from the 16 Oct run doesn't offer it yet. Until an admin enables it, the run falls back to FALLBACK_MODEL (gpt-5.6-sol) with a warning.
  • Continue a red sync: a new step looks for the newest open sync/upstream-* PR. merge then merges fork's new commits and upstream into that PR's branch, so its resolutions, its fixes and any fixes pushed by hand carry over, and only conflicts that are new since then come up. If fork's new commits conflict with that branch, the sync falls back to a fresh merge. Before checking, verify folds the history into one merge commit of FORK_SHA and TARGET with the same tree, so guards and the PR shape don't change. It folds only when both commits are ancestors of HEAD. Publish closes the PR it continued from with "Continued in …". The new fresh input turns this off.
  • The related tests run with --retry 1, so a single flaky test doesn't hand the sync to the maintainer. Files that the result takes from neither side also count as touched, so their tests still run after folding.
  • dry_run now opens no PR and closes none; the result is only in the artifact. Before, it opened a draft, which a later run would have treated as an attempt to continue.
  • Lessons from landing chore: merge upstream/main (cfafb4bb54) #158 by hand:
    • A failed vp install (chore: merge upstream/main (cfafb4bb54) #158: a hand-merged patches/ file with a broken hunk header, ERR_PNPM_PATCH_FAILED) now stops the checks with a bold "fix this first" in checks.md, instead of a list of follow-up failures against the wrong lockfile.
    • Typecheck runs every package on its own and reports all red ones together; vpr typecheck stopped at the first.
    • The audit also writes the lost lines themselves (audit-lines.txt). The round prompt shows them and asks Copilot to account for every file: restore the feature or explain the loss in its summary. chore: merge upstream/main (cfafb4bb54) #158 lost the PR panel's status grouping this way, unnoticed.
    • The prompt tells Copilot to fix red checks across the whole repo, not just conflicted files (moved imports, Effect API changes, the Alpha→Beta rename behind the "esveo code" branding), to regenerate patches/ files against the pristine package instead of hand-merging them, and to regenerate generated files (routeTree.gen.ts with the fork's routes, mobile generated-uniwind-themes.css) instead of picking a side.
    • Nothing waits on PR checks: repo CI is disabled on the fork, so the workflow's own verify is the only signal. Checks stay on Linux; the server tests that fail only on macOS don't matter here.
  • FORK.md describes the new behaviour (weekly schedule, follow-up runs).

All of this is in fork-only files (.github/workflows/fork-upstream-sync.yml, .github/scripts/fork-upstream-sync.sh, FORK.md, the changelog).

Validation

  • actionlint and shellcheck: nothing new. The remaining reports were already there, plus copilot-requests, which actionlint doesn't know yet.
  • I tested the script on a throwaway repo:
    • Continuing from an earlier attempt kept its conflict resolution and its later fix, and merged new upstream commits without a conflict.
    • The fold produced a single merge commit with parents (fork, upstream) and the same tree. Guards were green, and running it a second time changed nothing.
    • When fork's new commits conflict with the earlier attempt, it falls back to a fresh merge.
  • The install-failure stop, the per-package typecheck and audit-lines.txt ran against a throwaway repo with a stubbed vp.
  • I rendered the round 1 and round 2 prompts with the artifacts of runs 38680808193 and 38849990819.
  • Until this is merged, merge runs the scripts from fork, so a dispatch from this branch can only exercise the rounds and the wait timeout, not continuation.

🤖 Generated with Claude Code (Claude Opus 5.5, esveo code)

Paul Dittrich and others added 5 commits October 16, 2026 13:35
…sync

Copilot now gets up to three sessions, each started with the red verify
reports of the one before, a wait timeout long enough for verify, and a
larger per-session budget. A sync that stays red is continued the next day
from its open PR instead of being redone from fork with a bigger backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sync now runs on Mondays (04:00 UTC, 06:00 Berlin in summer, 05:00 in
winter), so each run can spend more:

- 1500 AI credits per Copilot session instead of 600, and sessions of up to
  80 minutes instead of 70.
- Still 3 rounds per run: a round is up to 80 min of Copilot plus up to 30
  min of verify, so three fit into the job's 355 minutes (GitHub-hosted jobs
  stop at 360). The loop skips a round that would not finish in time.
- A red run that Copilot did not mark blocked dispatches the next run, which
  continues from the draft PR it just opened, up to 3 runs per sync. That
  gives up to 9 rounds a week without restructuring the job. Worst case is
  3 x 3 x 1500 credits = $135 a week, about what 7 daily runs at 3 x 600
  could cost before.
- Model gpt-6.1-sol (GA in Copilot since 2026-09-29). The org's model list
  of 16 Oct does not offer it yet, so the run falls back to gpt-5.6-sol with
  a warning until an admin enables it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd the lost fork lines

Lessons from landing #158 by hand: a hand-merged patches/ file broke the
install and hid everything after it, vpr typecheck stopped at the first red
package, and a lost fork feature drowned in the audit. The prompt now also
asks for repo-wide fixes and regenerated generated files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Pawl-Patrol
Pawl-Patrol merged commit cdbb4dd into fork Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upstream-Sync repariert sich selbst, statt als roter Draft liegen zu bleiben

1 participant