Skip to content

Fix backup schedule timeout and scheduling state - #818

Merged
mattlord merged 2 commits into
mainfrom
backup_schedule_fixes
Aug 28, 2026
Merged

mattlord merged 2 commits into
mainfrom
backup_schedule_fixes

Conversation

@mattlord

@mattlord mattlord commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes #817

Fix backup-schedule create/delete loops caused by stale cron timestamps and lost Job history. The controller now uses persisted per-strategy status as its scheduling cursor and measures timeouts from the actual Job start time.

This prevents completed or timed-out Jobs from resetting scheduling state, including when successfulJobsHistoryLimit: 0 or the operator catches up a missed cron slot after downtime.

Changes

  • Use status.lastScheduledTimes as the durable scheduling cursor.
  • Repair stale status from newer Job annotations before cleaning up Job history.
  • Measure jobTimeoutMinute from Job.status.startTime; Jobs that have not started cannot time out.
  • Support jobTimeoutMinute: -1 to disable timeout cleanup. The default remains 10 minutes.
  • Delete timed-out Jobs with foreground propagation. Under concurrencyPolicy: Forbid, a terminating Job remains active until Kubernetes removes its Pods, blocking a replacement.
  • Preserve vtbackup PVCs when Job deletion fails.
  • Skip repeated timeout cleanup for Jobs already being deleted.
  • Update VitessBackupSchedule status only when it changes.

Use persisted status and Job annotations as the scheduling cursor, and
measure timeouts from the Job start time. Delete timed-out Jobs with
foreground propagation to prevent overlapping replacements.

Signed-off-by: Matt Lord <mattalord@gmail.com>
Copilot AI lite review requested due to automatic review settings August 28, 2026 15:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts VitessBackupSchedule scheduling and timeout behavior to use durable state (persisted status and Job annotations) as the scheduling cursor, and to measure job timeouts from the Kubernetes Job start time. It also changes timed-out Job deletion semantics to use foreground propagation to prevent overlapping replacement jobs.

Changes:

  • Use vbsc.Status.LastScheduledTimes (and repair it from Job annotations when newer) as the scheduling cursor.
  • Measure timeout from job.Status.StartTime and delete timed-out Jobs with foreground propagation.
  • Add regression tests for scheduling cursor persistence/repair and timeout semantics; update API docs to reflect the new timeout definition.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pkg/controller/vitessbackupschedule/vitessbackupschedule_controller.go Implements status/annotation cursor repair and switches timeout calculation + deletion propagation for timed-out Jobs.
pkg/controller/vitessbackupschedule/vtctldclient_backup_test.go Updates existing timeout test to set Job Status.StartTime and pass now into removeTimeoutJobs.
pkg/controller/vitessbackupschedule/scheduling_regression_test.go Adds regression coverage for status-as-cursor behavior and startTime-based timeouts (including propagation policy).
pkg/apis/planetscale/v2/vitessbackupschedule_types.go Updates API field documentation for JobTimeoutMinutes to specify startTime-based measurement.
docs/api/index.html Regenerates/updates rendered API docs for the new timeout definition.
docs/api.md Regenerates/updates markdown API docs for the new timeout definition.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Signed-off-by: Matt Lord <mattalord@gmail.com>
@mattlord
mattlord requested review from a team, frouioui and mhamza15 August 28, 2026 16:45
@mattlord
mattlord merged commit 168d0fd into main Aug 28, 2026
13 checks passed
@mattlord
mattlord deleted the backup_schedule_fixes branch August 28, 2026 19:55
mattlord added a commit that referenced this pull request Sep 2, 2026
Backport of #818 to release-2.17.

Signed-off-by: Matt Lord <mattalord@gmail.com>
mattlord added a commit that referenced this pull request Sep 2, 2026
[release-2.17] Fix backup schedule timeout and scheduling state (#818)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VitessBackupSchedule: stale-slot catch-up jobs are instantly deleted as "timed out", causing a create/delete loop and missed backups

4 participants