Skip to content

fix(frontend): sync global sysvar commits across CNs - #27534

Closed
ck89119 wants to merge 18 commits into
matrixorigin:mainfrom
ck89119:issue-26793-main
Closed

fix(frontend): sync global sysvar commits across CNs#27534
ck89119 wants to merge 18 commits into
matrixorigin:mainfrom
ck89119:issue-26793-main

Conversation

@ck89119

@ck89119 ck89119 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

Fixes #26793

What this PR does / why we need it:

A successful SET GLOBAL catalog commit could be followed by a new session on another CN reading an older catalog snapshot. Proxy connection reuse also reset backend sessions only when they entered the cache, so a global-variable update committed while a backend was idle could still be missed at handoff.

This PR:

  • fences a committed global-system-variable update against every CN that Proxy can route by reusing the existing SyncCommit query command;
  • fences every HAKeeper CNStore with a query address, including Draining/Drained and TimeoutState stores that may still be routable through Proxy's independently refreshed discovery cache;
  • uses the exact transaction commit timestamp and publishes the local global-variable cache only after all target CNs acknowledge the fence;
  • makes local commit-timestamp synchronization context-aware and propagates cancellation, timeout, and logtail wait errors instead of fatal-looping;
  • advances a CN-local global-variable cache publication epoch before SyncCommit ACK; catalog refreshes retry whenever either the epoch or local mutation generation changes, and stamp a per-variable publication floor with the applied commit TS;
  • publishes local SET GLOBAL cache values with the same commit TS used by the fence; if this CN has already applied a newer remote fence, the delayed setter creates and closes a background executor and reloads catalog under a stable epoch instead of directly publishing its older value;
  • refreshes cached Proxy backend sessions again at PopContext handoff, after the previous generation was cleaned at Push, and discards the backend on refresh failure or cancellation;
  • introduces MORPC protocol version 29 for context-aware SyncCommit (after main assigned v27/v28), preflights every target CN with GetProtocolVersion, and sends no Sync RPC if any target still has the legacy fatal-on-timeout handler;
  • requires protocol-v29 Sync responses to report an applied timestamp at least as new as the requested fence.

No new protobuf field, HAKeeper state machine, Proxy routing lease, or durable outbox is introduced.

Validation

  • go test and full -race runs passed for ./pkg/txn/client, ./pkg/queryservice/client, ./pkg/cnservice, ./pkg/frontend, and ./pkg/proxy.
  • Focused cache-generation, cross-fence publication, membership, protocol-gating, cancellation, and lifecycle tests passed with -race -count=100.
  • go list, go build, go vet, and make build passed.
  • Real 2-CN + Proxy validation passed in both CN directions.
  • Deterministic publication regressions cover an old refresh crossing a fence, the no-existing-entry case, an existing old cache interrupted by unrelated fence E2, same-CN reversed TS1/TS2 completion, and two independent CN manager/cache instances where remote TS2 reaches CN1 before its delayed TS1 setter returns.
  • The stale-setter executor test keeps the real ExeSqlInBgSes implementation and verifies executor creation, use, and closure; membership tests require Working/Unknown/Draining/Drained/TimeoutState stores with query addresses to be fenced.
  • A deterministic Cartesian A/B test verifies that an unrelated local publication cannot make a refresh return a shared cache whose other variable predates the completed fence; generation conflicts retry and honor cancellation.
  • Deterministic idle-cache validation created eight cached backends at global value 1, committed value 0 through a direct CN connection, and then reused cached backends through Proxy 16 times; all 16 sessions read 0 across both CNs.
  • CN timeout and restart validation confirmed bounded failure and the latest global value on the first routed session after restart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/api-change kind/bug Something isn't working size/XL Denotes a PR that changes [1000, 1999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Proxy BVT can read stale lower_case_table_names on a new CN session

2 participants