Skip to content

fix(ssubscribe): properly resubscribe in case of shard failover - #3098

Merged
nkaradzhov merged 2 commits into
redis:masterfrom
nkaradzhov:cluster-failover-issue
Oct 13, 2025
Merged

fix(ssubscribe): properly resubscribe in case of shard failover#3098
nkaradzhov merged 2 commits into
redis:masterfrom
nkaradzhov:cluster-failover-issue

Conversation

@nkaradzhov

@nkaradzhov nkaradzhov commented Oct 9, 2025

Copy link
Copy Markdown
Collaborator 
1) when RE failover happens, there is a disconnect
2) affected Client reconnects and tries to resubscribe all existing listeners
ISSUE #1: CROSSSLOT Error - client was doing ssubscribe ch1 ch2.. chN which, after the failover could result in CROSSSLOT ( naturally, becasuse now some slots could be owned by other shards )
FIX: send one ssubscribe command per channel instead of one ssubscribe for all channels
ISSUE #2: MOVED Error - some/all of the channels might be moved somewhere else
FIX: 1: propagate the error to the Cluster. 2: Cluster rediscovers topology.
3: Extract all existing subscriptions from all pubsub clients and resubscribe
over the new topology.

fixes: #2902

@nkaradzhov
nkaradzhov force-pushed the cluster-failover-issue branch from ff426bc to 2a766b7 Compare October 9, 2025 14:15
@nkaradzhov
nkaradzhov marked this pull request as draft October 9, 2025 14:15
@nkaradzhov

nkaradzhov commented Oct 9, 2025

Copy link
Copy Markdown
Collaborator Author

run sharded pub sub tests

@uglide

uglide commented Oct 9, 2025

Copy link
Copy Markdown
Contributor
Testcase Errors Failures Skipped Total
Root Suite 0 0 0 0
Sharded Pub/Sub E2E 0 0 0 0
Single Subscriber 0 0 0 3
Multiple Subscribers 0 1 0 2

---- Details for maintainers

@nkaradzhov
nkaradzhov force-pushed the cluster-failover-issue branch from 2a766b7 to b07e39b Compare October 10, 2025 08:48
@nkaradzhov

nkaradzhov commented Oct 10, 2025

Copy link
Copy Markdown
Collaborator Author

run sharded pub sub tests

@nkaradzhov
fix(ssubscribe): properly resubscribe in case of shard failover in RE…
06fc7b7 
… TODO cleanup debug logs

1) when RE failover happens, there is a disconnect
2) affected Client reconnects and tries to resubscribe all existing listeners
ISSUE #1: CROSSSLOT Error - client was doing ssubscribe ch1 ch2.. chN which, after the failover could result in CROSSSLOT ( naturally, becasuse now some slots could be owned by other shards )
FIX: send one ssubscribe command per channel instead of one ssubscribe for all channels
ISSUE #2: MOVED Error - some/all of the channels might be moved somewhere else
FIX: 1: propagate the error to the Cluster. 2: Cluster rediscovers topology.
3: Extract all existing subscriptions from all pubsub clients and resubscribe
over the new topology.

fixes: redis#2902
@uglide

uglide commented Oct 10, 2025

Copy link
Copy Markdown
Contributor
Testcase Errors Failures Skipped Total
Root Suite 0 0 0 0
Sharded Pub/Sub E2E 0 0 0 0
Single Subscriber 0 0 0 3
Multiple Subscribers 0 0 0 2

---- Details for maintainers

@nkaradzhov
nkaradzhov force-pushed the cluster-failover-issue branch from b07e39b to 4423777 Compare October 10, 2025 09:16
vaibhavkumar-sf
vaibhavkumar-sf reviewed Oct 10, 2025
View reviewed changes
Comment thread packages/client/lib/client/index.ts
vaibhavkumar-sf
vaibhavkumar-sf reviewed Oct 10, 2025
View reviewed changes
Comment thread packages/client/lib/cluster/cluster-slots.ts Outdated
@vaibhavkumar-sf

vaibhavkumar-sf commented Oct 10, 2025

Copy link
Copy Markdown

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ No key issues found

The PR code appears to be well-structured and follows good practices. The reviewer should still validate the logic and ensure the implementation meets requirements.

bobymicroby
bobymicroby reviewed Oct 10, 2025
View reviewed changes
Comment thread packages/client/lib/cluster/cluster-slots.ts
Comment thread packages/client/lib/cluster/cluster-slots.ts Outdated
bobymicroby
bobymicroby approved these changes Oct 10, 2025
View reviewed changes

@bobymicroby bobymicroby left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nkaradzhov

nkaradzhov commented Oct 10, 2025

Copy link
Copy Markdown
Collaborator Author

run sharded pub sub tests

@uglide

uglide commented Oct 10, 2025

Copy link
Copy Markdown
Contributor
Testcase Errors Failures Skipped Total

---- Details for maintainers

@nkaradzhov

nkaradzhov commented Oct 10, 2025

Copy link
Copy Markdown
Collaborator Author

run sharded pub sub tests

@uglide

uglide commented Oct 10, 2025

Copy link
Copy Markdown
Contributor
Testcase Errors Failures Skipped Total
Root Suite 0 0 0 0
Sharded Pub/Sub E2E 0 0 0 0
Single Subscriber 0 0 0 3
Multiple Subscribers 0 0 0 2

---- Details for maintainers

@nkaradzhov
nkaradzhov force-pushed the cluster-failover-issue branch from 113229c to 697402f Compare October 10, 2025 15:22
@nkaradzhov

nkaradzhov commented Oct 10, 2025

Copy link
Copy Markdown
Collaborator Author

run sharded pub sub tests

@uglide

uglide commented Oct 10, 2025

Copy link
Copy Markdown
Contributor
Testcase Errors Failures Skipped Total
Root Suite 0 0 0 0
Sharded Pub/Sub E2E 0 0 0 0
Single Subscriber 0 0 0 3
Multiple Subscribers 0 0 0 2

---- Details for maintainers

@nkaradzhov
nkaradzhov force-pushed the cluster-failover-issue branch from 697402f to 6187101 Compare October 13, 2025 08:01
@nkaradzhov
nkaradzhov force-pushed the cluster-failover-issue branch from 6187101 to 9bf5f7b Compare October 13, 2025 08:32
@nkaradzhov

nkaradzhov commented Oct 13, 2025

Copy link
Copy Markdown
Collaborator Author

run sharded pub sub tests

@uglide

uglide commented Oct 13, 2025

Copy link
Copy Markdown
Contributor
Testcase Errors Failures Skipped Total
Root Suite 0 0 0 0
Sharded Pub/Sub E2E 0 0 0 0
Single Subscriber 0 0 0 3
Multiple Subscribers 0 0 0 2

---- Details for maintainers

@nkaradzhov
nkaradzhov marked this pull request as ready for review October 13, 2025 08:55
@nkaradzhov
nkaradzhov merged commit 96d6445 into redis:master Oct 13, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bobymicroby bobymicroby bobymicroby approved these changes
+1 more reviewer
@vaibhavkumar-sf vaibhavkumar-sf vaibhavkumar-sf left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nkaradzhov @uglide @vaibhavkumar-sf @bobymicroby