Skip to content

Add Redis cluster mode support to RedisHook - #71067

Merged
Miretpl merged 2 commits into
apache:mainfrom
FrankYang0529:airflow-10504-redis-cluster
Aug 26, 2026
Merged

Miretpl merged 2 commits into
apache:mainfrom
FrankYang0529:airflow-10504-redis-cluster

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Why

  • Redis Cluster splits the keyspace into 16384 hash slots spread across masters. Any key-based command must reach the node owning that key's slot, otherwise the node answers MOVED instead of serving it.
  • RedisHook only built a standalone redis.Redis client, which does not follow those redirects. So every key-based operation in this provider can fail, depending on where the key happens to hash.
    • RedisTaskHandler (remote logging) — lrange to read, rpush/ltrim/expire to write. Its key is derived from dag id, run id and task id, so each task instance's log targets a different node.
    • RedisKeySensor uses exists to watch key. The key can be dynamic string, so it can be on different nodes.

How

  • Add a cluster connection extra. When set, get_conn() builds a redis.cluster.RedisCluster, which follows MOVED/ASK redirects.
  • Add a startup_nodes extra to supply extra bootstrap nodes.

Verification

  • uv run --project providers/redis pytest providers/redis/tests/unit/redis/hooks/test_redis.py -xvs
  • breeze testing providers-integration-tests --db-reset --integration redis

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread providers/redis/docs/connections.rst Outdated
Comment thread providers/redis/docs/connections.rst Outdated
Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py Outdated
Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py Outdated
Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py
Comment thread providers/redis/docs/connections.rst Outdated
Comment thread providers/redis/src/airflow/providers/redis/get_provider_info.py Outdated
@Miretpl

Miretpl commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Could you also adjust the PR description in terms of why it is needed? Currently, we only know the difference between standalone Redis and cluster deployment and how it handles connections.

@FrankYang0529
FrankYang0529 force-pushed the airflow-10504-redis-cluster branch from e4d7b23 to 254fa27 Compare August 17, 2026 12:43
@FrankYang0529

FrankYang0529 commented Aug 17, 2026 •

Copy link
Copy Markdown
Member Author

@Miretpl Thanks for the review. I addressed all comments and updated PR description.

Comment thread providers/redis/docs/connections.rst Outdated
Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py Outdated
@FrankYang0529
FrankYang0529 force-pushed the airflow-10504-redis-cluster branch from 254fa27 to 07e0ecd Compare August 18, 2026 01:18
@FrankYang0529
FrankYang0529 requested a review from Miretpl August 18, 2026 05:48
@FrankYang0529
FrankYang0529 force-pushed the airflow-10504-redis-cluster branch from 07e0ecd to e500dca Compare August 24, 2026 08:29

@Miretpl Miretpl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just small nit about exception messages and LGTM.

Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py Outdated
Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py Outdated
Comment thread providers/redis/src/airflow/providers/redis/hooks/redis.py Outdated
@FrankYang0529
FrankYang0529 force-pushed the airflow-10504-redis-cluster branch 2 times, most recently from 6680260 to 82a4445 Compare August 26, 2026 08:22
Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529
FrankYang0529 force-pushed the airflow-10504-redis-cluster branch from 82a4445 to 3898c37 Compare August 26, 2026 14:24
@Miretpl
Miretpl merged commit 36016df into apache:main Aug 26, 2026
156 checks passed
@FrankYang0529
FrankYang0529 deleted the airflow-10504-redis-cluster branch August 27, 2026 00:16
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
Signed-off-by: PoAn Yang <payang@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for Redis in Cluster Mode

4 participants