Repository navigation
Document per-set volumes for Celery workers - #74913
himanshuchawla0807-gif wants to merge 3 commits into
Conversation
Operators need to mount different input data for different queues without changing every worker set. Generated-by: OpenAI Codex
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
shahar1
left a comment
There was a problem hiding this comment.
Thanks for picking up #63179. I checked the new section against workersMergeValues in chart/templates/_helpers.yaml and the worker templates: the inheritance and list-replacement rules (a non-empty list replaces, an empty list keeps the inherited value) are described correctly, and the example matches that logic. One point worth fixing: the example disables the default worker set without saying that the default queue then has no consumer. There is also a small note on PVC access modes. CI has not run on this PR yet (first-time contributor workflows still need approval), so the docs build is unverified here.
Smaller observations
- See inline comments on
chart/docs/customizing-workers.rst:74,chart/docs/customizing-workers.rst:72.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The findings
below are observations, not blockers; an Apache Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.More on how Apache Airflow handles maintainer review:
Contributing guide.
Clarify multi-node PVC requirements for the inherited replica and autoscaling settings. Generated-by: OpenAI Codex
Document how to give separate Celery worker sets different volumes and mounts. The chart already supports this, but the worker customization guide only shows shared settings, leaving users without an example for independent data volumes.
Add a two-queue example using existing PVCs and explain inheritance, list replacement, preserving the default queue consumer, and PVC access modes for multiple worker nodes. No chart behavior changes.
closes: #63179
Validation:
prek run --from-ref upstream/main --stage pre-commitpassed, including Helm lint and Kubernetes schema validation.prek run --from-ref upstream/main --stage manual --skip compile-ui-assets-dev --skip view-skill-eval --skip run-skill-eval-codexpassed.breeze verify --jsonselectedbreeze build-docs helm-chart; that build passed locally in Breeze on the rebased branch.Was generative AI tooling used to co-author this PR?
Generated-by: OpenAI Codex following the guidelines
Drafted-by: OpenAI Codex (no human review before posting)