Skip to content

Document per-set volumes for Celery workers - #74913

Open
himanshuchawla0807-gif wants to merge 3 commits into
apache:mainfrom
himanshuchawla0807-gif:docs/celery-worker-set-volumes
Open

himanshuchawla0807-gif wants to merge 3 commits into
apache:mainfrom
himanshuchawla0807-gif:docs/celery-worker-set-volumes

Conversation

@himanshuchawla0807-gif

@himanshuchawla0807-gif himanshuchawla0807-gif commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Document how to give separate Celery worker sets different volumes and mounts. The chart already supports this, but the worker customization guide only shows shared settings, leaving users without an example for independent data volumes.

Add a two-queue example using existing PVCs and explain inheritance, list replacement, preserving the default queue consumer, and PVC access modes for multiple worker nodes. No chart behavior changes.

closes: #63179

Validation:

  • Rendered the revised example as StatefulSets and Deployments, checking all six worker workloads, default worker presence, distinct per-set PVCs and read-only mounts. Earlier inheritance/list-replacement controls also passed before review.
  • Regular static checks: prek run --from-ref upstream/main --stage pre-commit passed, including Helm lint and Kubernetes schema validation.
  • Manual checks: prek run --from-ref upstream/main --stage manual --skip compile-ui-assets-dev --skip view-skill-eval --skip run-skill-eval-codex passed.
  • breeze verify --json selected breeze build-docs helm-chart; that build passed locally in Breeze on the rebased branch.

Was generative AI tooling used to co-author this PR?
  • Yes, OpenAI Codex

Generated-by: OpenAI Codex following the guidelines


Drafted-by: OpenAI Codex (no human review before posting)

Operators need to mount different input data for different queues without changing every worker set.

Generated-by: OpenAI Codex
@boring-cyborg

boring-cyborg Bot commented Oct 17, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up #63179. I checked the new section against workersMergeValues in chart/templates/_helpers.yaml and the worker templates: the inheritance and list-replacement rules (a non-empty list replaces, an empty list keeps the inherited value) are described correctly, and the example matches that logic. One point worth fixing: the example disables the default worker set without saying that the default queue then has no consumer. There is also a small note on PVC access modes. CI has not run on this PR yet (first-time contributor workflows still need approval), so the docs build is unverified here.

Smaller observations

  • See inline comments on chart/docs/customizing-workers.rst:74, chart/docs/customizing-workers.rst:72.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The findings
below are observations, not blockers; an Apache Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review:
Contributing guide.

Comment thread chart/docs/customizing-workers.rst Outdated
Comment thread chart/docs/customizing-workers.rst
Clarify multi-node PVC requirements for the inherited replica and autoscaling settings.

Generated-by: OpenAI Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DOCS] Helm Chart : Differents Volumes/VolumeMount for each Celery Workers Sets

2 participants