From fa1fe6bae0dde5a84806b7a63d1eeb4c08a7c91d Mon Sep 17 00:00:00 2001 From: Himanshu Chawla <241715894+himanshuchawla0807-gif@users.noreply.github.com> Date: Fri, 16 Oct 2026 15:25:22 +0530 Subject: [PATCH 1/2] Document separate volumes for Celery worker sets Operators need to mount different input data for different queues without changing every worker set. Generated-by: OpenAI Codex --- chart/docs/customizing-workers.rst | 44 ++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/chart/docs/customizing-workers.rst b/chart/docs/customizing-workers.rst index 9a201eb5111f2..da83906d8c405 100644 --- a/chart/docs/customizing-workers.rst +++ b/chart/docs/customizing-workers.rst @@ -32,6 +32,50 @@ For example, to set resources on workers for CeleryExecutor: limits: cpu: 1 +Separate Celery worker sets +--------------------------- + +Use ``workers.celery.sets`` to configure workers for different queues. Each set +inherits ``workers.celery`` settings and can override them, including +``extraVolumes`` and ``extraVolumeMounts``. For example, mount different existing +PersistentVolumeClaims for two worker sets: + +.. code-block:: yaml + :caption: values.yaml + + executor: CeleryExecutor + workers: + celery: + enableDefault: false + sets: + - name: analytics + queue: analytics + extraVolumes: + - name: input-data + persistentVolumeClaim: + claimName: analytics-input + extraVolumeMounts: + - name: input-data + mountPath: /opt/airflow/input + readOnly: true + - name: research + queue: research + extraVolumes: + - name: input-data + persistentVolumeClaim: + claimName: research-input + extraVolumeMounts: + - name: input-data + mountPath: /opt/airflow/input + readOnly: true + +Create both PersistentVolumeClaims before deploying the chart, and route tasks +to the corresponding queue. ``enableDefault: false`` disables the default worker +set, so this example creates only the ``analytics`` and ``research`` sets. +Non-empty list overrides such as ``extraVolumes`` and ``extraVolumeMounts`` replace +the inherited list, so include all additional volumes and mounts needed by each +set. An empty list retains the inherited value. + Custom ``pod_template_file`` ---------------------------- From c05e6f13fc9e1d2e182648c2fb80b6ebe25b3122 Mon Sep 17 00:00:00 2001 From: Himanshu Chawla <241715894+himanshuchawla0807-gif@users.noreply.github.com> Date: Sat, 17 Oct 2026 21:47:46 +0530 Subject: [PATCH 2/2] Keep default queue workers in the per-set volume example Clarify multi-node PVC requirements for the inherited replica and autoscaling settings. Generated-by: OpenAI Codex --- chart/docs/customizing-workers.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/chart/docs/customizing-workers.rst b/chart/docs/customizing-workers.rst index da83906d8c405..f0238cd42cba4 100644 --- a/chart/docs/customizing-workers.rst +++ b/chart/docs/customizing-workers.rst @@ -46,7 +46,6 @@ PersistentVolumeClaims for two worker sets: executor: CeleryExecutor workers: celery: - enableDefault: false sets: - name: analytics queue: analytics @@ -70,8 +69,11 @@ PersistentVolumeClaims for two worker sets: readOnly: true Create both PersistentVolumeClaims before deploying the chart, and route tasks -to the corresponding queue. ``enableDefault: false`` disables the default worker -set, so this example creates only the ``analytics`` and ``research`` sets. +to the corresponding queue. The default worker set remains enabled to consume +tasks sent to the ``default`` queue. Each set inherits replica and autoscaling +settings. If its pods can run on multiple nodes, use a claim that supports +read-only mounts from multiple nodes, such as ``ReadOnlyMany``, with a compatible +storage driver. Non-empty list overrides such as ``extraVolumes`` and ``extraVolumeMounts`` replace the inherited list, so include all additional volumes and mounts needed by each set. An empty list retains the inherited value.