Skip to content

Deploy multiple (celery) workers with the Helm Chart #34219

Closed
#58547
Closed
Deploy multiple (celery) workers with the Helm Chart#34219
#58547
Labels
area:helm-chartAirflow Helm Chartkind:featureFeature Requests

Description

@RonaldGalea
RonaldGalea
opened on Sep 8, 2023

Description

Currently, the Helm Chart supports a single Celery worker deployment. The CeleryExecutor however, supports multiple queues. Ideally, the Helm Chart should allow deploying multiple Celery workers (different images, different resource allocations) to be able to use queues to send tasks to specialised workers (as already highlighted in the docs)

I am unsure if this is a matter of modifying the chart only or if Airflow really expects only one worker to be there, but in that case I don't see how queues could be used? Thanks a lot in advance.

Use case/motivation

It is often the case that tasks might require different hardware resources/different environments. For instance, if I have task A, which requires 1GB of memory, and task B, which requires 10GB, it's inefficient to run them both on the same Celery worker deployment.

Having multiple deployments would allow using queues to distribute tasks with finer granularity.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. RonaldGalea
    added
    kind:featureFeature Requests
    needs-triagelabel for new issues that we didn't triage yet
    on Sep 8, 2023

  2. boring-cyborg commented on Sep 8, 2023

    @boring-cyborg
    boring-cyborgBot
    on Sep 8, 2023

    Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

  3. hussein-awala commented on Sep 8, 2023

    @hussein-awala
    hussein-awala
    on Sep 8, 2023
    Member

    It would be great if we add this feature to the helm chart, but it won't be an easy task, because to implement it we will need to add a loop for each celery template and support two configuration sources: a new value of type list and the existing one for backward compatibility.
    We also need to update KEDA to do the same and update its queries to scale the different celery clusters based on tasks' queues.

    We can also use the existing value as default configurations for the different clusters, and override only the config provided in the list.

    cc: @jedcunningham wdyt?

    @RonaldGalea Would you like to work on it?

  4. hussein-awala
    added
    area:helm-chartAirflow Helm Chart
    and removed
    needs-triagelabel for new issues that we didn't triage yet
    on Sep 8, 2023

  5. jedcunningham commented on Sep 8, 2023

    @jedcunningham
    jedcunningham
    on Sep 8, 2023
    Member

    At one point we had an issue (or maybe a PR) that talked about the complexities of doing exactly this, but I can't find it right now.

    I don't see (or recall) any blockers. It's just not a trivial thing to add. It would be great to get it added.

  6. RonaldGalea commented on Sep 9, 2023

    @RonaldGalea
    RonaldGalea
    on Sep 9, 2023
    Author

    It would be great if we add this feature to the helm chart, but it won't be an easy task, because to implement it we will need to add a loop for each celery template and support two configuration sources: a new value of type list and the existing one for backward compatibility. We also need to update KEDA to do the same and update its queries to scale the different celery clusters based on tasks' queues.

    We can also use the existing value as default configurations for the different clusters, and override only the config provided in the list.

    cc: @jedcunningham wdyt?

    @RonaldGalea Would you like to work on it?

    Haven't done templating before but I'd be happy to give it a try. I'll see what I can learn this weekend

  7. RonaldGalea commented on Sep 10, 2023

    @RonaldGalea
    RonaldGalea
    on Sep 10, 2023
    Author

    I've had a look at how the helm chart is structured and my approach would be as follows:

    First, identify the minimum options that need to be configurable for the feature to be useful. The ones I'm seeing are:

    • worker image
    • command & args
    • replicas
    • resources
    • autoscaling (keda)

    Since setting the image specifically for workers is not yet configurable - that would be a first standalone PR.

    Next, I would leave the current workers: key exactly as it is (both for backward compatibility and for the fact that there will always be at least one worker type) and introduce an additional-celery-workers: key, where the aforementioned options can be specified and the rest will be the same as for the default worker type.

    I believe most worker-related components can be exactly the same for all workers (at least for now), so they should not require changes:

    • service account
    • service
    • network policy
    • DB connection setup for keda (e.g. pgbouncer network policy)

    What does need to change is Keda - we will have to create additional ScaledObjects to reflect the addition of Deployments/StatefulSets. What I'm a bit unsure of is the query. Currently, it just lists everything from a table named task_instance - does this table contain queue-related info/can it be easily added?

    Let me know if the overall approach sounds reasonable.

  8. guhuajun commented on May 20, 2024

    @guhuajun
    guhuajun
    on May 20, 2024

    Greetings,

    I am evaluating the features in Airflow project. I am still new to the Airflow project. But It's good to see the old friend Celery. It is the project I am familiar with since 2015.

    After reading your words, maybe you can start with simple one, Play Celery with django. It's easily start two workers with docker-compose and having some fun moments. Since we already have DagRun, once the dag files are synced/replicated to all workers, we should have a success.

    Good luck! airflow pools join will be a major milestone for this project. :)

    I've had a look at how the helm chart is structured and my approach would be as follows:

    First, identify the minimum options that need to be configurable for the feature to be useful. The ones I'm seeing are:

    * worker image

* command & args

* replicas

* resources

* autoscaling (keda)

    Since setting the image specifically for workers is not yet configurable - that would be a first standalone PR.

    Next, I would leave the current workers: key exactly as it is (both for backward compatibility and for the fact that there will always be at least one worker type) and introduce an additional-celery-workers: key, where the aforementioned options can be specified and the rest will be the same as for the default worker type.

    I believe most worker-related components can be exactly the same for all workers (at least for now), so they should not require changes:

    * service account

* service

* network policy

* DB connection setup for keda (e.g. pgbouncer network policy)

    What does need to change is Keda - we will have to create additional ScaledObjects to reflect the addition of Deployments/StatefulSets. What I'm a bit unsure of is the query. Currently, it just lists everything from a table named task_instance - does this table contain queue-related info/can it be easily added?

    Let me know if the overall approach sounds reasonable.

  9. ihnokim commented on Apr 18, 2025

    @ihnokim
    ihnokim
    on Apr 18, 2025
    Contributor

    I was wondering if there’s been any recent progress on this issue. This feature would be incredibly helpful for my use case. If things are currently delayed due to a busy schedule, would it be okay if I kindly offered to work on it and submit a pull request, perhaps based on @RonaldGalea ’s suggestions?

  10. potiuk commented on Apr 18, 2025

    @potiuk
    potiuk
    on Apr 18, 2025
    Member

    I was wondering if there’s been any recent progress on this issue. This feature would be incredibly helpful for my use case. If things are currently delayed due to a busy schedule, would it be okay if I kindly offered to work on it and submit a pull request, perhaps based on @RonaldGalea ’s suggestions?

    Sure. that's how it works here.

  11. littlebtc commented on Jun 9, 2025

    @littlebtc
    littlebtc
    on Jun 9, 2025

    I am also thinking about assigning custom images per worker.

  12. VergeDX commented on Jul 4, 2025

    @VergeDX
    VergeDX
    on Jul 4, 2025
    Contributor

    You can use HELM Docs - Post Rendering to add/copy/modify the Official Helm Chart's output.

  13. MagicFrogSJTU commented on Jul 16, 2025

    @MagicFrogSJTU
    MagicFrogSJTU
    on Jul 16, 2025
    Contributor

    Can I deploy multiple airflow helm release to enable multiple celery workers with helm chart?

    • the first one fully deploy all componets
    • the other ones only deploy workers
    • all release use the same redis and postgresql

    @VergeDX will this method be controllable?

  14. eladkal commented on Aug 15, 2025

    @eladkal
    eladkal
    on Aug 15, 2025
    Contributor

    I think that for the simple case of same worker images / service account etc this should be a fairly simple change? It might be best to support at least this one.

    There are use cases that can benefit from that. specifically the common use case of setting different parallelism per worker type like: 10 for small tasks worker and 3 for heavy tasks worker. So we just need to support multiple workers and queues

  15. jscheffl
    closed this as completedin #58547on Jan 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:helm-chartAirflow Helm Chartkind:featureFeature Requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions