Add compatibility tests for the celery executor

[shahar1]

Body

See: #65752 (comment)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[leeyspaul]

I can look into this one. I’ll start with the Celery case from #65752 and see where executor tests fit into the existing provider compatibility test setup. I’ll keep the first pass small and check back in here with updates.

[leeyspaul]

First small scoped PR is here: #65849

[leeyspaul]

@shahar1

I looked into this while working on #65849.

My current understanding is that we already have a provider compatibility matrix via PROVIDERS_COMPATIBILITY_TESTS_MATRIX, and it runs provider tests against older supported Airflow versions. The gap in #65752 seems to be that the existing Celery executor tests did not directly exercise the worker-side execute_workload deserialization and dispatch path that broke on Airflow 3.2.

So I interpreted this issue as: add focused executor/provider tests that run under the existing provider compatibility matrix, rather than create a separate executor-specific compatibility matrix.

Is that the intended direction, or were you thinking of broader executor-specific compatibility coverage beyond the existing provider compat matrix?

Tagging @potiuk as well based on #65752 (comment)

[potiuk]

Yes. Good assesment.

[leeyspaul]

@potiuk @shahar1

OK, so I'm continuing focus on CeleryExecutor. Given the minimum version announcement: https://github.com/apache/airflow/blob/main/providers/PROVIDER_RELEASES.rst

Suggested scope for Celery path within the current compat matrix:

• current Celery provider source against Airflow 3.1.x
• current Celery provider source against Airflow main/current dev

Specifically:

• import-time compatibility for celery_executor / celery_executor_utils
• workload execution branches across Airflow 3.0 / 3.1 / 3.2+
• version-gated symbols like ExecutorWorkload, BaseExecutor.run_workload, ExecuteCallback, ExecutorConf, and external_executor_id

Out of scope:

• exhaustive Airflow 2.x coverage, since Airflow 2 reached EOL on April 22, 2026, and provider policy says the minimum provider Airflow version is expected to move to Airflow 3.1.0 after May 20, 2026.

I'll continue to put up PRs linked to this issue, or we can create a sub-issue. Do let me know if you want to change directions a bit here, otherwise I'll continue with test coverage.

[SameerMesiah97]

@shahar1 @potiuk @leeyspaul

If scope of this issue refers to all executors including those for AWS (Lambda, Batch, ECS), Edge and K8s (my interpretation is that that is indeed the case), then you need to be aware there is ongoing work on several of them to add support for callback execution. Only the Lambda PR has been merged so far (#63035) but the remaining ones are in WIP (please refer to issue #62887) so if @leeyspaul or any other contributor wants to add new tests they may have to wait for those PRs to merge. Perhaps it would be better to link this issue on the existing PRs so that the authors can add the tests themselves. Alternatively, we could just limit the scope to celery for now.

[potiuk]

Perhaps it would be better to link

You just did :)