AIP-51 - Misc. Compatibility Checks

[rkarish]

Closes: #27930

[o-nikolas]

Thanks for the PR! I've left some comments and requests :)

[o-nikolas]

You'll also need to set a value for supports_sentry in this class as well since it does not implement the BaseExecutor interface (see the discussion starting from here for more context). You'll need to ensure both new fields are set on all the executors which don't inherit from BaseExecutor (LocalKubernetesExecutor as well) and add the appropriate unit tests for those.

[o-nikolas]

I wonder if we should actually keep this in for backwards compatibility (with a comment describing that it is deprecated) 🤔 i.e. someone in the community could be depending on this list existing. I wonder what others think about this.

[rkarish]

I searched the repository for references and it looked like it was safe to delete, but I didn't think about other users in the community being dependent on this. I think that adding a comment here and waiting to clean this up at a later time is a safer approach.

[pierrejeambrun]

Maybe add a # TODO: Remove in Airflow 3.0. We have a couple of those in the repo and will help to not miss this spot when releasing the next major.

[o-nikolas]

I think we want the default to be True, no? The previous code assumed all executors were picklable unless it was present in the list of UNPICKLEABLE_EXECUTORS. So any executor that goes through that code should be assumed to be picklable unless it says otherwise. WDYT?

[rkarish]

I agree that this makes more sense. For some reason I went with the opposite 🙃

[o-nikolas]

Do we actually want this to be True? The LocalExecutor is not picklable and so far we've gone with True for these only if all sub executors are True. WDYT?
There is some context in a PR comment here (from @pierrejeambrun's PR for is_local)

[o-nikolas]

You can use the helper method I added for this ExecutorLoader.import_default_executor_cls()

[rkarish]

Thanks for the feedback Niko! I will clean this up and let you know when I am finished.

[rkarish]

Hey Niko, I've submitted some more changes based on the feedback and requests. I reworked the pickling support attribute, switched to using your get executor class helper method, and implemented tests for pickling and sentry support for all of the executors. CI pipeline failed so I rebased the branch off main which fixed a failure in integration tests, but it's still failing static checks. Pre-commit was passing for me locally before rebasing, seems like something changed here. When I ran it locally after rebase it was changing several files so I didn't commit those changes.

Also, I didn't mean to remove all of the other reviewers, I clicked on request re-review and it did it automatically.

[o-nikolas]

Hey Niko, I've submitted some more changes based on the feedback and requests. I reworked the pickling support attribute, switched to using your get executor class helper method, and implemented tests for pickling and sentry support for all of the executors. CI pipeline failed so I rebased the branch off main which fixed a failure in integration tests, but it's still failing static checks. Pre-commit was passing for me locally before rebasing, seems like something changed here. When I ran it locally after rebase it was changing several files so I didn't commit those changes.

Also, I didn't mean to remove all of the other reviewers, I clicked on request re-review and it did it automatically.

Thanks @rkarish! Sorry for the delayed response but I have been on year end/holiday vacation. I'm just trying to get caught up now. I'll have a look at this PR very soon 🙏

[rkarish]

You're welcome and Happy New Year! I figured you were on vacation, no rush on the review!

[o-nikolas]

The changes look great! Perhaps @pierrejeambrun can be the second reviewer/approver, he's been very involved in the executor decoupling work.

[rkarish]

That would be great to get another set of eyes on this. Congratulations on becoming a Committer by the way! Would love for you to do the honors on this one 😄 .

[pierrejeambrun]

Just one suggestion, otherwise LGTM :)

[pierrejeambrun]

Maybe add a # TODO: Remove in Airflow 3.0. We have a couple of those in the repo and will help to not miss this spot when releasing the next major.

[o-nikolas]

That would be great to get another set of eyes on this. Congratulations on becoming a Committer by the way! Would love for you to do the honors on this one 😄.

Thanks @rkarish! And thanks to you for helping on AIP-51 😀