Repository navigation
Officially support HA for scheduler component (AIP-15) #9630
Description
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
- Detect orphaned task instances by SchedulerJob id and heartbeat #10729 Detect orphaned task instances by SchedulerJob id and heartbeat
- Mark task as failed when it fails "sending" in Celery #10881 Mark task as failed when it fails sending in Celery
- Allow CeleryExecutor to "adopt" an orphaned queued or running task #10949 Allow CeleryExecutor to "adopt" an orphaned queued or running task
- Kubernetes executor can adopt tasks from other schedulers #10996 Allow Kubernetes executor to adopt tasks
- Add task adoption to CeleryKubernetesExecutor #11244 Add adoption support in to the new CeleryKubernetesExecutor (added in Add CeleryKubernetesExecutor #10901)
- The big one - scheduler HA pr Fully support running more than one scheduler concurrently #10956
- Perform "mini scheduling run" after task has finished #11498 Add "fast follow" to end of task run (this is a mini "scheduler" run at the end of task execution.)
- Performance optimisation PRs
- Decide if we need Ensure idle database connections don't hang around for ever #10933 or not
- Ensure smart sensor DAG is loaded by scheduler even when DAG serialization is enabled.
Reacted by Aneesh Joseph, Kamil Breguła, Adam McLaurin , Junyoung Park, Ye Jiaqi and Zach Collins
Activity
mik-laj commentedon Aug 2, 2020
@ashb @kaxil Could you please share some more details about the change?
In particular, I would like answers to the following questions:
With these questions, I would like to remove barriers for new contributors who do not follow the activity of the entire community. Hope this helps speed up your work on key features for the new Airflow 2.0.
ashb commentedon Aug 2, 2020
Yes, I have started work on this, and got reasonably far with it (far enough to start running performance tests, but there are a few things I still have as to-do's such as SLAs or per task concurrency limits) but unfortunately I want able to get the code in a situation where I could pr it before going off on paternity leave - as soon as I'm back (September) getting this landed is my top priority.
mik-laj commentedon Sep 14, 2020
Link: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
ryw commentedon Dec 1, 2020
@ashb since you're done w/ work for 2.0 on this issue, maybe close this out and create any new tickets for things you want to do post-2.0?