Remove schedule downstream tasks after execution (aka "mini scheduler")

[ashb]

This has been questionable how much benefit it actually had, but with the move towards task DB isolation in Airflow 3 we won't be able to keep this anymore (as we didn't when AIP-44 DB isolation was enabled), so lets remove it now.

[tirkarthi]

Docs portion can also be removed.

airflow/docs/apache-airflow/administration-and-deployment/scheduler.rst

Lines 386 to 389 in b89f43e

	- :ref:`config:scheduler__schedule_after_task_execution`
	Should the Task supervisor process perform a "mini scheduler" to attempt to schedule more tasks of
	the same DAG. Leaving this on will mean tasks in the same DAG execute quicker,
	but might starve out other DAGs in some circumstances.

[kaxil]

Doc needs to be removed too but otherwise lgtm

[ashb]

Docs portion can also be removed.

airflow/docs/apache-airflow/administration-and-deployment/scheduler.rst

Lines 386 to 389 in b89f43e

	- :ref:`config:scheduler__schedule_after_task_execution`
	Should the Task supervisor process perform a "mini scheduler" to attempt to schedule more tasks of
	the same DAG. Leaving this on will mean tasks in the same DAG execute quicker,
	but might starve out other DAGs in some circumstances.

Thanks - I was grepping based on config option, guess I missed some

[raphaelauv]

In multiple context of high number of concurrent task ( more than 1000 and around 50 new task by sec ) we had problems and by disabling the mini scheduler things are running way more smoothly,

maybe we could deprecate the option in 2.10.4 and set it to false in 2.11.0 , wdyt ?

[kaxil]

In multiple context of high number of concurrent task ( more than 1000 and around 50 new task by sec ) we had problems and by disabling the mini scheduler things are running way more smoothly,

maybe we could deprecate the option in 2.10.4 and set it to false in 2.11.0 , wdyt ?

Yeah good point, fancy a PR to 2-10-test?

[jscheffl]

In multiple context of high number of concurrent task ( more than 1000 and around 50 new task by sec ) we had problems and by disabling the mini scheduler things are running way more smoothly,
maybe we could deprecate the option in 2.10.4 and set it to false in 2.11.0 , wdyt ?

Yeah good point, fancy a PR to 2-10-test?

I think deprecating the option as warning would be good. I also see it needs a re-work as with AIP-72 the function can not be "distributed on worker" as in the past.

Nevertheless when scheduling MANY tasks this was really a performance boost comparing Airflow v1 to Airflow v2 when this was added. Waiting for scheduler loop to schedule the next is really slowing down DAGs... so maybe we need a (future) similar mechanism to schedule next tasks immediately after one has finished to have the same low-latency like in the past... on the backend of course... (like a notification queue where it makes most-sense to schedule next because one task just finished...)