Add draining state for graceful Dag pausing

[dheerajturaga]

Pausing a Dag today is abrupt: it stops the Dag mid-flight, letting running tasks
finish but stranding queued and downstream tasks until it is unpaused. There is no
way to say "stop starting new scheduled runs, but let the work in flight finish."

That gap bites hardest during upgrades and maintenance windows. The workaround in
the wild is to rewrite every Dag's schedule to None, wait for runs to drain, do
the work, then put the schedules back — invasive, and easy to get wrong.

This adds a draining scheduling state that expresses it directly:

• Draining blocks new scheduler-created Dag runs, including scheduled and
  asset-triggered runs, while letting task instances in queued and running runs
  continue to be scheduled.
• Explicit triggers remain available while draining, matching paused-Dag behavior.
  A manual, operator-triggered, asset-materialization, or backfill request extends
  the drain until the resulting work finishes.
• The scheduler moves a draining Dag to paused once no unfinished runs remain and
  no newly accepted backfill is still initializing, recording a drain_completed
  audit log entry.

Draining is transient, not a third resting state. It always converges to
paused; the resting states are still active and paused. The UI reflects that —
the pause control stays a binary toggle, and drain is offered at the one moment it
carries meaning: flipping a Dag off while it still has unfinished runs asks whether
to drain then pause or pause now. A Dag with nothing running pauses in a single
click as before, since the two are equivalent there.

API. DAGResponse gains a scheduling_state field. PATCH /dags/{dag_id}
accepts either is_paused or scheduling_state and rejects a body carrying both.
Existing is_paused clients are unaffected.

Migration. Adds dag.is_draining (non-null, default false) with an index and a
NOT (is_paused AND is_draining) check constraint. It also adds a supporting index
for identifying an initializing backfill by Dag.

closes: #22006

Screenshots:

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5) and Claude Code (Opus 5)

Generated-by: Codex (GPT-5), Claude Code (Opus 5) following the guidelines

[eladkal]

Tested locally. Works nicely!
I have some comments but we can talk about them post merge

[github-actions]

Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	airflow-ctl/v0-1-test

You can attempt to backport this manually by running:

cherry_picker 35f33ed airflow-ctl/v0-1-test

This should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.