Start porting DAG definition code to the Task SDK

[ashb]

By "definition code" we mean anything needed at definition/parse time, leaving
anything to do with scheduling time decisions in Airflow's core.

Also in this PR I have attempted to keep it to only porting defintiion code
for simple DAGs, leaving anything to do with mapped tasks or execution time in
core for now, but a few things "leaked" across.

And as the goal of this PR is to go from working state to working state some
of the code in Task SDK still imports from "core" (various types, enums or
helpers) that will need to be resolved before 3.0 release, but it is fine for
now.

I'm also aware that the class hierarchy with
airflow.models.baseoperator.BaseOperator (and to a lesser extend with DAG) in
particular is very messy right now, and we will need to think how we want to
add on the scheduling-time functions etc, as I'm not yet sold that having Core
Airflow depend upon the Task-SDK classes/import the code is the right
structure, but we can address that later

We will also need to addresses the rendered docs for the Task SDK in a future
PR -- the goal is that "anything" exposed on airflow.sdk directly is part of
the public API, but right now the renedered docs show DAG as
airflow.sdk.definitions.dag.DAG which is certainly not what we want users to
see.

closes #43011

[ashb]

This is very much at the "boring ground work stage" - once we've got this covered we can add some of the execution changes which is not of the n"execution interface" side where it gets fun

[jscheffl]

Some comments. In general looks great!

A bit of a challenge to review such a big PR - and I assume it is just the starting :-D - if possible would be cool to split this a bit up for better review, e.g.

• One PR just the skeleton and AbstractOperator or so
• One PR for BaseOperator
• One PR for DAG only
• Maybe one for Nodes/TaskGroup etc clubbing the rest together.

Anyway after some time was able to digest. Would offer also a full review if needed. And please don't take my early comments too serious more or less thinking out loud.

[ashb]

@kaxil @jscheffl I've just pushed latest changes to this. Probably not worth looking at it again until I've finished it and have a decent idea that the tests are passing locally, but in the last commit I've just pushed I have removed the old code I've ported over, so it might be slightly clearer where I've just moved code.

[ashb]

There are lots more things to move over to the TaskSDK before AIP-72 will be complete which we will track in the project board, but since this is the big base, and slow to test due to unavoidable wide reaching changes we will merge this and follow up with future changes for the rest (things like Mapped operator, DagParam etc)

[kaxil]

Merging this, one of the failures around k8s test that took longer than an hour looks like a transient one:

Progress: airflow-python-3.9-v1.28.13   Error during running tests for Python 3.9, Kubernetes v1.28.13
Progress: airflow-python-3.10-v1.29.8   Image: "ghcr.io/apache/airflow/main/prod/python3.10-kubernetes" with ID "sha256:1547fde10197845f0938fd7d0fc5977273a4871e016c8102ce9c8eb33010d064" not [...]

If for some reason it fails on main too, we will debug it