Release Airflow 3.0

[kaxil]

Hello all,

Creating a meta-issue to track all the projects related to Airflow 3 and pointers on how contributors can help in this effort.

The Home Page for Airflow 3 discussions is: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0

Workstreams: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams

How to participate & help?

Check this doc and find items without an owner; this is the workstream that needs someone in the community to lead. Comment & tag me if you are interested in any of the workstreams.

There are various Airflow Improvement Proposals (AIPs) being worked on right now:

Name	Theme	GitHub Project	Status
AIP-72: Task Execution Interface	Run anywhere in any language	link	✅
AIP-69: Edge Executor	Run anywhere in any language	link	✅
AIP-65: Improve DAG history in UI	DAG versioning	link	✅
AIP-66: DAG Bundles & Parsing	DAG versioning	link	✅
AIP-38: Modern Web Application	UI modernization	link	✅
AIP-84: Internal API for UI Operations	UI modernization	link	✅
AIP-68: Extended Plugin Interface	UI modernization	TBD	⌛
AIP-79: Remove Flask AppBuilder as Core dependency	UI modernization	link	✅
AIP-74: Introducing Data Assets	Data Assets, Event driven scheduling	link	✅
AIP-75: New Asset-Centric Syntax	Data Assets, Event driven scheduling	link	✅
AIP-82: External event driven scheduling in Airflow	Data Assets, Event driven scheduling	link	✅
AIP-83: Remove Execution Date Unique Constraint	Misc	link	✅
AIP-78: Scheduler-managed backfill	Misc	link	✅
AIP-81: Enhanced Security in CLI via Integration of API	Misc	link	⌛

Apart from the above big rocks, there are other candidates as listed here.

---

The following items need an owner:

• Improvements to Sensors. Mailing list discussion
• Rename the concept of user-facing "Active" DAG in UI & API #41519
• Consolidate "Serialization" code #40974
• Consider changing API to not use regexp pattern #32360
• Improve Airflow's debugging story #40975
• Docs: Persona-based quick start guides #40851
• Evaluate DB migration restriction when upgrading to AF 3 #41643
• Dependency to FAB is removed in Connection Forms #42016

... and open issues tagged with airflow3.0:candidate with no assignees. There are other great ideas for features that don't need an AIP for Airflow 3.1 in this doc, so if someone has time to do it for 3.0, please add a comment.

Timeline

	Date	Milestone
✅	9 August 2024	The main branch becomes Airflow 3 as soon as Airflow 2.10 is released
✅	Week of 27 Jan 2025	Alpha release 1
✅	Week of 3 Feb 2025	Alpha release 2
✅	Week of 10 Feb 2025	Alpha release 3
✅	Week of 24 Feb 2025	Beta release 1
✅	Week of 31 March 2025	Airflow 3 release candidates
✅	April 2025	Airflow 3 release

[jscheffl]

Why not adding a "Project" --> https://github.com/apache/airflow/projects ?

[JossWhittle]

Is there any planned follow up to AIP-48 to expose a Dataset api to custom providers and give a mechanism for polling for Dataset changes using deferrable triggerers?

Since AIP-48's responsibilities were shrunk so it could be merged there has not been any visible discussion about a follow up AIP or any progress towards the remainder of it's goals on the 3.x roadmap.

[kaxil]

Why not adding a "Project" --> https://github.com/apache/airflow/projects ?

Because we will have multiple "Projects"

[kaxil]

Is there any planned follow up to AIP-48 to expose a Dataset api to custom providers and give a mechanism for polling for Dataset changes using deferrable triggerers?

Since AIP-48's responsibilities were shrunk so it could be merged there has not been any visible discussion about a follow up AIP or any progress towards the remainder of it's goals on the 3.x roadmap.

Not yet, but Airflow 2.9 included the support for Dataset event updates which could act as a proxy for a "push-based" mechanism until we have a poll-based mechanism

[JossWhittle]

@kaxil I found the internal api call to create a DatasetEvent the other day but I'm much of a muchness over whether I want to abuse it to solve my problem.

airflow/airflow/datasets/manager.py

Line 64 in 35faaf8

def register_dataset_change(

airflow/tests/api_connexion/endpoints/test_dataset_endpoint.py

Line 665 in 35faaf8

"/api/v1/datasets/events", json=event_payload, environ_overrides={"REMOTE_USER": "test"}

I want to be able to have a custom Dataset class listening to a message queue. Currently this is achieved using a separate continuously scheduled DAG with a deferable operator that consumes messages and triggers a DAG run of the actual processing DAG.

This could be changed to create DatasetEvent embedding the message(s) from the queue into the extra field, and have the processing DAG schedule on that Dataset. Would this be inherently dangerous to do?

Using an external DAG for polling in either case at least gets all the fault tolerance and deferability of a DAG, and means status and history is shown in the UI.

A downside I am seeing though is that when my DAGs finish and I write to a message queue, this is entirely decoupled from being able to declare that outgoing queue as a Dataset outlet.

In fact, outlets can't really be used at all here because we want to trigger on the message being pulled from the queue by another polling DAG, not by the current DAG simply succeeding which won't have passed the message(s) into the extra field.

This means the graph of inter-dag dependencies is always broken up which is unfortunate.

Perhaps in the meantime Dataset could get a constructor argument to prevent triggering a DatasetEvent when used as an outlet. This would allow outlets to be used to mark up inter-dag dependencies.

---

I think in a world where there is a polling mechanism, Dataset outlets on succeeding tasks should only hint to Airflow that poll-ers should poll, but shouldn't create a DatasetEvent directly. Poll-ers should be deferable and fault tolerant, so an outlet Dataset firing really just means waking the poller immediately if it is deferred. Otherwise it will pick it up on it's own.

[gopidesupavan]

Hi @kaxil , Would like to take part of this Airflow 3 Journey , happy to contribute here. I can take look into this Improvements to Sensors.

[amoghrajesh]

@kaxil for items such as "Airflow Standalone Improvements" and "Improve Debugging Story" - I think we need more than just the heading because these are pretty open ended and I'd love to contribute to them and also "Remove deprecated code" :)

[gyli]

Hi @kaxil, I would love to take Consolidate "Serialization" code, could you please create an issue for this one?

[gopidesupavan]

Hi @kaxil

Over the past few days, I've thoroughly examined the core codebase and now have a strong grasp of the sensor component. I've also been contributing to Airflow for several months.

I would like to further explore the core areas. I feel like this is a great opportunity at this point.

Could you please throw some light on what the expectation on this Change sensors to use async by default?

In confluence it is mentioned. https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams
Sensor Improvements about sensor modes.

Remove poke/reschedule mode from sensors

But here in above it mentioned remove poke mode.

Is it removing both modes and keeping completely default always with async?

Appreciate your help 😄 ..

And not sure if anyone else is already working on this. am happy to contribute other areas also 😄