Emit OL `FAIL` event when manually marking a DAG as failed

[wslulciuc]

Apache Airflow version

2.9.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Manually marking a DAG as failed doesn't not emit an OL FAIL event.

What you think should happen instead?

An OL FAIL event would be emitted when manually marking a DAG as failed.

How to reproduce

To reproduce, start a DAG, then manually mark the DAG as failed. Then, view the DAG logs, no OL logs are observed.

Operating System

N/A

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[mobuchowski]

I think it makes a lot of sense. However I'm not sure if we should do that now - we probably won't get that change into 2.10, and in 3.0 we might have single place responsible for that due to AIP-72.

[mobuchowski]

@wslulciuc Looked at this some time ago.

Natural approach to do this would be to add listener calls to API endpoints (or, somewhere deeper that eventually modifies database) and call it a day. However:

• API operates on different objects than current listener - it operates directly on DB (SQLAlchemy) objects instead of creating for example DagRun etc objects
• There are more than single API calls to do that one thing. It seems one is for UI, second is for regular API.
• There is unclear what we should do with "followup tasks". Airflow can react to completed or failed DAG or Task with scheduling other things, or failing them. Should we manually issue "empty" events for them?
• When we complete or fail task manually, we don't actually get the same information that we got from inside worker.
• The whole mechanism will completely change for Airflow 3 anyway due to AIP-72.

I think we should work on this as part of AIP-72. I don't think there's good workaround for that unfortunately.

[mobuchowski]

This will work in Airflow 3, with support in the new API merged #45652

Support for TI events will follow when it will be migrated to the new API.