Allow user DAGs to emit DagWarning (or similar) #29275

Description

I would like a way to emit a warning or error message from a DAG that shows up as a banner in the web UI, similar to how exceptions are handled during DAG parsing or how DagWarnings are displayed.

Use case/motivation

I have a script under my dags_folder that is dynamically discovering and building DAGs from a networked filesystem. If one dynamically discovered DAG raises an exception, I don't want that to affect any of the other DAGs that are being discovered, so I swallow exceptions and log about them. However, I have not found a good way to communicate to my users that that exception occurred. Ideally I think it should look similar to when an exception is raised in a traditional DAG - an error banner on the web with (optionally) the full traceback.

In my testing, I've found that if any exception is raised during DAG parsing, it causes all DAGs originating from that file to be removed. So this sort of thing does not work:

#!/usr/bin/env python3
from datetime import datetime
import traceback

from airflow.decorators import dag, task


exceptions = []
for dag_id in 'ABCDE':
    try:
        @dag(
            dag_id=dag_id,
            schedule=None,
            start_date=datetime(2023, 1, 31),
        )
        def test_multi_exception():
            if dag_id in 'BE':
                raise RuntimeError(f'Failed to create dynamic DAG {dag_id}')

            @task
            def t():
                print('inside task')

            t()

        test_multi_exception()

    except Exception as e:
        exceptions.append(''.join(traceback.format_exception(etype=type(e), value=e, tb=e.__traceback__)))

if exceptions:
    raise RuntimeError('\n'.join(exceptions))

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct