Repository navigation
Some DAGS stuck in schedule state after migrating from 2.10.5 to 3.0.0 and also DAG code is not visible for them #49618
Description
Apache Airflow version
3.0.0
If "Other Airflow 2 version" selected, which one?
No response
What happened?
Some DAGS stuck in the schedule state after migrating from 2.10.5 to 3.0.0 and also DAG code is not visible for them
What you think should happen instead?
No response
How to reproduce
- ADD below DAG in AF2
- Migrate from AF2 to AF3
- Try to check soruce_code
DAG
from airflow.models import DAG
from airflow.operators.datetime import BranchDateTimeOperator
from airflow.operators.dummy import DummyOperator
from airflow.operators.python import PythonOperator
from airflow.utils.trigger_rule import TriggerRule
from airflow.utils.dates import days_ago
from airflow_dag_introspection import assert_the_task_states
from datetime import datetime, time
import time as Time
docs = """
####Purpose
This dag tests the BranchDateTimeOperator to ensure that tasks within the timeframe set run and tasks outside of that timeframe are skipped.\n
It acheives this test by setting the start date of the tasks that are skipped to start one day earlier than set in the dag parameter 'start_date' so that the skipped tasks are always outside of the time range.\n
####Expected Behavior
This dag has 6 tasks 4 of the tasks are expected to succeed while 2 of the tasks are expected to be skipped.\n
This dag should pass.
"""
with DAG(
dag_id="branch_date_time_operator",
start_date=days_ago(2),
schedule_interval=None,
concurrency=1,
doc_md=docs,
tags=['core']
) as dag:
d0 = DummyOperator(task_id="dummy0", start_date=days_ago(3))
d1 = DummyOperator(task_id="dummy1", start_date=days_ago(3))
d2 = DummyOperator(task_id="dummy2")
d3 = DummyOperator(task_id="dummy3")
bdto = BranchDateTimeOperator(
task_id="branch_datetime",
follow_task_ids_if_true=["dummy0", "dummy1"],
follow_task_ids_if_false=["dummy2", "dummy3"],
# target_upper and target_lower are time ranges for tasks to be in
target_upper=time(hour=2, minute=33),
target_lower=time(hour=0, minute=0, second=22),
)
py0 = PythonOperator(
task_id="assert_the_states",
python_callable=assert_the_task_states,
op_args=[{
"branch_datetime": "success",
"dummy0": "skipped",
"dummy1": "skipped",
"dummy2": "success",
"dummy3": "success",
}],
trigger_rule=TriggerRule.ALL_DONE,
)
bdto >> [d0, d1, d2, d3] >> py0
Operating System
Linux
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
[-]DAG code not showing for some DAG after migrating from 2.10.5 to 3.0.0[/-][+]Some DAGS stuck in schedule state after migrating from 2.10.5 to 3.0.0 and also DAG code is not visible for them[/+]vatsrahul1001 commentedon Apr 23, 2025
Ok, so I debugged this. I noticed that the above DAG has an import error, however, it's still showing in DAG list page and maybe due to this, all this is happening
pandinug commentedon Apr 23, 2025
The
schedule_intervalis deprecated as of version 3 (as per release notes), causing the DAG import error. Because of this the DAG cannot be parsed and the source code remains empty.The short fix would be to replace
schedule_intervalwith justscheduleand have the DAG be parsed again. The code will now be visible and the DAG can be ran.vatsrahul1001 commentedon Apr 23, 2025
@pandinug we have alot of issue with this DAG one is schedule_interval other is
dummy_operatorwe need to update that toEmptyOperator.Real issue here is we should mark these DAG as stale in DB and after which they won't show up in DAG list page
tirkarthi commentedon Apr 23, 2025
IIRC, Airflow 2 used to show the import error in the respective dag page too at the top.
phanikumv commentedon Apr 23, 2025
Cc @sunank200
vatsrahul1001 commentedon Apr 23, 2025
@tirkarthi yes, however we used to get that error if we are already on respective dag page. On DAG listing page we were not showing DAGs which had an import error.
tirkarthi commentedon Apr 23, 2025
Yes, I don't have an Airflow 2 setup to migrate and test. Is
has_import_errorsin the dag table set to true for this dag? Also now clicking on the task instance and dependencies blocking execution are shown in the details tab for task instance so wondering if there is any detail there that could have made this easier to debug though I agree that the dag itself shouldn't have appeared in the list page.15 remaining items
Mark DAGModel stale and associate bundle on import errors
Mark DAGModel stale and associate bundle on import errors
Mark DAGModel stale and associate bundle on import errors
fix cli clear command (#49631)
Mark DAGModel stale and associate bundle on import errors
fix cli clear command (apache#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)
fix cli clear command (#49631)