Repository navigation
DAGs remain in the UI after renaming the dag_id in the same python file #11901
Description
Apache Airflow version: 1.10.12
What happened:
When I rename the dag_id, the new dag_id shows up in the UI but the old dag_id does not disappear and remains in the UI.
What you expected to happen:
When Airflow lists out the python files and tries to deactivate the deleted dags during dag processing, the old DAG's file location is still in the list of the alive DAG location because the new DAG is now defined in the old DAG's python file. Even when you manually set is_active to False in the metastore, the dag processing process will set it back to True.
This happens whether or not DAG serialization is enabled.
How to reproduce it:
I was able to reproduce it by following these steps:
- Add a python file in the dags folder and initialize the DAG
- Verify the dag shows up in the UI
- Change the dag_id in the same file to something else
- Verify the new dag shows up in the UI
- Both the old DAG and the new DAG are visible on the UI as well as the metadata db.
➜ dags pwd
/Users/alan/projects/astro/dags
➜ dags ls
wow.py
from datetime import datetime
from airflow.models import DAG
dag = DAG(
dag_id='yay',
schedule_interval='@once',
start_date=datetime(2020, 1, 1),
catchup=False
)In the screenshot, both DAGs are marked as active but there only one DAG defined in wow.py.
Anything else we need to know:
Activity
kaxil commentedon Nov 20, 2020
This needs fixing in Master and is not that critical, so will fix it in 2.0.0
potiuk commentedon Dec 7, 2020
Hey @kaxil @ashb - are we going to fix it ? Or maybe it already is fixed with the serialization ?
mattforbes-gr commentedon Feb 9, 2021
I believe this is still an issue with serialized dags. We are dynamically generating dags and had to resort to randomizing our dag py file name to force stale serialized dags to be deleted.
ashb commentedon Feb 10, 2021
No, still remains to be fixed.
leonsmith commentedon Mar 1, 2021
Started to debug this a little.
I’m thinking the problem is the wrong assumption that you can set
is_active=Trueon any dags with the same file path.So the fix should be patching out the else statement here:
https://github.com/apache/airflow/blob/1.10.12/airflow/models/dag.py#L1917
I think
is_activeis set toTruein theDagBag.sync_to_dbfunc when parsing the dags.Going to run a bit of a test & will update to see if that resolves the issue
HaloKo4 commentedon Mar 1, 2021
if the expected behavior is to remove/rename the old dag_id how are we expected to read its past logs?
This will require to also modify past log files in the disk otherwise they will be unaccessable.
kaxil commentedon Mar 1, 2021
For now, you will have to just use a different file for the new (or renamed) DAG.
25 remaining items
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)
Deactivating DAGs which have been removed from files (#17121)