Repository navigation
DAG triggering without all Datasets being triggered #41101
Description
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.9.2
What happened?
We have an airflow environment in AWS (MWAA). We have multiple DAGs that are triggered based on a number of datasets. In 2.8.1 everything was working fine. After upgrading to 2.9.2 some of the DAGs are triggering without all the datasets updates being met. Sometimes they work and other times they don't. If there is logging (or DEBUG logging) that can be enable to troubleshoot this more please provide links to information and I'll try and provide more information.
What you think should happen instead?
DAGs should only be triggered when all the Datasets have updates since the last time the DAG ran.
How to reproduce
There isn't anything we can do to trigger this issue. We have 1100 DAGs, 600+ of which are triggered by Datasets. Of these we have 40-50 that fail nightly. They aren't the same DAGs or Datasets that are having issues. As mentioned above if there are logs or debugging that can be collected to provide better information or reproducibility I'd be glad to enable and collect the needed information.
Operating System
AWS MWAA 2.9.2 (latest)
Versions of Apache Airflow Providers
No response
Deployment
Amazon (AWS) MWAA
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jul 29, 2024
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
slords commentedon Jul 29, 2024
potiuk commentedon Jul 30, 2024
I think you should realise that we have absolutely no idea about your setup and screenshots are not helping to understand that, so people here can't even know what is wrong. People here will simply miss context.
The only way it can be acted upon is if there are correlated logs - i.e. find a concrete DAGs that has been triggered wrongly. You can set debug log level on the whole airflow (but I have no idea how to do in MWAA - likely there should be docs) https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level and you can get scheduler logs (again no idea how to do it in MWAA) - correlated with the DAG that gets triggered wrongly - showing why it has been scheduled and explaining the context. Explaining what has been triggered, why you think it should not be and showing related log entries.
You have to remember - that here people here when they think they can help - in their free time - and it's up to you to provide detailed information (that they don't have), so such a vague issue does not help those people (who often do it at expense of their free time) does not help them - simply this is not a helpdesk but a forum where people help each other if they can.
But also remember that MWAA is a service you pay for and that they can likely access all the information that people here can't - so if you won't be able to extract and provide the information with more details, then you pay for the service and I guess they should be able to investigate more - with all the access and possibilities to look at your problem.
slords commentedon Aug 1, 2024
The issue is that DAGs are triggering even though not all of the datasets have been updated. Sometimes the DAGs trigger successfully, but other times they don't. This makes it really hard to figure out what's wrong.
To fix this, we need to find the logs that show why the DAG was scheduled even though not all the datasets had updates. MWAA doesn't let you set the debug flag on any log, only INFO. I'm working with AWS to see if there is an internal way to turn on DEBUG so we can get to these logs.
You don't need to know about my setup to see what the problem is. I've attached some screenshots that clearly show which datasets are required and that, on two of the five occasions that the DAG ran, it didn't have all the required datasets available.
I understand how open source works and I know that most people won't raise bugs. But, I've been contributing to open source projects for years, and I firmly believe that any issue I can help fix makes the product better for everyone. Filing a bug report is a good way to get an issue identified and hopefully resolved. It's also possible that others are experiencing the same problem but haven't said anything yet. If they see this bug report, they can confirm the issue and maybe even provide some additional information that will help us find a solution.
I was disappointed by the unhelpful and dismissive attitude that came across in your response. Not everyone has the same level of technical expertise or resources, and it can be frustrating to be made to feel like a nuisance. Just because I don't have all the answers doesn't mean my bug report isn't valid. I'm providing as much information and context as I can, and I'm happy to cooperate with anyone who wants to help troubleshoot the issue.
github-actions commentedon Aug 16, 2024
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
github-actions commentedon Aug 26, 2024
This issue has been closed because it has not received response from the issue author.
Fix dataset-triggered DAGs firing without all upstream events (backpo…
Fix dataset-triggered DAGs firing without all upstream events (backpo…