Repository navigation
Improve Airflow's debugging story #40975
Description
Activity
Let me add to it what I wrote about OTEL in the https://lists.apache.org/thread/b2bvn8sbxfncg9qpvry9w142944mnlj6 - this might be a great tool to hlep with things. I a not sure if I want to take lone ownership about that one - maybe there will be someone else who would like to take a look and explore things as well - but I am happy to be deeply involved in that one.
Reacted by Dmitry Pustoshilov and Amogh DesaiI'd like to be involved in this effort in some capacity. At least: brainstorming, qa, and documentation.
Happy to help out with some of the logging and error handling implementation.
The debug snapshot idea sounds very useful @potiuk. It may give a canonical view of the user's environment. I suppose Jaeger provides a similar tool called Anonymizer, which generates a shareable json of a trace - probably same one that you were referring to in your mail. We can build our own debug snapshot util, or can think of using this tool with Jaeger since it supports the existing OTEL metrics and traces.
@kaxil Honestly? It sounds a bit scary going from contributing minor patches to being responsible for an important feature in an upcoming release. I prefer to actively observe and learn, at least once, how something like this is done and take on a similar responsibility after I know how much time/work it requires.
Absolutely, that's completely fine
@kaxil Honestly? It sounds a bit scary going from contributing minor patches to being responsible for an important feature in an upcoming release. I prefer to actively observe and learn, at least once, how something like this is done and take on a similar responsibility after I know how much time/work it requires.
@omkar-foss Do you want to take a stab at leading it?
@omkar-foss Do you want to take a stab at leading it?
@kaxil I would love to take the lead on this, but right now I suppose I'm still a rookie in the ways of the Airflow community.
So for this one, I'll prefer to assist all of you in every way possible, while trying to get a better grasp of the processes, codebase etc. Hope that's okay, thanks for considering me though 😇
Whoever takes on this task should conduct a user research on the mailing list, Slack, Meetup or Airflow Summit to identify other common debugging problems that can be fixed.
@kaxil Any idea if there's a predefined user research template that has been used for prior releases?
If not, I'd like to propose the following for conducting the survey:
- We can create a survey form with questions pertaining to understanding the users' debugging journey. Probably can use something like SurveyMonkey.
- We can have groups of questions in the form, each group for a section in "Current Challenges" above. For example, the groups could be Logging, Traceback, Error Handling, Tooling & Integrations.
- We can have 3 to 5 questions for each group. Let's try to keep the survey as brief and concise as possible.
- The survey form can be circulated on the mailing list, Slack, and other places.
- We can collate the feedback from the survey form and prioritize items accordingly.
Please let me know your thoughts on this, thanks.
@omkar-foss The main question is who the target audience of the research is, where possible answers are: maintainers, contributors, power users, general public, etc. Based on @kaxil's instructions, I'd say mostly power-users and above. If that is the case, I'm assuming most will be willing to participate in a survey, even if it has questions on topics people might not have an opinion on. If on the other hand, we're looking to get more participants, I think a literal survey is not the way, since people might open it, see how long it is, and just give up. That, of course, would be a terrible waste, because there are likely many use-cases that will not be represented.
For the above reason, I was thinking something like a feature voting platform (example1, example2) could be suitable - that way, if someone has a pain-point related to how a particular system works, they can look for existing posts or briefly explain what they have in mind (possibly with a template like a bug report) and allow others to vote or add to these suggestions. This also takes care of much of the aggregation work of the results.
Hey @Dev-iL, I agree with your reasoning above. I checked out the sample Feature Upvote board that you've shared above and it surely feels simpler (and quicker) to submit compared to a regular survey form.
I suppose we'll need an initial list of features on the upvote board for the participants to vote, would be great to hear if you've any thoughts around it.
Not sure how much help I can be on this, but I'm here so feel free to tag me if you need any assistance! :)
@omkar-foss The main question is who the target audience of the research is, where possible answers are: maintainers, contributors, power users, general public, etc. Based on @kaxil's instructions, I'd say mostly power-users and above.
I'd say mostly power-users - yes, but also the tooling and debuggability should be targeted for "new" users. I think power-users mostly know their ways - they can do remote debugging, they know how to connect their IDEs to the code, they are able to even use pdb, py-spy and other tools while remote shelling to container instances etc.
But the goal here is to shorten the path between "I wrote some DAG and it does not work" to "how do I most effectively find inspect and understand what's going on there" - for a user who just wrote their first few dags.
I think an assumption should be that that person has some Python experience, they have an IDE (PyCharm/ VSCode) and they are willing to follow some instructions on setting up things first - while ideally this should be one-time setup and they should be able to re-use it easily (and teach others how to do it).
If that is the case, I'm assuming most will be willing to participate in a survey, even if it has questions on topics people might not have an opinion on. If on the other hand, we're looking to get more participants, I think a literal survey is not the way, since people might open it, see how long it is, and just give up. That, of course, would be a terrible waste, because there are likely many use-cases that will not be represented.
I think yes - survey is a good idea if well prepared and those power-users might indeed be willing to share their experiences - we can even leverage the upcoming Airlfow summit and do some prices / recognition and generally a bit more fuss about it - so if we could do it still in August and maybe run the survey during the Summit as well, we could likely make it much more efficient.
@potiuk @omkar-foss In the interest of moving ahead with this, I've made a google doc so we can start hashing out this survey collaboratively. Currently, it's publicly open for commenting - please send me your google account via slack so I could add you to the editors. If there are any privacy or other concerns, I don't mind moving the document to another platform.
Reacted by Omkar P15 remaining items
- added a commit that references this issue
on Oct 18, 2024 Hi all! As per discussion, we'll be tracking all issues related to Airflow Debugging Story (based on debugging survey responses) on this project: Debugging Improvements - Airflow 3
Reacted by Amogh Desai and Dev-iL- added a commit that references this issue
on Oct 23, 2024 - added a commit that references this issue
on Oct 24, 2024 Also see #40802 (comment) discussion. I believe with OTEL and traces (and even including limited set of logs in the traces) we are closer to address big gap in debugging of Airflow where we can give our users a tool to provide us way more diagnostics information that will allow us to analyse, diagnose, and fix many problems much more efficiently.
Reacted by Omkar P- added a commit that references this issue
on Nov 13, 2024 - added a commit that references this issue
on May 27, 2025 - added a commit that references this issue
on Sep 22, 2025 - added a commit that references this issue
on Oct 20, 2025 - added a commit that references this issue
on Feb 26, 2026 - added 2 commits that reference this issue
on Jul 27, 2026 - added a commit that references this issue
on Sep 3, 2026
Summary
As we prepare for the release of Airflow 3.0, one of the key areas that need significant enhancement is the debugging experience.
Current Challenges
dag.testandtask.testdoes a good job already but we should see if we can do even better.airflow dags parsedoes a job at it, worth checking if it is sufficient or not.Whoever takes on this task should conduct a user research on the mailing list, Slack, Meetup or Airflow Summit to identify other common debugging problems that can be fixed.