Repository navigation
JSON notation in Airflow DAG comments causing KeyError #9753
Description
-->
Apache Airflow version: 1.10.11 (code is working fine in 1.10.10)
Environment: Python 3.7
- Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release): Catalina 10.15.5
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened:
KeyError: '"randomKey"'
What you expected to happen:
Expected the block comment on the top of the DAG to be copied as is into the DAG description box in the UI
How to reproduce it:
Add the following code to the top of your DAG
"""
This is a test dag. this is the description of it. This is how you can trigger it with CLI.
airflow trigger_dag --conf '{"randomKey":30}' airflow-test-dag
--conf options:
randomKey: - Optional
"""
Anything else we need to know: Works fine with Airflow 1.10.10 and Python 3.7.
** UI Logs: **
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 380, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 286, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 1749, in graph
doc_md = wrapped_markdown(getattr(dag, 'doc_md', None), css_class='dag-doc')
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 258, in wrapped_markdown
).format(css_class=css_class)
File "/usr/local/lib/python3.7/site-packages/markupsafe/__init__.py", line 213, in format
return self.__class__(formatter.vformat(self, args, kwargs))
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 190, in vformat
result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 230, in _vformat
obj, arg_used = self.get_field(field_name, args, kwargs)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 295, in get_field
obj = self.get_value(first, args, kwargs)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/string.py", line 252, in get_value
return kwargs[key]
File "/usr/local/lib/python3.7/site-packages/markupsafe/__init__.py", line 249, in __getitem__
return self._kwargs[key]
KeyError: '"randomKey"'
Activity
turbaszek commentedon Jul 10, 2020
@prakshalj0512 what version of flask do you use?
prakshalj0512 commentedon Jul 10, 2020
kaxil commentedon Jul 10, 2020
Can you show us your DAG please?
prakshalj0512 commentedon Jul 10, 2020
https://github.com/teamclairvoyant/airflow-maintenance-dags/blob/master/db-cleanup/airflow-db-cleanup.py
Reference Lines 176-177
kaxil commentedon Jul 10, 2020
I was able to reproduce it but looks like it can be easily fixed if you remove the following from docstrings from the DAG (https://github.com/teamclairvoyant/airflow-maintenance-dags/blob/63ddbfb19d53fa7b44343dc0525cf1b60f16fed6/db-cleanup/airflow-db-cleanup.py#L6-L9):
prakshalj0512 commentedon Jul 10, 2020
Yes, I narrowed it down to the curly braces that were causing the issue. Not sure what's causing them to be interpreted as part of JSON as opposed to the comment.
potiuk commentedon Jul 12, 2020
Isn't that the issue with DAG serialization of the comment field? It looks awfully like it.
kaxil commentedon Jul 12, 2020
No, I could reproduce it without DAG Serialization.
potiuk commentedon Jul 12, 2020
Right I know what it is and the problem is also in master. Fix is coming.