Repository navigation
Parameters not rendered correctly in 2.7.0 UI #33923
Description
Apache Airflow version
2.7.0
What happened
When initialising integer parameters as 0, or arrays as empty list, the default behaviour of the UI is to render it as null.
This results in default parameters being ignored unless re-imputed manually.
Passing a config like this:
params={
"integer_param": Param(0, type="integer", minimum=0),
"list_param": Param([], type="array"),
},
results in this in the UI when the DAG loads, forcing the user to manually input again 0, or editing the json for having an empty list
What you think should happen instead
0 rendered as 0, empty lists as empty lists
How to reproduce
This DAG should be an MVP for reproducing the error
from datetime import datetime
from airflow import DAG
from airflow.models.param import Param
from airflow.operators.python_operator import PythonOperator
default_args = {
"owner": "data-enablement",
"start_date": datetime(2023, 4, 24),
"retries": 30,
"startup_timeout_seconds": 600,
}
with DAG(
"test-error",
default_args=default_args,
schedule=None,
params={
"integer_param": Param(0, type="integer", minimum=0),
"list_param": Param([], type="array"),
},
render_template_as_native_obj=True,
description="MVP for reproducing error",
) as dag:
print_int_task = (
PythonOperator(
task_id="print_int",
op_args=[
"{{ params.integer_param}}",
],
python_callable=(lambda integer_param: print(integer_param)),
),
)
print_list_task = PythonOperator(
task_id="print_list",
op_args=[
"{{ params.list_param}}",
],
python_callable=(lambda list_param: print(list_param)),
)
print_int_task, print_list_task
Operating System
macOS M1 locally
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
Using this image:
https://hub.docker.com/layers/apache/airflow/2.7.0-python3.10/images/sha256-d6cb37572c21546ba3e60274d3cc1e782cc517a31443b8051702b19584e7379b?context=explore
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Aug 30, 2023
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
SamWheating commentedon Aug 30, 2023
I can look into this, feel free to assign to me.
josh-fell commentedon Aug 31, 2023
All yours @SamWheating!
jscheffl commentedon Aug 31, 2023
Empty lists being rendered as
null(also the same for empty strings and objects) was actually a desire. This is the (only?) way to enforce users to enter values into lists for the required input check. If we would change thenullthing we need to think about required filed validation as well and make it consistent across all types.If you want to have lists optional then you need to use
type=["null", "array"].I agree the integer thing is actually a bug. With
0as default value a0should be accepted.I'd also be OK to contribute a fix if a desire is having this on short term.
Update: I also just realized the "null"/
Nonefield behavior is documented in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.htmlmatteoannotell commentedon Aug 31, 2023
But then if one defaults the value of a parameters to an empty list, and build everything that uses it around the premises of receiving an iterable, things will break when receiving
None, which is not iterable?jscheffl commentedon Aug 31, 2023
Question is what "everything" means in your case. In Jinja you can
{{ params.mylist | default([]) }}and inside Python code it would befor item in params.mylist or []: ...if you require to loop w/o checking the content. Main purpose of typing validation is that you can trust submitted params before handling it in the code. If you field is not required you anyway need to take care to handle "missing data" for example if called via CLI or REST API.On the other hand if an empty list would be passed as
[]to the DAG, validation for required fields would be harder. Then you need to add aminItemsschema definition if you want to enforece that users add at least one item.. Makes the form validation much more complex, standard jQuery mechanisms for required fields are not working easily in this example.I'd prefer to keep it consistent that "missing data" is passed as
null/Noneas it is stated in the docs currently. Otherwise we would need to define the empty handling different per type and document this.matteoannotell commentedon Aug 31, 2023
I see your point, but
for item in params.mylist or []feels like a double control if the param is already specified to be an array inParam([], type="array")?Besides that, there is also the minor annoyance of having to manually providing a parameter to the UI, otherwise the DAG cannot be triggered.
Neither of my points are true dealbreakers and I agree that there are simple workarounds, but it feels like something that should/could (?) perhaps work out of the box
SamWheating commentedon Aug 31, 2023
I agree that there should be a way to pass either a null or an empty list through the form interface without any workarounds, but I think it will require some more changes (maybe we'll have to include the
[ ]in the form field?).I'll draft up some possible implementations and we can chat on the PR.
jscheffl commentedon Aug 31, 2023
If the field is a list and is required anyway then I would expect that also at least one element needs to be returned - an empty list for me does not represent a valid value if defined as
Param([], type="array"). Or is your use case expecting that the user can pass no element in the array as valid option?The main difference we have is how to define a "non required or empty array" as result and technically all is possible. But the form can not distinguish whether
[]orNoneis expected.We have the same matter also with all other non-bool types (string, int, object), an empty string field also is passed as
None. I still opt for consistency.SamWheating commentedon Aug 31, 2023
I think that for any of these there is a clear difference between "empty" values (
"",0,{},[]) and null values, and we should probably allow users to provide either the empty or null value as required. So while I agree that we should have consistency, I think I disagree on what that consistent interface should look like.I also acknowledge that this definitely adds some complexity implementation-side and probably won't make any difference for 99% of use cases.
Yes, I think that both
nulland[]are valid but distinctly different.matteoannotell commentedon Aug 31, 2023
Yes, empty list would be a valid option in my DAG (with all discussed so far still holding, I could just rewrite the implementation so that null is allowed and the rest of the DAG is adjusted accordingly)
jscheffl commentedon Sep 1, 2023
I'd propose to close this ticket and separate-out the empty list thing into a second discussion/feature ticket. This will not make it (including discussion) into 2.7.1.
The main bug of missing rendering a
0number value was fixed in #33965