Skip to content

Required Param requires None to be explicitly passed as default #28940

Closed
Closed
Required Param requires None to be explicitly passed as default#28940
Assignees
amoghrajesh
Labels
_eol_affected_version:2.5Issues Reported for 2.5area:corekind:bugThis is a clearly a bug

Description

@aamster
aamster
opened on Jan 14, 2023

Apache Airflow version

2.5.0

What happened

I have a dag like

import datetime

from airflow.models import Param
from airflow.models.dag import dag


@dag(
    dag_id="foo",
    schedule=None,
    params={'month': Param()},
    start_date=datetime.datetime.now()
)
def foo():
    pass


foo()

I am trying to pass this param via CLI. I tried passing it like

airflow dags test foo --conf '{"month": "2022-01"}'

gives

Traceback (most recent call last):
  File "/Users/adam.amster/venvs/airflow/lib/python3.10/site-packages/airflow/models/dagbag.py", line 437, in _process_modules
    dag.validate()
  File "/Users/adam.amster/venvs/airflow/lib/python3.10/site-packages/airflow/models/dag.py", line 667, in validate
    self.params.validate()
  File "/Users/adam.amster/venvs/airflow/lib/python3.10/site-packages/airflow/models/param.py", line 230, in validate
    raise ParamValidationError(f"Invalid input for param {k}: {ve}") from None
airflow.exceptions.ParamValidationError: Invalid input for param month: No value passed and Param has no default value

Note

Found out that you need to explicitly pass None as the default, otherwise it will crash with above.

What you think should happen instead

month should be passed to the dag

How to reproduce

No response

Operating System

mac osx

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. aamster
    added
    area:core
    kind:bugThis is a clearly a bug
    on Jan 14, 2023

  2. boring-cyborg commented on Jan 14, 2023

    @boring-cyborg
    boring-cyborgbot
    on Jan 14, 2023

    Thanks for opening your first issue here! Be sure to follow the issue template!

  3. aamster
    changed the title [-]Param does not work from command line[/-] [+]Param requires `None` to be explicitly passed from command line if it is a required argument[/+] on Jan 14, 2023
  4. aamster
    changed the title [-]Param requires `None` to be explicitly passed from command line if it is a required argument[/-] [+]Param requires `None` to be explicitly passed as default[/+] on Jan 14, 2023
  5. aamster
    changed the title [-]Param requires `None` to be explicitly passed as default[/-] [+]Required `Param` requires `None` to be explicitly passed as default[/+] on Jan 14, 2023
  6. potiuk
    added
    good first issue
    on Jan 15, 2023

  7. potiuk commented on Jan 15, 2023

    @potiuk
    potiuk
    on Jan 15, 2023
    Member

    Marked it as good first issue. Hopefully somoene will fix it soon (but it's up for grabs for you if you would like to fix it ). PRs are most welcome to fix it.

  8. amoghrajesh commented on Jan 16, 2023

    @amoghrajesh
    amoghrajesh
    on Jan 16, 2023
    Contributor

    @potiuk @aamster I can take this up if it is not already taken. Would be more than happy to have this fixed.

  9. uranusjr
    assigned
    amoghrajesh
    on Jan 16, 2023

  10. Taragolis commented on Feb 19, 2023

    @Taragolis
    Taragolis
    on Feb 19, 2023
    Contributor

    Found out that you need to explicitly pass None as the default, otherwise it will crash with above.

    For me it make sense. If no default value provided than parameter is a mandatory, and otherwise if default parameter provided than parameter is optional

  11. hussein-awala commented on Feb 19, 2023

    @hussein-awala
    hussein-awala
    on Feb 19, 2023
    Member

    I agree with @Taragolis, we should check if the required parameters (with no default value) are provided or not, and fail the dag run if they are not provided.
    I'm working on a #29174 to deprecate dag run conf and support providing the params directly, and this exception will be raised before creating the dag run (code)

  12. hussein-awala commented on Feb 19, 2023

    @hussein-awala
    hussein-awala
    on Feb 19, 2023
    Member

    It seems like currently we don't support required params, to fix this, I think we need to differentiate between parsing the params during the dag parsing/creation and parsing the params during the dag run execution.

    During the dag parsing, we should not raise an exception if the default is not provided because this param is defined as required param. But during the dag run execution, if the param is required and the value is not provided, we should raise an exception to fail the dag run or the task if the params are defined at the task level.

    I don't think this is a good first issue.

  13. potiuk
    removed
    good first issue
    on Feb 19, 2023

  14. potiuk commented on Feb 19, 2023

    @potiuk
    potiuk
    on Feb 19, 2023
    Member

    Sure. Removed the label then

  15. eladkal
    added
    _eol_affected_version:2.5Issues Reported for 2.5
    on Feb 28, 2023

  16. aamster commented on Jun 20, 2023

    @aamster
    aamster
    on Jun 20, 2023
    Author

    This is even worse for the case of a Param with type of i.e. int. Then you need to pass a default value like -999.

  17. jscheffl commented on Mar 31, 2024

    @jscheffl
    jscheffl
    on Mar 31, 2024
    Contributor

    There had been a couple of PRs by me, for example #31301 and #34248 which handle optional fields and how Params are handled for this. Just got note of this bug report and must state after a manual re-test: Can not be re-produced. Did not explicitly test but might have been resolved in 2.8.0 already. Therefore closing.

  18. jscheffl
    closed this as completedon Mar 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

_eol_affected_version:2.5Issues Reported for 2.5area:corekind:bugThis is a clearly a bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions