Skip to content

Use type hints to indicate multiple return values when using @task #8996

Closed
#10349
Closed
Use type hints to indicate multiple return values when using @task#8996
#10349
Assignees
AndersonReyes
Labels
AIP-31Task Flow API for nicer DAG definitionarea:dev-toolskind:featureFeature Requests
Milestone
Airflow 2.1

Description

@turbaszek
turbaszek
opened on May 24, 2020

Description

Use type hints to indicate multiple return values of @task decorated function to improve dereferencing.

Using type hints to enhance code behaviour is for example done in:
https://databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release-of-apache-spark-3-0.html

Use case / motivation

Currently:

@task(multiple_outputs=True)
def my_function():
    return 1, 2, 3

expected:

@task
def my_function() -> Tuple[int, int, int]:
    return 1, 2, 3

Related Issues
https://github.com/apache/airflow/pull/8962/files#r429341828
https://github.com/apache/airflow/pull/8962/files#r428948073

Activity

  1. turbaszek
    added
    kind:featureFeature Requests
    AIP-31Task Flow API for nicer DAG definition
    on May 24, 2020

  2. AndersonReyes commented on Aug 15, 2020

    @AndersonReyes
    AndersonReyes
    on Aug 15, 2020
    Contributor

    @turbaszek is this still up for grabs? I want to give it a shot as first contribution

  3. AndersonReyes commented on Aug 15, 2020

    @AndersonReyes
    AndersonReyes
    on Aug 15, 2020
    Contributor

    after some googling looks like this can be achieved like this

    ....
    if python_callable:
        sig = signature(python_callable).return_annotation
        if get_origin(sig) in (dict, tuple):
            multiple_outputs = True
....
  4. kaxil
    assigned
    AndersonReyes
    on Aug 24, 2020
  5. kaxil
    added this to the Airflow 2.0.0 milestone on Aug 24, 2020
  6. vikramkoka
    added
    area:dev-tools
    on Nov 3, 2020
  7. vikramkoka
    modified the milestones: Airflow 2.0.0, Airflow 2.0.0-beta1 on Nov 3, 2020

  8. vikramkoka commented on Nov 3, 2020

    @vikramkoka
    vikramkoka
    on Nov 3, 2020
    Contributor

    @kaxil , @turbaszek Should this be in beta1 from the perspective of feature complete?

  9. turbaszek commented on Nov 4, 2020

    @turbaszek
    turbaszek
    on Nov 4, 2020
    MemberAuthor

    @kaxil , @turbaszek Should this be in beta1 from the perspective of feature complete?

    @vikramkoka I would say this adds only a syntax sugar not a "real feature"

  10. kaxil
    modified the milestones: Airflow 2.0.0-beta1, Airflow 2.0.0-beta2 on Nov 4, 2020
  11. ashb
    modified the milestones: Airflow 2.0.0-beta3, Airflow 2.0.0-beta4 on Nov 18, 2020
  12. potiuk
    modified the milestones: Airflow 2.0.0-beta4, Airflow 2.0.0 (rc1) on Nov 30, 2020

  13. potiuk commented on Dec 7, 2020

    @potiuk
    potiuk
    on Dec 7, 2020
    Member

    Hey @turbaszek -> is ti somethig we want to add fo 2.0.0rc1 ? Or should we postpone to 2.1 ? If so - can you change the milestone please?

  14. ashb
    modified the milestones: Airflow 2.0.0rc1, Airflow 2.1 on Dec 7, 2020
  15. ashb
    closed this as completedin #10349on Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

AIP-31Task Flow API for nicer DAG definitionarea:dev-toolskind:featureFeature Requests

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions