Skip to content

Fix region argument in MappedOperator based on AwsBaseOperator / AwsBaseSensor - #38178

Merged
Taragolis merged 5 commits into
apache:mainfrom
Taragolis:aws-base-on-partial
Mar 18, 2024
Merged

Taragolis merged 5 commits into
apache:mainfrom
Taragolis:aws-base-on-partial

Conversation

@Taragolis

Copy link
Copy Markdown
Contributor

Originally reported in Slack

Hi everyone! :nyanparrot:
I just upgraded the airflow packages versions:

apache-airflow==2.8.3
apache-airflow-providers-amazon==8.19.0

And now I have a broken DAG:

Broken DAG: [/opt/airflow/dags/distributor_catalogue.py]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 265, in partial
    validate_mapping_kwargs(operator_class, "partial", kwargs)
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/mappedoperator.py", line 117, in validate_mapping_kwargs
    raise TypeError(f"{op.__name__}.{func}() got {error}")
TypeError: EcsRunTaskOperator.partial() got an unexpected keyword argument 'region'

The use of region is this one:

EcsRunTaskOperator.partial(
    task_id="task_id",
    aws_conn_id="aws_manual_test",
    cluster="tf_ecs_airflow",
    task_definition="tf_ecs_airflow",
    launch_type="FARGATE",
    region="eu-west-3",
    network_configuration={
        "awsvpcConfiguration": {
            "securityGroups": [Variable.get("ecs_securitygroup")],
            "subnets": [Variable.get("ecs_subnet1"), Variable.get("ecs_subnet2"), Variable.get("ecs_subnet3")],
            "assignPublicIp": "ENABLED",
        },
    },
    trigger_rule="all_done",
    waiter_delay=60,
    waiter_max_attempts=3000,
).expand(overrides=generate_params_list)

Does someone know why it is broken?
Thanks a lot 🥹 (edited)


This happen because partial check signature, and if argument not exists in any MRO references it will failed.
We grab region directly from kwargs and remove it, for avoid provide it into the BaseOperator.

Fix pretty simple, add region into AwsBaseOperator / AwsBaseSensor signatures, it should fix all operators which migrated to this helper operators and use deprecated region attribute over Dynamic Task Mapping


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis
Taragolis requested review from ferruzzi and vincbeck March 15, 2024 11:41
@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 15, 2024
Comment thread airflow/providers/amazon/aws/operators/base_aws.py Outdated
Comment thread tests/providers/amazon/aws/operators/test_ecs.py Outdated
@Taragolis Taragolis added the full tests needed We need to run full set of tests for this PR to merge label Mar 15, 2024
@Taragolis Taragolis closed this Mar 15, 2024
@Taragolis Taragolis reopened this Mar 15, 2024
Comment thread airflow/providers/amazon/aws/operators/base_aws.py Outdated
@Taragolis

Copy link
Copy Markdown
Contributor Author

Yeah... full test works fine. Except mypy, but it not related to this PR

Comment thread tests/providers/amazon/aws/operators/test_base_aws.py Outdated
@Taragolis
Taragolis force-pushed the aws-base-on-partial branch from 994395c to 0fab3a5 Compare March 15, 2024 21:01
@Taragolis

Taragolis commented Mar 15, 2024 •

Copy link
Copy Markdown
Contributor Author

Fingers crossed, seems like this test should work (without side effects)

@Taragolis Taragolis changed the title Fix region argument in partial for bases on AwsBaseOperator / AwsBaseSensor Fix region argument in MappedOperator based on AwsBaseOperator / AwsBaseSensor Mar 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers full tests needed We need to run full set of tests for this PR to merge provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants