Repository navigation
Parameter invalid (cluster) in amazon-> operator -> EcsRunTaskOperator when using deferrable #40207
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
Indeed, the operator
EcsRunTaskOperatoruse some variables that might not have been set when executingexecute_complete. Ideallyexecute_completeshould only use information from the event, the executor/worker runningexecuteandexecute_completemight (and most likely) be differentI'll take it.
Reacted by Elad KalifReacted by VincentReacted by frantoja
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
No response
Apache Airflow version
2.8.1
Operating System
AWS MWAA
Deployment
Amazon (AWS) MWAA
Deployment details
No response
What happened
I have a custom class inheritance from EcsRunTaskOperator.
When I set deferrable = True, the Airflow task runs and changes to the deferred state successfully and also the ECS task runs successfully. However, when it returns (TaskDoneTrigger - task finished), the method execute_completes -> self._after_execution() -> self._check_success_task() -> response = self.client.describe_tasks(cluster=self.cluster, task=[self. arn]) shows an error:
botocore.exceptions.ParamValidationError: Parameter validation failed
Invalid type for parameter cluster, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
What you think should happen instead
I believe (and tested) that the variable self.cluster (approximately line 696 in the file airflow/providers/amazon/aws/operators/ecs.py) "looses the value"
How to reproduce
Set the parameter deferrable to true in EcsRunTaskOperator
Anything else
No response
Are you willing to submit PR?
Code of Conduct