Repository navigation
Support code packages in MWAA Serverless workflow operators - #72622
Conversation
MWAA Serverless added PythonOperator and BashOperator support, which requires shipping the Python modules and shell scripts those tasks run separately from the YAML workflow definition, through the Code request parameter. Neither MwaaServerlessCreateWorkflowOperator nor MwaaServerlessUpdateWorkflowOperator exposed Code, and neither accepts a kwargs passthrough, so the parameter was unreachable. Workflows using those operators could not be created, and their code could not be updated, from Airflow. Add a `code` parameter to both operators, named after the API field and passed through untouched so a future member of the Code union needs no operator change. prune_dict omits it when unset, so existing DAGs are unaffected.
o-nikolas
left a comment
There was a problem hiding this comment.
This is nice good, short and to the point. Though there are still other params that these operators don't forward through to the API call, and adding more of this handling/testing seems not worth it. I'm okay with merging this one, but if you want to do a follow-up @yamakazushi adding a create_workflow_kwargs and update_workflow_kwargs where the lesser used inputs can be passed through (and any future additions) would be great.
|
CC @kars0508 |
|
o-nikolas, thank you for reviewing my commits. As you mentioned, it is good that other parameters will be added but this commit is my first commits excluding documentation fix. Therefore, I would like to focus on adding |
Sounds good, I'm looking forward to the follow-up 😃 |
MWAA Serverless added PythonOperator and BashOperator support, which requires shipping the Python modules and shell scripts those tasks run separately from the YAML workflow definition, through the Code request parameter.
Neither MwaaServerlessCreateWorkflowOperator nor
MwaaServerlessUpdateWorkflowOperator exposed Code, and neither accepts a kwargs passthrough, so the parameter was unreachable. Workflows using those operators could not be created, and their code could not be updated, from Airflow.
Add a
codeparameter to both operators, named after the API field and passed through untouched so a future member of the Code union needs no operator change. prune_dict omits it when unset, so existing DAGs are unaffected.Was generative AI tooling used to co-author this PR?
kiro
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.