Skip to content

Fix deferred dbt Cloud tasks failing with tenacity retry_args - #74489

Draft
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-dbt-cloud-trigger-unserializable-hook-params
Draft

FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-dbt-cloud-trigger-unserializable-hook-params

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

#74199 (in apache-airflow-providers-dbt-cloud 4.11.0rc1) made DbtCloudRunJobOperator pass hook_params to DbtCloudRunJobTrigger. A task with deferrable=True that worked on 4.10.0 now fails when its hook_params holds retry_args with tenacity objects.

Why

Users put objects such as tenacity.stop_after_attempt(3) in retry_args. Airflow cannot store those objects with the trigger. On Airflow 3.2 and later, the task fails with TypeError: Cannot serialize object of type <class 'tenacity.stop.stop_after_attempt'> when it defers.

How

DbtCloudRunJobTrigger keeps only the hook_params values that json.dumps accepts and logs a warning for each value it drops. The hook inside the trigger then uses the default value for each dropped argument.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Signed-off-by: PoAn Yang <payang@apache.org>
@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor

Thanks for finding this and for the fix. The regression is from #74199, which is mine, and your diagnosis is right.

I verified hook_params round tripped through serialize() but used {"retry_limit": 7, "retry_delay": 3.5}, so the input never reached the serializer's limits. retry_args is the one key in that dict whose documented values are objects, and it is the case I should have used.

The approach looks right to me. Keeping only the JSON serializable values and warning per dropped key is strictly better than both alternatives: rc1 fails at defer time after the dbt job has already started, and 4.10.0 dropped every hook_params value silently. Falling back to retry_limit and retry_delay for the trigger's hook is a sensible floor, and worth having in the docstring as you did.

The note about airflow.sdk.serde replacing BaseSerialization in 3.2 is useful: it explains why this is invisible on older versions, and I had 3.3.2 installed, so my own check would have caught it with the right value.

Happy to review or take any follow-up you would rather hand off.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants