Repository navigation
Fix deferred dbt Cloud tasks failing with tenacity retry_args - #74489
FrankYang0529 wants to merge 1 commit into
Conversation
Signed-off-by: PoAn Yang <payang@apache.org>
|
Thanks for finding this and for the fix. The regression is from #74199, which is mine, and your diagnosis is right. I verified The approach looks right to me. Keeping only the JSON serializable values and warning per dropped key is strictly better than both alternatives: rc1 fails at defer time after the dbt job has already started, and 4.10.0 dropped every The note about Happy to review or take any follow-up you would rather hand off. |
#74199 (in apache-airflow-providers-dbt-cloud 4.11.0rc1) made
DbtCloudRunJobOperatorpasshook_paramstoDbtCloudRunJobTrigger. A task withdeferrable=Truethat worked on 4.10.0 now fails when itshook_paramsholdsretry_argswith tenacity objects.Why
Users put objects such as
tenacity.stop_after_attempt(3)inretry_args. Airflow cannot store those objects with the trigger. On Airflow 3.2 and later, the task fails withTypeError: Cannot serialize object of type <class 'tenacity.stop.stop_after_attempt'>when it defers.How
DbtCloudRunJobTriggerkeeps only thehook_paramsvalues thatjson.dumpsaccepts and logs a warning for each value it drops. The hook inside the trigger then uses the default value for each dropped argument.Was generative AI tooling used to co-author this PR?