Repository navigation
Only rebuild built-in value types when converting next_kwargs for older workers - #73577
Merged
potiuk merged 9 commits intoOct 19, 2026
Merged
Conversation
…er workers When serving a task instance to a worker on an older Execution API version, the version converter decodes the stored serde next_kwargs and re-encodes them in the legacy format. Decoding reconstructs every value in the API server, including instances of arbitrary classes, although only the worker ever needs those objects. Only decode payloads whose values are built-in value types - the types the legacy format can carry anyway - and pass anything else through as stored. Generated-by: Claude Opus 5
Member
Author
|
@ashb @amoghrajesh — gentle reminder: this needs a review to make the 3.4.0 cut on 2026-10-19. CI is green and it's ready from my side. |
4 tasks done
Member
Author
|
@uranusjr @amoghrajesh Could you take a look at this when you have a moment? It's targeted at 3.4.0 and needs a review to make the cut on 2026-10-19. CI is green. |
Task SDK 1.1.x always decodes next_kwargs with BaseSerialization.deserialize,
so passing a serde payload through unchanged makes the deferred task crash
on resume with KeyError('__var'). That broke resuming with Asset or
AssetAlias values, which the legacy format supports natively.
Assets and asset aliases are now translated structurally from their
encoded fields into the legacy envelope, so nothing beyond built-in value
types is instantiated in the API server. Any other class makes the task
resume through the trigger-failure path, so older workers fail it with a
TaskDeferralError naming the type instead of an opaque KeyError. Payloads
already in the legacy format are still passed through.
Generated-by: Claude Opus 5
A tuple or set holding an Asset or AssetAlias failed the built-in value check because of the nested asset, so its serde envelope was treated as unconvertible and the task resumed through the trigger-failure path, even though the legacy format carries tuples and sets of assets. Tuple, set and frozenset envelopes are now translated structurally into the legacy tuple/set encoding, with their items converted recursively and set items sorted when comparable, as BaseSerialization.serialize does. The legacy format has no frozenset, so it is sent as a set. No class beyond the built-in value types is constructed. Generated-by: Claude Opus 5
…nsets of assets
A dict was taken for a legacy BaseSerialization envelope as soon as it had
a "__type" key, so user data such as {"__type": "record", "asset": ...}
resumed the task through the failure path, and a top-level dict with both
"__type" and "__var" plus other keys was passed through unconverted. Only
a dict whose keys are exactly "__type" and "__var" is now treated as an
envelope; any other dict is converted recursively.
Frozensets holding assets were sent as legacy sets, which older workers
cannot decode when the frozenset sits inside a set. The legacy format has
no frozenset, so these now resume through the trigger-failure path with a
clear message. Frozensets of built-in values keep the built-in path.
Generated-by: Claude Opus 5
…next_kwargs The built-in value check followed only "__var" of any dict holding "__type" and "__var", while serde's decoder unwraps such a dict and then decodes the inner dict as a class envelope without re-checking it. A class envelope placed inside a legacy dict envelope was therefore passed as built-in and constructed in the API server, and other keys of such a dict were silently dropped. The check now mirrors the decoder and fails safe: only exact envelopes count as envelopes, a dict envelope's content must be a plain dict, class envelopes must have exactly the serde keys, and any other dict holding a reserved key is left to the structural translation. A top-level payload already in the legacy format is passed through only when every dict in it is an envelope the legacy decoder can read; otherwise the task resumes through the trigger-failure path. Legacy payloads are no longer decoded in the API server at all. Generated-by: Claude Opus 5
The structural Asset translation copied only name, uri, group and extra, so a non-default access_control was dropped, while encode_asset_like() emits it whenever producer_teams is set, consumer_teams is not None or allow_global is false. The translation now builds the access_control dict exactly as encode_asset_like() does, from the serde data of AssetAccessControl, with nothing constructed. An Asset or AssetAlias whose serde data holds a field or shape that cannot be mapped with confidence (unknown keys, watchers, class envelopes in extra, malformed access control) resumes through the trigger-failure path rather than losing data silently. Generated-by: Claude Opus 5
Generated-by: Claude Opus 5
Generated-by: Claude Opus 5
Generated-by: Claude Opus 5
vatsrahul1001
approved these changes
Oct 18, 2026
Contributor
There was a problem hiding this comment.
LGTM. Could use reviews from @amoghrajesh @uranusjr @kaxil before merging.
Contributor
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
potiuk
added a commit
that referenced
this pull request
Oct 19, 2026
…args for older workers (#73577) (#75115) * Only rebuild built-in value types when converting next_kwargs for older workers When serving a task instance to a worker on an older Execution API version, the version converter decodes the stored serde next_kwargs and re-encodes them in the legacy format. Decoding reconstructs every value in the API server, including instances of arbitrary classes, although only the worker ever needs those objects. Only decode payloads whose values are built-in value types - the types the legacy format can carry anyway - and pass anything else through as stored. Generated-by: Claude Opus 5 * Keep converting Asset values in next_kwargs for older workers Task SDK 1.1.x always decodes next_kwargs with BaseSerialization.deserialize, so passing a serde payload through unchanged makes the deferred task crash on resume with KeyError('__var'). That broke resuming with Asset or AssetAlias values, which the legacy format supports natively. Assets and asset aliases are now translated structurally from their encoded fields into the legacy envelope, so nothing beyond built-in value types is instantiated in the API server. Any other class makes the task resume through the trigger-failure path, so older workers fail it with a TaskDeferralError naming the type instead of an opaque KeyError. Payloads already in the legacy format are still passed through. Generated-by: Claude Opus 5 * Translate tuples and sets of assets in next_kwargs for older workers A tuple or set holding an Asset or AssetAlias failed the built-in value check because of the nested asset, so its serde envelope was treated as unconvertible and the task resumed through the trigger-failure path, even though the legacy format carries tuples and sets of assets. Tuple, set and frozenset envelopes are now translated structurally into the legacy tuple/set encoding, with their items converted recursively and set items sorted when comparable, as BaseSerialization.serialize does. The legacy format has no frozenset, so it is sent as a set. No class beyond the built-in value types is constructed. Generated-by: Claude Opus 5 * Only treat exact legacy envelopes as legacy and fail clearly on frozensets of assets A dict was taken for a legacy BaseSerialization envelope as soon as it had a "__type" key, so user data such as {"__type": "record", "asset": ...} resumed the task through the failure path, and a top-level dict with both "__type" and "__var" plus other keys was passed through unconverted. Only a dict whose keys are exactly "__type" and "__var" is now treated as an envelope; any other dict is converted recursively. Frozensets holding assets were sent as legacy sets, which older workers cannot decode when the frozenset sits inside a set. The legacy format has no frozenset, so these now resume through the trigger-failure path with a clear message. Frozensets of built-in values keep the built-in path. Generated-by: Claude Opus 5 * Never deserialize non-builtin classes behind legacy-looking dicts in next_kwargs The built-in value check followed only "__var" of any dict holding "__type" and "__var", while serde's decoder unwraps such a dict and then decodes the inner dict as a class envelope without re-checking it. A class envelope placed inside a legacy dict envelope was therefore passed as built-in and constructed in the API server, and other keys of such a dict were silently dropped. The check now mirrors the decoder and fails safe: only exact envelopes count as envelopes, a dict envelope's content must be a plain dict, class envelopes must have exactly the serde keys, and any other dict holding a reserved key is left to the structural translation. A top-level payload already in the legacy format is passed through only when every dict in it is an envelope the legacy decoder can read; otherwise the task resumes through the trigger-failure path. Legacy payloads are no longer decoded in the API server at all. Generated-by: Claude Opus 5 * Keep asset access control when translating next_kwargs for older workers The structural Asset translation copied only name, uri, group and extra, so a non-default access_control was dropped, while encode_asset_like() emits it whenever producer_teams is set, consumer_teams is not None or allow_global is false. The translation now builds the access_control dict exactly as encode_asset_like() does, from the serde data of AssetAccessControl, with nothing constructed. An Asset or AssetAlias whose serde data holds a field or shape that cannot be mapped with confidence (unknown keys, watchers, class envelopes in extra, malformed access control) resumes through the trigger-failure path rather than losing data silently. Generated-by: Claude Opus 5 * Add newsfragment for older-worker next_kwargs conversion Generated-by: Claude Opus 5 * Tell users how to avoid next_kwargs failures on older workers Generated-by: Claude Opus 5 * Name Task SDK 1.1.x as the affected worker versions (cherry picked from commit b0239d2) Generated-by: Claude Opus 5 Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 20, 2026
…args for older workers (#73577) (#75115) * Only rebuild built-in value types when converting next_kwargs for older workers When serving a task instance to a worker on an older Execution API version, the version converter decodes the stored serde next_kwargs and re-encodes them in the legacy format. Decoding reconstructs every value in the API server, including instances of arbitrary classes, although only the worker ever needs those objects. Only decode payloads whose values are built-in value types - the types the legacy format can carry anyway - and pass anything else through as stored. Generated-by: Claude Opus 5 * Keep converting Asset values in next_kwargs for older workers Task SDK 1.1.x always decodes next_kwargs with BaseSerialization.deserialize, so passing a serde payload through unchanged makes the deferred task crash on resume with KeyError('__var'). That broke resuming with Asset or AssetAlias values, which the legacy format supports natively. Assets and asset aliases are now translated structurally from their encoded fields into the legacy envelope, so nothing beyond built-in value types is instantiated in the API server. Any other class makes the task resume through the trigger-failure path, so older workers fail it with a TaskDeferralError naming the type instead of an opaque KeyError. Payloads already in the legacy format are still passed through. Generated-by: Claude Opus 5 * Translate tuples and sets of assets in next_kwargs for older workers A tuple or set holding an Asset or AssetAlias failed the built-in value check because of the nested asset, so its serde envelope was treated as unconvertible and the task resumed through the trigger-failure path, even though the legacy format carries tuples and sets of assets. Tuple, set and frozenset envelopes are now translated structurally into the legacy tuple/set encoding, with their items converted recursively and set items sorted when comparable, as BaseSerialization.serialize does. The legacy format has no frozenset, so it is sent as a set. No class beyond the built-in value types is constructed. Generated-by: Claude Opus 5 * Only treat exact legacy envelopes as legacy and fail clearly on frozensets of assets A dict was taken for a legacy BaseSerialization envelope as soon as it had a "__type" key, so user data such as {"__type": "record", "asset": ...} resumed the task through the failure path, and a top-level dict with both "__type" and "__var" plus other keys was passed through unconverted. Only a dict whose keys are exactly "__type" and "__var" is now treated as an envelope; any other dict is converted recursively. Frozensets holding assets were sent as legacy sets, which older workers cannot decode when the frozenset sits inside a set. The legacy format has no frozenset, so these now resume through the trigger-failure path with a clear message. Frozensets of built-in values keep the built-in path. Generated-by: Claude Opus 5 * Never deserialize non-builtin classes behind legacy-looking dicts in next_kwargs The built-in value check followed only "__var" of any dict holding "__type" and "__var", while serde's decoder unwraps such a dict and then decodes the inner dict as a class envelope without re-checking it. A class envelope placed inside a legacy dict envelope was therefore passed as built-in and constructed in the API server, and other keys of such a dict were silently dropped. The check now mirrors the decoder and fails safe: only exact envelopes count as envelopes, a dict envelope's content must be a plain dict, class envelopes must have exactly the serde keys, and any other dict holding a reserved key is left to the structural translation. A top-level payload already in the legacy format is passed through only when every dict in it is an envelope the legacy decoder can read; otherwise the task resumes through the trigger-failure path. Legacy payloads are no longer decoded in the API server at all. Generated-by: Claude Opus 5 * Keep asset access control when translating next_kwargs for older workers The structural Asset translation copied only name, uri, group and extra, so a non-default access_control was dropped, while encode_asset_like() emits it whenever producer_teams is set, consumer_teams is not None or allow_global is false. The translation now builds the access_control dict exactly as encode_asset_like() does, from the serde data of AssetAccessControl, with nothing constructed. An Asset or AssetAlias whose serde data holds a field or shape that cannot be mapped with confidence (unknown keys, watchers, class envelopes in extra, malformed access control) resumes through the trigger-failure path rather than losing data silently. Generated-by: Claude Opus 5 * Add newsfragment for older-worker next_kwargs conversion Generated-by: Claude Opus 5 * Tell users how to avoid next_kwargs failures on older workers Generated-by: Claude Opus 5 * Name Task SDK 1.1.x as the affected worker versions (cherry picked from commit b0239d2) Generated-by: Claude Opus 5 Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When serving a task instance to a worker on an older Execution API version, the version
converter decodes the stored serde
next_kwargsand re-encodes them in the legacy format.Decoding reconstructs every value in the API server, including instances of arbitrary
classes, although only the worker ever needs those objects.
The converter now only decodes payloads whose values are built-in value types (tuples, sets,
dates and datetimes, timedeltas, timezones, decimals, UUIDs). Assets and asset aliases, which
the legacy format also carries, are translated into it field by field (including asset access
control) without constructing them; tuples and sets holding them are translated the same way.
Any other class is not reconstructed: the task resumes through the trigger-failure path, so
older workers fail it with a
TaskDeferralErrornaming the type instead of crashing on apayload they cannot decode. Payloads already stored in the legacy format are passed through
only when well-formed, and are never decoded in the API server.
Test plan:
fail the task with a clear message on older workers
legacy format without being constructed
reserved keys keep all their fields
already in the database) still pass
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions