Repository navigation
Go SDK: serialize a native Dag to Dag JSON - #74319
Merged
jason810496 merged 10 commits intoOct 6, 2026
Merged
Conversation
jason810496
reviewed
Oct 6, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
Thanks for the PR, LGTM overall.
The first 5 comments are my manual review, rest of the comments are reviews by claude.
1 task done
Signed-off-by: PoAn Yang <payang@apache.org>
Both If and Switch tasks skip downstream tasks, so write the flag for any decider. Also step an edge out of an empty nested group to the group holding it, as Python's find_leaves does, so the conformance_nested_group_edges case passes.
genspec -fields writes spec_fields.gen.go, a table of schema key and default for each spec field keyed by Go field name. serialize.go keeps only the skip and set lists.
EmailOnFailure and EmailOnRetry stay settable, as in the TS SDK, but the serializer skips them because Python writes them only with an email recipient. Say so in the generated doc.
A negative MaxActiveRuns passed the old check. Also note that this validation should eventually run only at build time.
Inputs now takes *TaskRef and Literal(value) inputs, and a literal serializes as a literal arg binding like the TS SDK does. A literal must be JSON, hold no TaskRef, and decode into its parameter type through the runtime's binding decoder. It adds no edge.
conformance_test.go is now package airflow_test, so it can only call exported API. It sets spec fields from the generated field names, and export_test.go exposes only SerializeDag and the field-name lookup.
test_dags.yaml gains cases for branch, switch, edge labels, group options, TriggerDagRun, literal inputs and a cron with seconds. A case lists the features it requires, and compare.py --supports hands an SDK only the cases it supports. go_native.json now comes from the Go output for every case, and recorded_payload_test.go is removed.
Inputs takes literal values as well as task results.
jason810496
force-pushed
the
airflow-go-sdk-serialize-native-dag
branch
from
October 6, 2026 12:54
006ac3c to
b041adf
Compare
FrankYang0529
marked this pull request as ready for review
October 6, 2026 13:49
FrankYang0529
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jscheffl and
potiuk
as code owners
October 6, 2026 13:49
As Python's extract >> Label("rows") >> transform does, a labelled edge from a task
that no group holds to a group also labels the edges to the group's first tasks.
jason810496
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement Go SDK: Serialize a native Dag to Dag JSON.
DagRef.serializewrites{"__version": 3, "dag": {...}}for a registered Dag. Thedagobject holds the timetable, the tasks, the edge labels and adag_dependenciesentry for eachTriggerDagRuntask. The task groups form a tree, written aschildren[label] = [kind, value], and each group records its group edges.TriggerDagRuntask gets the fields that Python writes for aTriggerDagRunOperator. TheTriggerDagRuntask also carries the other parameters, such aspoke_interval. An unsetLogicalDateis written asNOTSET.genspecgeneratesspec_fields.gen.go, which maps each field ofDagSpec,TaskSpecandTaskGroupSpecto its schema property and schema default, as the TS SDK generates its field table. A field that is unset or at its schema default is left out.max_active_runsand four other fields unset, Python takes their values from the Airflow config. A bundle cannot read that config, so the serializer leaves the five fields out. Airflow core fills them in from the config when it receives the serialized Dag.DagSpechas a newQueuefield, because the Go SDK has nodefault_argsto give every task a queue. A Go task runs on the queue thatDagSpec.Queuenames, unless itsTaskSpecsets aQueue. ATriggerDagRuntask does not take the Dag'sQueue, because a Python worker runs the task.TriggerDagRunSpec.Confused to be copied as ajson.Number, and is now copied as anint64, auint64or afloat64. A bundle sends the serialized Dag to Airflow as msgpack, which writes ajson.Numberas a string. An integer that fits in neither 64-bit integer type is rejected.Review follow-ups
Literal inputs.
airflow.Inputstakes a task orairflow.Literal(value). A literal is serialized as the TS SDK writes it,{"name": "argN", "kind": "literal", "value": ...}, and adds no edge.Switch. A
dag.Switchtask is written with_can_skip_downstream, like adag.Iftask.Edge labels. A labelled edge from a task that no group holds to a task group also labels the edges to the group's first tasks, as Python's
extract >> Label("rows") >> transformdoes.Empty nested group. An edge out of an empty nested group falls back to its parent group, as Python's
find_leavesdoes.Validation.
@continuousrequiresMaxActiveRunsto be 1.EmailOnFailureandEmailOnRetryare documented as having no effect yet.Shared conformance cases. The recorded Go payload moved into
scripts/ci/lang_sdk_serialization/test_dags.yamlas cases for branches, switches, edge labels, group options,TriggerDagRun, literal inputs and a six-field cron.requires, andcompare.py --supportslets an SDK run only the cases it can build. The TS SDK runs the cases that need no feature.go_native.jsonis generated from these cases.Public API in the conformance test. The conformance test builds every Dag through the public API (
airflow.Dag,dag.Task, the spec structs) from an external test package.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines