Skip to content

Go SDK: serialize a native Dag to Dag JSON - #74319

Merged
jason810496 merged 10 commits into
apache:mainfrom
FrankYang0529:airflow-go-sdk-serialize-native-dag
Oct 6, 2026
Merged

jason810496 merged 10 commits into
apache:mainfrom
FrankYang0529:airflow-go-sdk-serialize-native-dag

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Implement Go SDK: Serialize a native Dag to Dag JSON.

  • DagRef.serialize writes {"__version": 3, "dag": {...}} for a registered Dag. The dag object holds the timetable, the tasks, the edge labels and a dag_dependencies entry for each TriggerDagRun task. The task groups form a tree, written as children[label] = [kind, value], and each group records its group edges.
  • A TriggerDagRun task gets the fields that Python writes for a TriggerDagRunOperator. The TriggerDagRun task also carries the other parameters, such as poke_interval. An unset LogicalDate is written as NOTSET.
  • genspec generates spec_fields.gen.go, which maps each field of DagSpec, TaskSpec and TaskGroupSpec to its schema property and schema default, as the TS SDK generates its field table. A field that is unset or at its schema default is left out.
  • When a Dag leaves max_active_runs and four other fields unset, Python takes their values from the Airflow config. A bundle cannot read that config, so the serializer leaves the five fields out. Airflow core fills them in from the config when it receives the serialized Dag.
  • DagSpec has a new Queue field, because the Go SDK has no default_args to give every task a queue. A Go task runs on the queue that DagSpec.Queue names, unless its TaskSpec sets a Queue. A TriggerDagRun task does not take the Dag's Queue, because a Python worker runs the task.
  • Each number in TriggerDagRunSpec.Conf used to be copied as a json.Number, and is now copied as an int64, a uint64 or a float64. A bundle sends the serialized Dag to Airflow as msgpack, which writes a json.Number as a string. An integer that fits in neither 64-bit integer type is rejected.

Review follow-ups

  • Literal inputs. airflow.Inputs takes a task or airflow.Literal(value). A literal is serialized as the TS SDK writes it, {"name": "argN", "kind": "literal", "value": ...}, and adds no edge.

    extracted := dag.Task(extract)
    dag.Task(load, airflow.Inputs(extracted, airflow.Literal("uk")))
  • Switch. A dag.Switch task is written with _can_skip_downstream, like a dag.If task.

  • Edge labels. A labelled edge from a task that no group holds to a task group also labels the edges to the group's first tasks, as Python's extract >> Label("rows") >> transform does.

  • Empty nested group. An edge out of an empty nested group falls back to its parent group, as Python's find_leaves does.

  • Validation. @continuous requires MaxActiveRuns to be 1. EmailOnFailure and EmailOnRetry are documented as having no effect yet.

  • Shared conformance cases. The recorded Go payload moved into scripts/ci/lang_sdk_serialization/test_dags.yaml as cases for branches, switches, edge labels, group options, TriggerDagRun, literal inputs and a six-field cron.

    • Each case lists the features it requires, and compare.py --supports lets an SDK run only the cases it can build. The TS SDK runs the cases that need no feature.
    • go_native.json is generated from these cases.
  • Public API in the conformance test. The conformance test builds every Dag through the public API (airflow.Dag, dag.Task, the spec structs) from an external test package.


Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

Generated-by: Claude Code (Opus 5.5) following the guidelines

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, LGTM overall.
The first 5 comments are my manual review, rest of the comments are reviews by claude.

Comment thread go-sdk/airflow/conformance_test.go Outdated
Comment thread go-sdk/airflow/dag.go
Comment thread go-sdk/airflow/recorded_payload_test.go Outdated
Comment thread go-sdk/airflow/serialize.go Outdated
Comment thread go-sdk/airflow/serialize.go Outdated
Comment thread go-sdk/airflow/serialize.go Outdated
Comment thread go-sdk/airflow/dag.go Outdated
Comment thread go-sdk/airflow/trigger_dag_run.go
Comment thread go-sdk/airflow/serialize.go Outdated
Comment thread go-sdk/airflow/serialize.go Outdated
@potiuk potiuk removed the backport-to-v3-4-test Backport to v3-4-test label Oct 6, 2026
FrankYang0529 and others added 9 commits October 6, 2026 12:29
Signed-off-by: PoAn Yang <payang@apache.org>
Both If and Switch tasks skip downstream tasks, so write the flag for any decider.
Also step an edge out of an empty nested group to the group holding it, as Python's
find_leaves does, so the conformance_nested_group_edges case passes.
genspec -fields writes spec_fields.gen.go, a table of schema key and default for each
spec field keyed by Go field name. serialize.go keeps only the skip and set lists.
EmailOnFailure and EmailOnRetry stay settable, as in the TS SDK, but the serializer skips them
because Python writes them only with an email recipient. Say so in the generated doc.
A negative MaxActiveRuns passed the old check. Also note that this validation should
eventually run only at build time.
Inputs now takes *TaskRef and Literal(value) inputs, and a literal serializes as a literal
arg binding like the TS SDK does. A literal must be JSON, hold no TaskRef, and decode into
its parameter type through the runtime's binding decoder. It adds no edge.
conformance_test.go is now package airflow_test, so it can only call exported API. It sets
spec fields from the generated field names, and export_test.go exposes only SerializeDag
and the field-name lookup.
test_dags.yaml gains cases for branch, switch, edge labels, group options, TriggerDagRun,
literal inputs and a cron with seconds. A case lists the features it requires, and compare.py
--supports hands an SDK only the cases it supports. go_native.json now comes from the Go output
for every case, and recorded_payload_test.go is removed.
Inputs takes literal values as well as task results.
@jason810496
jason810496 force-pushed the airflow-go-sdk-serialize-native-dag branch from 006ac3c to b041adf Compare October 6, 2026 12:54
@FrankYang0529
FrankYang0529 marked this pull request as ready for review October 6, 2026 13:49
As Python's extract >> Label("rows") >> transform does, a labelled edge from a task
that no group holds to a group also labels the edges to the group's first tasks.
@jason810496
jason810496 merged commit 64ac6e1 into apache:main Oct 6, 2026
97 of 98 checks passed
@FrankYang0529
FrankYang0529 deleted the airflow-go-sdk-serialize-native-dag branch October 6, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants