Record the Go SDK's Mixed Lang Task and Native Dag task interfaces by jason810496 · Pull Request #72043 · apache/airflow

jason810496 · 2026-08-25T02:40:36Z

related: Bind TaskFlow stub-task call arguments in the Go SDK runtime #70209, Go-SDK: Dag/Task specs and downstream wiring in the authoring API #67155, Support TaskFlow Dag definitions natively in the Go-SDK #70158

Write the design for TaskFlow and native Dag for the Go-SDK up front, to settle the design early and surface as much context as possible for reviewers on the actual implementation PRs.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code Sonnet 5 following the guidelines

Add ADRs for the Mixed Lang Dag interface already shipped via apache#70209, the proposed Native Dag interface (apache#67155/apache#70158), and the common task constructs a native Go author will need next (TaskGroup, ShortCircuit, Branch, TriggerDagRun). Recording the rationale here gives reviewers and future contributors a single reference for these tradeoffs instead of reconstructing them from scattered PR discussions.

The design review on apache#72043 landed on different shapes than the ones these ADRs first described. A Mixed Lang Dag borrowed Dag vocabulary for something that defines no Dag on the Go side, tasks received the SDK's own context type where Go authors expect Go's, and an edge could only be declared in one direction. Recording what the review arrived at keeps these documents a reference for the interface the SDK will ship rather than the one it proposed, before either interface reaches users.

How a native Dag expresses grouping, conditions, branching, and triggering another Dag's run is a decision every Lang SDK faces, and so is reaching Python provider operators from another language. Both bind core serialization and the Execution API messages rather than anything specific to Go, so the Java SDK and any SDK after it need them in the cross-cutting set instead of buried in the Go SDK's directory. Naming these constructs after Python's operator classes was rejected in the review on apache#72043: an author who has never seen Airflow should not have to learn its operator taxonomy to write an if statement.

The design review on apache#72043 landed on different shapes than the ones these ADRs first described. A Mixed Lang Dag borrowed Dag vocabulary for something that defines no Dag on the Go side, tasks received the SDK's own context type where Go authors expect Go's, and an edge could only be declared in one direction. Recording what the review arrived at keeps these documents a reference for the interface the SDK will ship rather than the one it proposed, before either interface reaches users.

How a native Dag expresses grouping, conditions, branching, and triggering another Dag's run is a decision every Lang SDK faces, and so is reaching Python provider operators from another language. Both bind core serialization and the Execution API messages rather than anything specific to Go, so the Java SDK and any SDK after it need them in the cross-cutting set instead of buried in the Go SDK's directory. Naming these constructs after Python's operator classes was rejected in the review on apache#72043: an author who has never seen Airflow should not have to learn its operator taxonomy to write an if statement.

Review on apache#72043 pointed out that reaching the logger through a package-level accessor over a stdlib context reads oddly in Go, and it also lets a caller pass a context the SDK never populated: the task then compiles and fails at run time on a missing value. A context type the task is required to accept carries the logger, client, and run identifiers as methods, keeps supervisor cancellation working because it embeds the stdlib context, and turns that class of mistake into a compile error. The edge verbs no longer need a third name either, now that ordering is expressed in both directions.

Review on apache#72043 settled three things these ADRs had not caught up with. A bundle should expose native Dags and Mixed Lang task handlers through one registration call rather than two, since a bundle usually provides both and nothing is made safer by splitting them. A task should receive one purpose-built context rather than a stdlib context plus package-level accessors, which read oddly and let a caller pass a context the SDK never populated. And ordering should reach a task group, as Python's own DependencyMixin has always allowed, rather than tasks alone. Each document now opens with its decision, so a reader knows the outcome before the reasoning, and the file-and-line grounding moves to an appendix where it no longer competes with it. The context type is a struct rather than an interface: review pointed out that the context package warns against domain types holding a request-scoped context, not against a purpose-built context type, which was the reason recorded here and in the shipped doc comment for choosing an interface.

Both documents read as an argument that arrives at a decision somewhere in the middle, which asks a reviewer to reconstruct the outcome from the reasoning. Opening with the numbered decision and moving the wire protocol and serialization detail to an appendix lets a reader stop after the consequences and still know what was decided. The grouping decision also gains what the Go SDK settled separately: a group stands at the end of an edge the way a task does, so every Lang SDK needs one base type both satisfy. Python has had exactly that since TaskGroup and every operator began inheriting DependencyMixin.

jason810496

Hi @ashb,

I just reworded all the ADRs by moving the implementation details note into the appendix, moving the decision to the first section, and make the code example as a dedicated section.

It should be more readable and self-contain, thanks for the review.

@task

Writing a bundle today starts with an empty struct, an interface assertion, and a RegisterDags callback handed to a separate server package, so an author meets three concepts before declaring a single task. Nothing about that inversion is load-bearing: Registry is already Bundle plus AddDag, the write side of the value that later answers task lookups, and bundlev1.New already builds one outside the callback for tests. Making the bundle a value the author holds lets main read build, register, serve, and lets the lifecycle state what the two interfaces only implied — registration closes when Serve is called. The handler constructor is named for the Python decorator it implements, @task.stub, rather than for tasks in general, since a native Go task is registered a different way entirely.

@task

What an author registers is the Go side of one Python @task.stub task, and calling it a handler named the callback shape rather than the thing itself. StubTask matches the vocabulary a Dag author already has from Python, and it lines up with the native side, where a task is also what gets registered.

Python declares the task and owns its arguments and its place in the graph; Go supplies only the body. TaskHandler names that relationship, where a task-shaped name would suggest the Go side declares a task of its own, which is what the native interface does instead. It also keeps the registration vocabulary distinct between the two interfaces, so a reader can tell which one a call belongs to.

An ADR has to carry the why, not the shape alone. The review asked why a handler must take one context, why a Mixed Lang task writes both ids out, and what the native Dag surface actually is; those answers belong in the document rather than in a review thread. Each ADR now writes its surface out once, so the interface can be judged without reconstructing it from prose.

The self review rejected deriving a native Dag's task_id from the Go function name, so both SDK paths now name it the same way. Labelling the endpoint instead of the call is what lets one fan-out carry a different label per edge, and an edge verb that returns what it pointed at is what lets those calls chain.

The review asked how a TaskSpec can be generated from core's serialization schema and still satisfy a sealed option interface. It can only do so from inside the airflow package, and the schema's missing titles and serialized-only fields are what the generator has to work around, so both belong in the record. The claim that requiring a context turns a test mistake into a compile error was wrong in both directions: passing context.Background() is valid today, and a handler signature is checked by reflection at registration, never by the compiler. It is removed rather than reworded.

A rename is something the ADR decides, not something that follows from it, so Consequences is the wrong home for it. The package-accessor alternative keeps the reason it loses, that nothing surfaces until the task runs, and drops the build-time comparison, which claimed a guarantee a reflection-checked signature never gave.

jason810496

I just addressed all the comments and went thought all the statement and rephrase or remove the AI slop part the I caught.

ashb

I think the only thing I'm not sure about is if we have to give a task name every time in dag.Task(), or if we could default to the fn name and make TaskName an option?

Especially if you are writing a task natively/entirely in go, then the dunder_case "default" of python doesn't matter, and using the name of the go function is a sensible default.

A native Dag is Go's from end to end, so naming every task next to the function that implements it bought nothing; the id now comes from the function, and TaskSpec is where an author pins something else. Allowing one spec per task keeps the SDK from having to define what merging two of them would mean. ShortCircuitOperator skips its whole downstream closure and ignores the trigger rules inside it, which is more than a conditional branch should mean, so If serializes as a branch operator the way Switch already does. A decider returns the task reference it picked rather than an id string, which lets the compiler check the candidate and leaves the id lookup to the runtime. The logger and the client keep taking the context as an argument instead of holding one, since a stored request-scoped context is the shape the context package warns against.

jason810496

Thanks @ashb for the discussion and the review, I just applied what we had discussed and resolve all the comment, in the latest commit (990ab2f). Thanks.

jason810496 requested review from amoghrajesh and ashb as code owners August 25, 2026 02:40

boring-cyborg Bot added the area:go-sdk label Aug 25, 2026

jason810496 self-assigned this Aug 25, 2026

jason810496 requested a review from uranusjr August 25, 2026 02:40

jason810496 added the type:doc-only Changelog: Doc Only label Aug 25, 2026

ashb reviewed Aug 25, 2026

View reviewed changes

Comment thread go-sdk/adr/0006-mixed-lang-dag-interface.md Outdated

Comment thread go-sdk/adr/0008-taskgroup-shortcircuit-branch.md Outdated

Comment thread go-sdk/adr/0008-taskgroup-shortcircuit-branch.md Outdated

Comment thread go-sdk/adr/0008-taskgroup-shortcircuit-branch.md Outdated

jason810496 marked this pull request as draft September 7, 2026 02:14

jason810496 force-pushed the docs/go-sdk/adr-interface-design branch from ca3033e to 543507b Compare September 7, 2026 07:29

jason810496 added 2 commits September 7, 2026 07:33

jason810496 force-pushed the docs/go-sdk/adr-interface-design branch from 543507b to 16c07f2 Compare September 7, 2026 07:34

jason810496 marked this pull request as ready for review September 7, 2026 07:41

jason810496 requested a review from ashb September 7, 2026 07:42

ashb reviewed Sep 8, 2026

View reviewed changes

Comment thread go-sdk/adr/0007-native-dag-interface.md Outdated

jason810496 marked this pull request as draft September 8, 2026 11:25

jason810496 changed the title ~~Record the Go SDK's Mixed Lang and Native Dag task interfaces~~ Record the Go SDK's Mixed Lang Task and Native Dag task interfaces Sep 9, 2026

jason810496 added 2 commits September 9, 2026 02:59

jason810496 marked this pull request as ready for review September 9, 2026 03:16

jason810496 requested a review from ashb September 9, 2026 03:16

jason810496 commented Sep 9, 2026

View reviewed changes

jason810496 mentioned this pull request Sep 9, 2026

Add a Language SDK spec to the Task SDK docs #72765

Merged

1 task done

jason810496 marked this pull request as draft September 9, 2026 05:29

jason810496 marked this pull request as ready for review September 9, 2026 08:23

jason810496 added 2 commits September 9, 2026 08:38

ashb reviewed Sep 14, 2026

View reviewed changes

jason810496 marked this pull request as draft September 14, 2026 11:32

jason810496 added 6 commits September 14, 2026 21:09

Address some comments myself

cd43315

Self review with comments

e1c044b

jason810496 marked this pull request as ready for review September 15, 2026 11:42

jason810496 commented Sep 15, 2026

View reviewed changes