Repository navigation
Go SDK: answer TaskHandlerParseRequest with registered handlers - #73975
Draft
jason810496 wants to merge 6 commits into
Conversation
jason810496
added this pull request to stack #73978
September 30, 2026 17:59
This was referenced Sep 30, 2026
This was referenced Sep 30, 2026
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 1, 2026 06:26
337387a to
b410623
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 1, 2026 06:30
b410623 to
f17015e
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 1, 2026 10:55
f17015e to
421cb23
Compare
This was referenced Oct 1, 2026
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
2 times, most recently
from
October 2, 2026 00:54
78995cf to
b1151dc
Compare
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 2, 2026 05:48
b1151dc to
0fb5c4c
Compare
1 task done
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 2, 2026 11:36
0fb5c4c to
ccecc81
Compare
This was referenced Oct 3, 2026
The Dag processor will check each stub task against the Go handler it runs, so the binding plan has to say how the handler's params bind and which values they accept. Flat params are positional and nameless. A lone struct, tagged or not, is named, and arg tags match exactly. Each param gets a JSON Schema in the vocabulary build_arg_bindings emits, stating only what decoding enforces. Params is an empty list rather than null when there are none, because null means the params cannot be listed.
The Dag processor asks the binary a stub task would run which task handlers it registers, and a Go bundle exits on that first frame as an unknown message. Serve now replies with one TaskHandlerParsingResult listing every handler registered with airflow.TaskHandler, keyed by Dag id in registration order, so the answer depends only on the bundle. It waits for the parent's acknowledgement before returning, so the parent has the result before the process exits. A Dag from airflow.Dag is never a task handler and is left out. Empty params and task_handlers go out as empty values, because the parent rejects a null task_handlers and reads null params as "cannot list".
The Dag processor will probe the binary that the worker's scan picks, and that scan followed each directory's filesystem order. When two binaries list the same Dag, two hosts could pick different ones. The scan now uses walk_files (#73126), as the TypeScript scan does, so the first match in sorted path order wins on every host. A worker runs another binary than before only when two binaries list the same Dag.
The probe needs each coordinator to find the artifact a stub task runs and to start its runtime. ExecutableCoordinator finds the bundle with the scan that execute_task uses, so the Dag processor checks the binary a worker would run, and returns its resolved path, as the task command does. The parse command runs the trailer and binary hash check a task's bundle gets and returns the absolute path, so exec never searches PATH for a relative name.
The probe tests fake the runtime, so nothing showed that a real Go bundle answers the way the Dag processor parses it. The test packs go-sdk/example/bundle, finds it as a stub task of simple_dag would, and probes it. It needs a Go toolchain, so it runs only with AIRFLOW_LANG_SDK_REAL_PROBE_TESTS=1.
The Go SDK is the first runtime to answer TaskHandlerParseRequest, and the Java and TypeScript runtimes follow the spec and the contributor guide. Neither said which coordinator hooks find and start the runtime, that the first frame can be a parse request, what the answer must contain, or what the Dag processor does with a mismatch under each binding. The spec stays at 1.0: the lifecycle adds steps and renames no term.
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 5, 2026 18:12
ccecc81 to
cf557b9
Compare
jason810496
removed this pull request from stack #73978
October 6, 2026 02:27
jason810496
added this pull request to stack #74318
October 6, 2026 02:35
1 task done
jason810496
force-pushed
the
jason/core-taskhandler-refactor/06-go-sdk-task-handler-parse
branch
from
October 6, 2026 02:44
cf557b9 to
04698db
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top), on native #74037 (stack #74170): #73973, #73974, #73975, #74317, #73976, #74067, #74135, #74140, #74141, #73971, #74030, #74031, #74032, #74136, #74137, #74138, #74139
A Go bundle now answers #73974's probe with every handler registered through
airflow.TaskHandlerand how its params bind, andExecutableCoordinatorcan find and start the binary a stub task would run. Until now a Go bundle exited on aTaskHandlerParseRequestas an unknown first frame. Nothing calls the probe yet: #74135 does.answers:
airflow.TaskHandlerregistrations are declared. Tasks of a Dag built withairflow.Dagare never task handlers, so the answer leaves them out.positionaland nameless, because Go reflection has no parameter names. A lone struct isnamed:arg:tags match exactly, other fields ignoring case and_. A lone struct with no bindable fields, such astime.Time, isnamedwithparams=None, so only the handler's presence is checked.value_schemastates only what decoding into the Go type enforces, in the vocabularybuild_arg_bindingsemits for Python annotations. Pointers, slices and maps accept null.any, customjson.Unmarshalertypes,json.Numberand[]byteget no schema, because they accept more than one JSON shape.Servewaits for the parent to acknowledge the answer, so the parent has the result before the process exits._find_task_handler_artifactis the worker's own scan, and_build_parse_task_handler_commandruns the same trailer and hash check a task gets and returns an absolute path, so exec never searchesPATH.Compatibility: a bundle packed from main's Go SDK before this change reports
2026-10-30but cannot answer, so its probe fails and #74135 logs a warning. Repacking fixes it. Bundles from the released betas report2026-06-16and are not probed.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines