Repository navigation
Language SDK runtimes do not handle execution_timeout in the task process #74717
Description
Under which category would you file this issue?
Task SDK
Apache Airflow version
main
What happened and how to reproduce it?
A Python task has two layers for execution_timeout. The task runner raises AirflowTaskTimeout inside the task process when the time is up, so on_kill, the retry policy and callbacks can run (task_runner.py). It also sends SetExecutionTimeout, so the supervisor kills the process if it gets stuck (#73806).
The Go, TypeScript and Java runtimes have neither layer. This is the same for native Dag tasks and @task.stub tasks, because both are started by the same coordinator and supervisor.
#74755 (for #74716) adds the outer layer. The server sends execution_timeout in TIRunContext, and the supervisor arms its deadline from it. If I read it right, this also covers native Dag tasks, because the Go, TypeScript and Java serializers write is_stub: true on their tasks too. But the task is then stopped with SIGTERM at execution_timeout + killed_task_cleanup_time (60s by default), counted from when the runtime starts. The task code gets no signal before that, so it cannot clean up.
Steps to reproduce (Go)
- Pack a native Go Dag with one task that sleeps 90 seconds, logs every 5 seconds, and has
ExecutionTimeout: 10 * time.Second. Put it in the Dags folder. - Add a Python task with
execution_timeout=timedelta(seconds=10)that also sleeps 90 seconds, as a control. - In
breeze shell --backend postgres, start the api-server, dag-processor and scheduler withLocalExecutor, using the same coordinator config as the Go SDK e2e test. Trigger both Dags.
Native Go Dag
func goSleep(actx airflow.Context) error {
start := time.Now()
for i := 1; i <= 18; i++ {
select {
case <-actx.Done():
actx.Logger().InfoContext(actx, "context cancelled", "elapsed_s", time.Since(start).Seconds())
return actx.Err()
case <-time.After(5 * time.Second):
actx.Logger().InfoContext(actx, "still running", "elapsed_s", time.Since(start).Seconds())
}
}
actx.Logger().InfoContext(actx, "finished after full sleep", "elapsed_s", time.Since(start).Seconds())
return nil
}
func main() {
bundle := airflow.Bundle()
dag := airflow.Dag("timeout_repro_native", airflow.DagSpec{Queue: "golang"})
dag.Task(goSleep, airflow.TaskSpec{TaskID: "go_sleep", ExecutionTimeout: 10 * time.Second})
bundle.Register(dag)
if err := bundle.Serve(); err != nil {
log.Fatal(err)
}
}Result
| task | serialized execution_timeout |
final state | ran for |
|---|---|---|---|
| Python control | 10.0 | failed, AirflowTaskTimeout |
about 13s |
native Go go_sleep |
10.0 | success | about 90s |
The Go task log has a still running line every 5 seconds up to elapsed_s=90.15, then finished after full sleep. It has no cancel or SIGTERM line. I only ran Go. The TypeScript and Java parts come from reading the code.
What you think should happen instead?
The runtime should handle the timeout itself, like the Python task runner. Each SDK would:
- Read
execution_timeoutfromti_context(added in Enforce execution_timeout on stub tasks run by a language SDK #74755). - Send
SetExecutionTimeoutright before it runs the task, so the supervisor deadline starts there and not at runtime start. - Cancel the task when the time is up, and fail it with its retries applied.
How to cancel is up to each SDK, for example airflow.Context in Go and an AbortSignal in TypeScript. I could not find a cancel signal for tasks in the Java SDK yet.
- Go
- TypeScript
- Java
Operating System
macOS 15 host, Breeze CI image (Python 3.11, Postgres 14)
Deployment
None
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
Java also needs to refresh its vendored supervisor schema (java-sdk/sdk/schema/schema.json) to get the new field.
The supervisor deadline from #74755 should stay. Released runtimes, and bundles built with an older SDK, do not send SetExecutionTimeout, so it is their fallback.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Andrushika commentedon Oct 14, 2026
Triage:
area:task-sdk,area:java-sdk,area:go-sdk,area:ts-sdkRishabhBajaj97 commentedon Oct 14, 2026
I'd like to take a stab at the Go SDK side.
Scoping to a focused first PR, I'd propose going with the "defensive exclude" path you hinted at in the issue body — add
execution_timeoutto the exclude list ingo-sdk/internal/genspec/authoring.go, next tois_setup/is_teardown/etc. that are already excluded for the same "SDK cannot yet honour" reason. The field disappears fromTaskSpec, so an author setting it gets a compile error instead of a task that silently ignores the timeout — matching the "worse than a missing field" principle already documented there.Properly implementing the runtime timer +
SetExecutionTimeoutdispatch is a larger structural change (the currentbundle.Taskinterface exposes onlyExecute(), so there's no path for the task runner to readExecutionTimeoutat execution time without plumbing a new field through the task-handler map and the interface). Happy to do that as a follow-up PR once this defensive exclude lands, or to combine both if reviewers prefer. The TypeScript and Java pieces are also out of this first PR's scope — tracked separately.If a maintainer disagrees with the "exclude first, implement later" direction, say the word and I'll switch to the full runtime implementation.
Drafted-by: Claude Code (Opus 4.7) (human reviewed)
[-]execution_timeout is not supported on native language SDK Dag tasks[/-][+]Language SDK runtimes do not handle execution_timeout in the task process[/+]Andrushika commentedon Oct 15, 2026
Thanks for looking at this. With #74755 the supervisor stops an overrunning task from outside, so I think we no longer need to exclude the field. I updated the issue to track the in-process part for each SDK, and the Go part is still open if you want to take it.
ChickenBenny commentedon Oct 16, 2026
Hi, I’d like to work on this~
ChickenBenny commentedon Oct 17, 2026
@RishabhBajaj97, are you still planning to work on the Go part? If you are,I'll stay on TypeScript and I'm happy to review your PR, since I worked on the supervisor side in #74869. If you don't have the time right now, let me know and I can take Go as well. Either way is fine with me.
RishabhBajaj97 commentedon Oct 20, 2026
Hi @ChickenBenny, I will pick this up this thursday, if you can do it prior to that, please go ahead!
ChickenBenny commentedon Oct 21, 2026
@RishabhBajaj97 a heads-up before you start: the design here has shifted since the issue was written, so it may be worth holding off on the timer part until the direction is settled. @Andrushika suggested in #74869 that the supervisor become the only clock and just notify each SDK when the timeout elapses, so the SDKs may not need their own timer at all. If you're interested in the background, the discussion in #74869 is a good place to start.