Skip to content

Java SDK: Declare a Dag's task graph with @Builder.Deps - #71189

Merged
jason810496 merged 13 commits into
apache:mainfrom
jason810496:feature/java-sdk-native-dag
Oct 6, 2026
Merged

jason810496 merged 13 commits into
apache:mainfrom
jason810496:feature/java-sdk-native-dag

Conversation

@jason810496

@jason810496 jason810496 commented Aug 5, 2026 •

Copy link
Copy Markdown
Member

Merge order:

  1. Java SDK: Declare a Dag's task graph with @Builder.Deps #71189 — Declare a Dag's task graph with @Builder.Deps (current one)
  2. Java SDK: Group a native Dag's tasks with task groups #74230 — Group a native Dag's tasks with task groups
  3. Java SDK: Serialize native Dags to DagSerialization v3 #71190 — Serialize native Dags to DagSerialization v3
  4. Java SDK: Pack each native Dag's source file into the bundle JAR #74096 — Pack each native Dag's source file into the bundle JAR

Why

Support the TaskFlow syntax to define the graph for native Dag with annotation syntax, follow the user interface design in ADR-0002.

Example

@Builder.Dag(id = "java_etl", schedule = "@daily")
public class EtlPipeline {

  @Builder.Task(id = "extract", retries = 2)
  public long extract() { return 42L; }

  @Builder.Task(id = "transform")
  public long transform(long extracted, double factor) { return (long) (extracted * factor); }

  @Builder.Task(id = "audit")
  public void audit() { /* side effect only */ }

  @Builder.Deps
  static class Wiring implements EtlPipelineDeps {
    void depends() {
      var rows = extract();
      load(transform(rows, lit(0.9)));
      rows.before(audit()); // ordering-only edge
    }
  }
}

The processor generates the interface that wiring class implements:

public interface EtlPipelineDeps extends Deps {
  default TaskRef<Long> extract() {
    return Refs.node(new TaskDef("extract", EtlPipelineBuilder.Extract.class).config("retries", 2));
  }

  default TaskRef<Long> transform(Arg<? extends Number> extracted, Arg<? extends Number> factor) {
    return Refs.call(new TaskDef("transform", EtlPipelineBuilder.Transform.class), extracted, factor);
  }
}

How

  • A view, not a twin object. The Dag class holds long extract(Client) while the view holds TaskRef<Long> extract() — two methods that would clash in one class. The wiring class implements the view, so the Dag class keeps its inheritance slot and, being static, cannot reach the real task methods.
  • Read by recording, not by parsing. depends() runs once at registration with the Dag in scope. Nothing reads a syntax tree, so this works under any compiler rather than only javac.
  • Passing a handle into another call wires a data edge and feeds the upstream's XCom into that parameter. @Builder.Task method must be called in the wiring class; a task the wiring missed fails at Dag-parse time.
  • Bundle.register(Class) now also accepts a @Builder.Dag class and registers the Dag its generated builder builds.
  • The task-level .config(...) assertion in generateBuilderLowersConfigAttributes moves from the builder golden into the wiring-view golden, because that is where the TaskDef is now constructed. It is relocated coverage, not lost coverage.

Was generative AI tooling used to co-author this PR?
  • Yes, with help of Claude Code Opus 5 and Codex (GPT-5) following the guidelines

@phanikumv phanikumv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this PR seems way too huge, is there a way to create smaller PRs so that it is easier to review?

@jason810496

Copy link
Copy Markdown
Member Author

this PR seems way too huge, is there a way to create smaller PRs so that it is easier to review?

This PR depends on the other three PRs get merged first ( #69757, #71057, #71188) and contains the changes of all three as well. I just add the "Diff for early review: " now.

@jason810496 jason810496 self-assigned this Aug 6, 2026
@jason810496
jason810496 force-pushed the feature/java-sdk-native-dag branch 3 times, most recently from 22dca7c to c698338 Compare August 14, 2026 06:44
@jason810496
jason810496 force-pushed the feature/java-sdk-native-dag branch from c698338 to 68fd401 Compare August 21, 2026 07:30
@jason810496
jason810496 force-pushed the feature/java-sdk-native-dag branch 6 times, most recently from 6d1d542 to a6f69ba Compare September 21, 2026 08:37
@jason810496
jason810496 marked this pull request as ready for review October 3, 2026 13:48
@jason810496
jason810496 requested a review from uranusjr as a code owner October 3, 2026 13:48

@FrankYang0529 FrankYang0529 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Leave one question.

Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice just one nit found by claude

RESERVED_VIEW_NAMES includes notify, which the ADR's own example uses: @Builder.Task(id = "notify") public void notify(Client client) at ADR-0002 lines 67-68. The view strips the Client parameter, so it would generate default TaskRef notify(), which cannot override Object.notify(). Rejecting it is right, but the ADR example needs renaming, it appears at lines 67, 80, 104, 132 and 135.

Comment thread java-sdk/sdk/build.gradle.kts Outdated

@jason810496 jason810496 left a comment •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review folks!

Comment thread java-sdk/sdk/build.gradle.kts Outdated
Comment thread java-sdk/adr/0002-native-dag-interface.md
@jason810496
jason810496 merged commit ab4da0b into apache:main Oct 6, 2026
100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants