Repository navigation
Add an API endpoint and UI tab comparing two stored Dag versions - #73322
Merged
Merged
Conversation
boring-cyborg
Bot
added
area:airflow-ctl
area:API
Contributor
Author
Screen.Recording.2026-09-18.at.09.46.58.mov |
ephraimbuddy
force-pushed
the
dag-version-diff-api-ui
branch
from
September 25, 2026 08:10
abe488a to
13feeac
Compare
ephraimbuddy
marked this pull request as ready for review
September 25, 2026 08:14
ephraimbuddy
requested review from
ashb,
bbovenzi,
bolkedebruin,
bugraoz93,
choo121600,
dheerajturaga,
guan404ming,
henry3260,
jason810496,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git,
vatsrahul1001 and
vincbeck
as code owners
September 25, 2026 08:14
pierrejeambrun
left a comment
pierrejeambrun
left a comment
Member
There was a problem hiding this comment.
Nice LGTM overall.
A few suggestions / comments.
Do you have more screenshot with 'value showns' and or with 'observed state' enabled ?
ephraimbuddy
force-pushed
the
dag-version-diff-api-ui
branch
from
September 25, 2026 19:32
c6fa693 to
d36a851
Compare
Contributor
Author
|
Values Shown: 
Values Hidden: Boundary tooltip Column Menu version picker: |
Contributor
|
Just need to rebase and regenerate docs |
ephraimbuddy
force-pushed
the
dag-version-diff-api-ui
branch
from
October 4, 2026 08:59
d36a851 to
1de4c24
Compare
Comparing two versions of a Dag meant reading serialized payloads out of the database by hand. This exposes the diff engine over the API so a reviewer can ask what differs between the versions a run was pinned to. The disclosure decision belongs to the server rather than the caller: a reader of a Dag's versions gets the changed structure with identifying components masked, and only a reader of its code gets the paths that name them and the values behind them, since the serialized payload is returned unmasked. The change count needs saying out loud on the wire, because it counts underlying changes rather than records and a dropped path takes its occurrences with it, so it is exact only while nothing was dropped. Source is deliberately absent: comparing code is already served by the source endpoint and the Dag code view.
Nothing could call the diff endpoint in a typed way: the OpenAPI spec and the generated TypeScript and airflowctl clients had no knowledge of it. Its five closed value sets needed naming rather than inlining. A field annotated with a bare literal union makes the generators name the type after the property, so "category" and "impact" became type names and displaced the warning category that already held one of them, renaming it to category2 for no reason. Names as generic as "mode" and "operation" would have been next. Declaring enums gives the generators names of their own to use. Two of the sets restate values the engine and the model own, so a test pins them to both.
Seeing what changed between the versions a run was pinned to meant calling the API by hand or reading serialized payloads out of the database. This puts the comparison where someone investigating a run already is. Whether values are shown is reported rather than offered, together with what would grant them: the server decides from the caller's access to Dag code, so a control here would imply the viewer could change it, and saying only that they are hidden leaves them no way to find out why. A value too long to read in a table cell is truncated, because a task added or removed carries its whole serialized payload. The reason a comparison could not be made is rendered as its own token rather than folded into a sentence, since it is a machine value that translators would rewrite. Source comparison stays in the Code tab, which already diffs code; this tab says so rather than offering a second answer that could disagree with it.
The Dag code view owned this picker while it was the only thing comparing versions. A second page now needs the same one, and reaching into another page's folder for it leaves the component with two consumers and no owner: whoever edits the code view can break the other page without seeing it. Duplicating it instead would leave two pickers over the same version list to drift apart.
The chosen versions lived in component state, so a comparison could not be linked, bookmarked or survive a reload. The record limit was fixed at the server default, leaving no way to see more once a comparison was truncated.
A change record carries a value key only when that side has a value, which is how the engine tells a side that does not exist apart from one holding null. Serializing unset fields flattened the two to null, leaving the UI able to report only "null" where there is no value at all. The version pickers had also come to rely on an untranslated placeholder default, and the airflow-ctl datamodels had not been regenerated since the endpoint's schemas entered the spec, which would fail the generation check.
The comparison told small untruths about itself: a one-change comparison read "1 changes", the operation, category and impact cells showed raw API tokens instead of labels, two identical versions produced a bare table header, and a stored empty string looked like a missing value. Typing a bound fired one comparison per keystroke, each re-reading and diffing two whole serialized Dags, and wrote unvalidated text to the URL — so a shared link could carry a bound the endpoint rejects while the field showed another. The value and digest fields now publish the absent-vs-null rule the endpoint relies on, and the security model lists this comparison alongside source retrieval as something Dag code access unlocks.
ephraimbuddy
force-pushed
the
dag-version-diff-api-ui
branch
from
October 4, 2026 13:58
1de4c24 to
ef02857
Compare
Contributor
Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker b58cfa9 airflow-ctl/v0-1-testThis should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the API and UI layer on top of the serialized Dag version diff engine: an
endpoint that compares two stored versions of a Dag, and a Versions tab that
renders the comparison.
The endpoint reports observed state — what the two stored versions currently
hold — rather than why a version was created. Disclosure is split in two:
DagAccessEntity.VERSIONread returns which parts of the Dag differ withidentifying path components masked, while the values behind those differences,
and the paths naming them, are returned only to a caller who may also read the
Dag's code. A record carries a value key only when that side has one, which is
how an absent side stays distinguishable from a stored null.
The tab keeps both chosen versions and the change bound in the URL, so a
comparison can be linked and reopened.
The diff engine this builds on merged in #69864, and this branch has been
rebased onto it, so the diff here is now this PR's changes alone.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines