Skip to content

Add metrics for how often a dag gets serialised. - #68906

Merged
potiuk merged 1 commit into
apache:mainfrom
Ei-Sandi:dag-serialisation-metrics
Jul 30, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
Ei-Sandi:dag-serialisation-metrics

Conversation

@Ei-Sandi

@Ei-Sandi Ei-Sandi commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Adds a new dag.serialization counter metric that is emitted every time a Dag is serialized and written to the metadata DB.
  • The metric is tagged with dag_id and bundle_name, so serialization frequency can be tracked per Dag and per bundle.

(This gives operators visibility into how often DAGs are being re-serialized, to ensure we aren't unnecessarily re-serializing DAGs. )

Tests

  • Covered by unit tests.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    Co-authored-by: Claude Sonnet 4.6, Claude Opus 4.8 following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Ei-Sandi
Ei-Sandi marked this pull request as ready for review June 23, 2026 16:35

@xBis7 xBis7 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good! I have a few suggestions for the tests. Thanks!

BTW, I've tested it manually with breeze and metrics work as expected.

Comment thread airflow-core/tests/unit/models/test_serialized_dag.py Outdated
Comment thread airflow-core/tests/unit/models/test_serialized_dag.py
@Ei-Sandi
Ei-Sandi force-pushed the dag-serialisation-metrics branch from 27b6bfa to 88a6071 Compare June 23, 2026 17:56
@Ei-Sandi

Copy link
Copy Markdown
Contributor Author

@xBis7 The comments has been addressed. Can you please review it again ? Thank you.

Comment thread airflow-core/tests/unit/models/test_serialized_dag.py Outdated
Comment thread airflow-core/tests/unit/models/test_serialized_dag.py Outdated
@Ei-Sandi
Ei-Sandi force-pushed the dag-serialisation-metrics branch from 88a6071 to 31f4624 Compare June 24, 2026 12:15

@xBis7 xBis7 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Ei-Sandi
Ei-Sandi force-pushed the dag-serialisation-metrics branch from 31f4624 to 5aa50eb Compare June 24, 2026 14:24
@potiuk potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jun 25, 2026
@Ei-Sandi
Ei-Sandi force-pushed the dag-serialisation-metrics branch from 5aa50eb to 7e3d88e Compare July 15, 2026 14:28
@eladkal eladkal added this to the Airflow 3.3.1 milestone Jul 15, 2026

@Miretpl Miretpl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know this part of the codebase - looks good. I will wait a couple of days with the merge for any other possible reviews.

@vatsrahul1001

Copy link
Copy Markdown
Contributor

This looks more like a new feature and more suitable for minor 3.4.0

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one tiny suggestion.

Comment thread airflow-core/src/airflow/models/serialized_dag.py Outdated
@Ei-Sandi
Ei-Sandi force-pushed the dag-serialisation-metrics branch from 7e3d88e to 3f9506f Compare July 29, 2026 21:21
@potiuk
potiuk merged commit ba105ea into apache:main Jul 30, 2026
78 checks passed

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed this before merging and want one thing resolved first — the metric is sound but its two emission sites don't mean the same thing.

Things I checked and am happy with:

  • The two stats.incr calls sit before mutually exclusive return True paths in write_dag, so there's no double-counting, and the early return False (min_update_interval) correctly doesn't emit.
  • name_variables: ["dag_id", "bundle_name"] with a matching legacy_name accurately describes what's emitted. Worth calling out because #69078 declares name_variables: [] while tagging with two variables — yours is the correct pattern.

The nit is inline. Not a big change, but I'd rather it went in right than be corrected after people start alerting on it.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Comment thread airflow-core/src/airflow/models/serialized_dag.py
@Ei-Sandi
Ei-Sandi deleted the dag-serialisation-metrics branch July 31, 2026 14:03
@Ei-Sandi Ei-Sandi mentioned this pull request Jul 31, 2026
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dag-processor ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants