Skip to content

Adjust warnings, import errors, and Dag source models for sdk/importers - #72474

Merged
shahar1 merged 8 commits into
apache:mainfrom
dilnazanlid:aip-85-error-warning
Sep 28, 2026
Merged

shahar1 merged 8 commits into
apache:mainfrom
dilnazanlid:aip-85-error-warning

Conversation

@dilnazanlid

@dilnazanlid dilnazanlid commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Why

AIP-85 allows Dags to come from non-Python and composite sources (YAML/JSON importers,
ZipImporter, external bundles). Three pieces of metadata currently assume a single local .py file:

  1. Import errors only track filename, but a Dag definition is no longer strictly 1:1 with a file —
    an archive member inside a container (e.g. archive.zip/dags/my_dag.py) needs an abstract source
    reference while retaining filename for compatibility.
  2. DagWarning rejects any type outside the built-in enum, preventing importers from reporting
    format-specific warnings.
  3. Dag source is always highlighted as Python in the UI, regardless of the Dag's actual definition language.

This is groundwork for AIP-85: it introduces the necessary schema and model extensions without altering
runtime behavior for existing Python Dags.

What

Import errors: add source_reference, keep filename

  • Migration ca8499dc1004 adds a nullable source_reference column (String(2000)) to import_error.
  • Retains filename as the primary file path column across the database, model, and API to preserve
    full backward compatibility for existing tooling and API consumers.
  • Exposes source_reference: str | None on ImportErrorResponse and ParseImportError.
  • Implements ParseImportError.full_file_path() using Path.is_absolute() rather than substring prefix
    matching, avoiding false positives across sibling bundle paths.
  • Streamlines delete_dag cleanup to delete import errors matching source_reference or filename.

Dag warnings: open-ended types, enum retained

  • DagWarning.__init__ accepts DagWarningType | str, normalizing enum members to their string value
    and preserving importer-supplied warning types verbatim.
  • DAGWarningResponse.warning_type and the list_dag_warnings filter accept DagWarningType | str.
    This retains DagWarningType in the published schema and generated clients as the vocabulary for
    core-emitted warnings, while gracefully accepting importer-defined warnings.
  • Filtering by an unknown warning_type returns 200 with an empty list instead of 422.

Dag source language

  • Migration e5a91c7f42b3 adds a nullable language column (String(64)) to dag_code.
  • DAGSourceResponse and DagCode persist and expose language: str | None.
  • UI Code view passes code?.language ?? "python" to Monaco Editor and CodeDiffViewer, defaulting
    to Python when unstated.
Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Google Gemini following the guidelines

jason810496

This comment was marked as resolved.

@jason810496 jason810496 moved this from Todo to In Progress in AIP-85 Dag Importer Sep 9, 2026
@dilnazanlid
dilnazanlid force-pushed the aip-85-error-warning branch 4 times, most recently from d67d024 to 9007a4c Compare September 15, 2026 09:42

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update.

Comment thread airflow-core/src/airflow/models/errors.py Outdated
Comment thread airflow-core/src/airflow/api/common/delete_dag.py
Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/dag_warning.py Outdated

@jason810496 jason810496 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just head up that I will take over this PR and get this merge this week since GCP team is OOO this week.
Dilnaz continue on the work.

@dilnazanlid
dilnazanlid force-pushed the aip-85-error-warning branch 3 times, most recently from bc576b6 to 2ee8f50 Compare September 25, 2026 11:19
@dilnazanlid dilnazanlid changed the title [AIP-85] Adjust warnings, import errors, and DAG source models Adjust warnings, import errors, and DAG source models for sdk/importers Sep 25, 2026
@shahar1 shahar1 changed the title Adjust warnings, import errors, and DAG source models for sdk/importers Adjust warnings, import errors, and Dag source models for sdk/importers Sep 26, 2026
Comment thread airflow-core/src/airflow/models/errors.py
Comment thread airflow-core/src/airflow/models/dagcode.py
Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/import_error.py Outdated
Comment thread airflow-core/src/airflow/models/dagwarning.py Outdated
The API response and the Code view's syntax highlighting already expose a
language for each Dag's source, but nothing persisted it, so the field was
always null and the UI silently fell back to Python for every Dag.
Renaming filename to source_reference might break compatibility across the
database schema, model attributes, and public REST API endpoints that
rely on filename to refer to concrete files.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@shahar1
shahar1 merged commit 30d0d7b into apache:main Sep 28, 2026
156 checks passed

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:dag-processor area:db-migrations PRs with DB migration area:UI Related to UI/UX. For Frontend Developers. kind:documentation

Projects

Development

Successfully merging this pull request may close these issues.

5 participants