Skip to content

Add AssetAndTimeSchedule timetable - #58543

Merged
Lee-W merged 15 commits into
apache:mainfrom
aaron-y-chen:feat-58056-asset_and_time_schedule
Oct 1, 2026
Merged

Lee-W merged 15 commits into
apache:mainfrom
aaron-y-chen:feat-58056-asset_and_time_schedule

Conversation

@aaron-y-chen

@aaron-y-chen aaron-y-chen commented Nov 20, 2025 •

Copy link
Copy Markdown
Contributor

Closes: #58056

Why

Use case: #58056

How

1. Add AssetAndTimeSchedule

Add a timetable that combines a time-based timetable with an asset condition.

It behaves like a scheduled timetable, not an asset-triggered timetable: when it creates a run, the run type is still SCHEDULED.

2. Gate DagRun creation before creating the DagRun

The scheduler creates a scheduled DagRun only when both conditions are satisfied: the schedule time is due and the required asset condition is ready.

If assets are missing, no DagRun is created and next_dagrun_create_after stays on the pending slot, so the same slot can be retried later without creating placeholder DagRuns.

3. Keep asset-aware scheduler behavior consistent

Timetables expose asset-aware scheduling through asset_triggered and asset_gated behavior flags, following the existing periodic pattern. These flags are also mirrored on the Task SDK base timetable so custom timetables can opt in without scheduler type checks.

DagModel.dags_needing_dagruns uses these flags to route asset-triggered Dags separately from asset-gated scheduled Dags. Asset-gated Dags still flow through the standard scheduled-run creation path (_create_dag_runs).

When DagModel.timetable_asset_gated is set, the scheduler re-checks the timetable's asset_condition under ADRQ row locks before creating the scheduled DagRun. If the condition is still satisfied, the created run links and consumes the selected asset events. The ADRQ locking, event-provenance, and consumption helpers are shared with the asset-triggered path.

What

Here are my test DAGs:

upstream_asset_producer

import pendulum
from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator
from airflow.sdk.definitions.asset import Asset

my_asset = Asset("/my/example/asset")

with DAG(
    dag_id="upstream_asset_producer",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    schedule="0 * * * *",
    catchup=False,
    tags=["example", "upstream"],
) as upstream_dag:
    BashOperator(
        task_id="produce_asset",
        bash_command="echo 'Asset produced!'",
        outlets=[my_asset],
    )

downstream_asset_and_time_consumer

import pendulum
from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator
from airflow.sdk.definitions.asset import Asset
from airflow.timetables.trigger import CronTriggerTimetable
from airflow.timetables.assets import AssetAndTimeSchedule

my_asset = Asset("/my/example/asset")

with DAG(
    dag_id="downstream_asset_and_time_consumer",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    schedule=AssetAndTimeSchedule(
        timetable=CronTriggerTimetable("4 * * * *", timezone="UTC"),
        assets=[my_asset],
    ),
    catchup=False,
    max_active_runs=1,
    tags=["example", "downstream", "asset-and-time"],
) as downstream_dag:
    BashOperator(
        task_id="consume_asset",
        bash_command="echo 'Asset condition and time condition were both met!'",
    )

From the screenshots, the downstream DAG runs as expected: it runs only after both the timetable and the assets are ready.

  1. Upstream dag

    AssetAndTimeSchedule_example_1_upstream

  2. Asset production time

    AssetAndTimeSchedule_example_2_asset

  3. Downstream, AssetAndTimeSchedule

    AssetAndTimeSchedule_example_3_downstream


Important

🛠️ Maintainer triage note for @nailo2c · by @potiuk · 2026-06-22 06:31 UTC

Your review threads from @jscheffl look addressed — please confirm this PR is ready for maintainer review confirmation:

  • 1 thread(s) show your engagement (post-review commits and/or replies).

The ball is in your court — you've been assigned to this PR. Reply yes / ready (and mark the threads resolved) and a maintainer will pick it up from the queue.

Automated triage — may be imperfect; a maintainer takes the next look.

@aaron-y-chen
aaron-y-chen requested a review from Lee-W as a code owner March 22, 2026 04:49
@aaron-y-chen
aaron-y-chen force-pushed the feat-58056-asset_and_time_schedule branch from c752a20 to 760844c Compare March 22, 2026 05:01
Comment thread airflow-core/src/airflow/timetables/assets.py
Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py Outdated
Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py Outdated
Comment thread airflow-core/src/airflow/models/dag.py
@eladkal eladkal changed the title feat-58056: Implement AssetAndTimeSchedule Add AssetAndTimeSchedule Mar 24, 2026
@eladkal eladkal added this to the Airflow 3.3.0 milestone Mar 24, 2026
@eladkal eladkal added kind:feature Feature Requests type:new-feature Changelog: New Features and removed kind:feature Feature Requests labels Mar 24, 2026
@eladkal eladkal changed the title Add AssetAndTimeSchedule Add AssetAndTimeSchedule timetable Mar 24, 2026
Comment thread task-sdk/src/airflow/sdk/definitions/timetables/assets.py Outdated
Comment thread airflow-core/src/airflow/timetables/assets.py Outdated
Comment thread airflow-core/src/airflow/timetables/assets.py Outdated
@uranusjr

uranusjr commented Aug 6, 2026

Copy link
Copy Markdown
Member

We can probably do some refactoring to extract some common logic between the And and Or timetables into a mixin. This can be a different PR.

@aaron-y-chen

Copy link
Copy Markdown
Contributor Author

Hi @uranusjr, I'd appreciate it if you could take another look when you have time.

@shahar1 shahar1 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review

LGTM — the asset-gated scheduling logic (no-placeholder-DagRun retry, ADRQ row-locked re-check, Task SDK flag mirroring without DB leakage, migration symmetry) is correct and well covered by tests across Core and the Task SDK. CI is green. Two minor nits inline, neither blocking.

Smaller observations

  • airflow-core/src/airflow/jobs/scheduler_job_runner.py:2832 — the new shared _lock_queued_asset_records helper unconditionally adds .options(joinedload(AssetDagRunQueue.asset)). The pre-existing _create_dag_runs_asset_triggered call site (which used to inline this query without the join) never reads .asset, so it now pays for an extra JOIN/hydration on every asset-triggered scheduler pass for no benefit. Not a correctness issue, just a small efficiency regression from the refactor — worth gating the eager load behind a parameter if it matters at scale.
  • airflow-core/tests/unit/timetables/test_assets_timetable.py:398,421-422,450-451 — test_infer_manual_data_interval_and, test_next_dagrun_info_and, and test_generate_run_id_and use DateTime.now() instead of time_machine (this repo's testing standard: "Use time_machine for time-dependent tests. Do not use datetime.now()"). This mirrors the existing sibling AssetOrTimeSchedule tests in the same file rather than introducing a new deviation, and the assertions are structural (isinstance(...)) so it isn't flaky today — but new code shouldn't reproduce the pattern.

This review was drafted by an AI-assisted tool and confirmed by an Airflow maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.

Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment thread airflow-core/tests/unit/timetables/test_assets_timetable.py
@ashb

ashb commented Sep 29, 2026

Copy link
Copy Markdown
Member

@shahar1 I'd like @Lee-W to review this before we merge please.

@Lee-W

Lee-W commented Sep 29, 2026

Copy link
Copy Markdown
Member

I will review it early tomorrow. Thanks @ashb for the reminder and @aaron-y-chen for the ongoing effort!

Comment thread airflow-core/src/airflow/timetables/simple.py

@Lee-W Lee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major. I'm good with merging it.

Comment thread airflow-core/src/airflow/models/dag.py Outdated
Comment thread airflow-core/src/airflow/timetables/assets.py Outdated
Comment thread airflow-core/tests/unit/timetables/test_assets_timetable.py
@Lee-W

Lee-W commented Sep 30, 2026

Copy link
Copy Markdown
Member

also need to resolve conflcit

aaron-y-chen and others added 15 commits September 30, 2026 07:48
Add a new built-in timetable that combines a time-based schedule with
an asset condition: a SCHEDULED DagRun is created only when both the
timetable's scheduled time has arrived and every required asset has
queued an event. When the run is created, those asset events are
consumed so the next scheduled run waits for fresh updates. Unlike
AssetOrTimeSchedule, this does not create asset-triggered runs.

Implementation outline:
- airflow.timetables.assets.AssetAndTimeSchedule (core) and
  airflow.sdk.definitions.timetables.assets.AssetAndTimeSchedule
  (Task SDK shim) delegate next_dagrun_info / infer_manual_data_interval
  / generate_run_id to the wrapped timetable, retaining the
  asset_condition that gates creation.
- DagModel.dags_needing_dagruns returns a third bucket
  asset_gated_ready_dag_ids of dags whose asset condition is already
  satisfied; the SQL query excludes AssetAndTimeSchedule dags whose
  assets are not yet ready, preventing them from saturating the
  max_dagruns_to_create_per_loop batch and starving pure time dags.
- _create_dagruns_for_dags dispatches the new bucket to
  _create_dag_runs_asset_gated. The method re-locks ADRQ rows with
  skip_locked, re-evaluates the asset condition as a HA race guard,
  creates a SCHEDULED DagRun, links the consumed AssetEvent rows via
  dag_run.consumed_asset_events.extend(...), updates active_runs and
  exceeds_max_non_backfill so a follow-up asset event cannot bypass
  max_active_runs, and finally deletes only the consumed ADRQ rows.
- _start_queued_dagruns is untouched: asset gating happens at DagRun
  creation time, not after. There is no placeholder QUEUED DagRun
  waiting for assets and no dagrun_timeout-based FAILED path, both of
  which would have violated the DagRun state-model invariants
  documented in airflow-core/docs/core-concepts/dag-run.rst.

Tests:
- AssetAndTimeSchedule serialization/deserialization, data-interval
  inference, next_dagrun_info delegation, and generate_run_id.
- Scheduler-level: time-not-ready, assets-not-ready, both-ready,
  oldest-pending-slot semantics on late asset arrival, only-consumes-one-slot
  when multiple schedule slots are pending, ADRQ row locking race guard,
  max_active_runs respected after a queued run is created, AssetEvent
  rows linked via consumed_asset_events, and starvation prevention
  showing the new SQL filter keeps gated dags from starving pure time
  dags when assets are missing.
- DagModel.dags_needing_dagruns regression coverage for the new
  third-bucket return value.

closes: apache#58056
Document the new gating model: a scheduled DagRun is created only
when both the timetable's scheduled time has arrived and every
required asset has queued an event. If the scheduled time arrives
before assets are ready, no DagRun is created; the scheduler holds
the oldest pending slot and re-checks each loop until assets arrive.

closes: apache#58056
Review feedback asked for the scheduler to handle timetables through
behavioral contracts instead of special-cased code paths. The dedicated
asset-gated creation path duplicated both the scheduled-run creation
logic and the asset-triggered ADRQ/event mechanics, and could drift
from either. Asset gating is now a generic step of the normal
scheduled-run path, driven only by DagModel.timetable_asset_gated and
Timetable.asset_condition, with the ADRQ locking, event-provenance and
consumption helpers shared with the asset-triggered path.
The previous tests could pass when schedule calculation or run ID delegation returned the wrong value, and wall-clock inputs made the results harder to reason about.
Co-authored-by: Wei Lee <hello@wei-lee.me>
The validation error now starts with an uppercase letter, while the expected regex still used lowercase. This mismatch caused all four nested timetable cases to fail in CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduler ready for maintainer review Set after triaging when all criteria pass. type:new-feature Changelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add AssetAndTimeSchedule to schedule DAGs

9 participants