Description
DynamoDBValueSensor only supports blocking poll mode — it holds an Airflow worker slot for the entire wait duration and poke_interval determines how quickly it reacts after the target value is set.
Use case/motivation
A common Airflow pattern is using DynamoDB as a lightweight approval or job-state store — wait for a status field to flip to a target value (e.g. APPROVED, COMPLETED) before proceeding. The wait window can be hours for human approval workflows.
With the current blocking sensor:
- A worker slot is held for the entire wait window (hours)
- Reaction latency equals one full poke_interval — typically set long (e.g. 6000s) precisely to avoid burning worker slots, which makes latency worse
- This creates a forced tradeoff: short poke_interval wastes workers, long poke_interval means slow reaction
Deferrable mode breaks the tradeoff:
- Sensor defers immediately, freeing the worker slot
- The Triggerer process handles the async polling loop at low cost
- poke_interval can be set to 30–60 seconds without worker pressure
- Result: hours of dead worker time eliminated, reaction latency drops from O(hours) to O(seconds)
Proposed addition: DynamoDBValueTrigger (async, runs in Triggerer) + deferrable=True on DynamoDBValueSensor, following the same pattern as existing triggers in providers/amazon/src/airflow/providers/amazon/aws/triggers/.
Sensor interface (backwards compatible):
DynamoDBValueSensor(
...,
deferrable=True, # opt-in, default False
waiter_delay=30, # seconds between trigger polls
)
When deferrable=False (default), existing behaviour is completely unchanged.
Related issues
- Existing SqsSensorTrigger: providers/amazon/src/airflow/providers/amazon/aws/triggers/sqs.py
Are you willing to submit a PR?
Code of Conduct
Description
DynamoDBValueSensor only supports blocking poll mode — it holds an Airflow worker slot for the entire wait duration and poke_interval determines how quickly it reacts after the target value is set.
Use case/motivation
A common Airflow pattern is using DynamoDB as a lightweight approval or job-state store — wait for a status field to flip to a target value (e.g. APPROVED, COMPLETED) before proceeding. The wait window can be hours for human approval workflows.
With the current blocking sensor:
Deferrable mode breaks the tradeoff:
Proposed addition: DynamoDBValueTrigger (async, runs in Triggerer) + deferrable=True on DynamoDBValueSensor, following the same pattern as existing triggers in providers/amazon/src/airflow/providers/amazon/aws/triggers/.
Sensor interface (backwards compatible):
When deferrable=False (default), existing behaviour is completely unchanged.
Related issues
Are you willing to submit a PR?
Code of Conduct