Repository navigation
Run a DAG from command line #18
Description
Is there a way to run a DAG from command line?
I see two options:
- Use
backfilland give same start and end date. - Use
run- possible only iftask_idis optional. Currently,runseems to support only running a specific task within a specific DAG. Is it possible to maketask_idoptional, so that if it's not provide, theruncommand will run the full DAG?
Is there any other way of doing this that I may have missed in the documentation?
Activity
artwr commentedon Jun 11, 2015
I believe
backfillis the canonical way,runbeing indeed reserved for a particular task in a dag. Note that to save a few keystrokes, you can only provide one of (start_date, end_date) to thebackfillcommand and the other will be set to the same date.dirkgently007 commentedon Jun 11, 2015
Thanks @artwr . That helps!
mistercrunch commentedon Jun 11, 2015
The scheduler is the way to run your day to day processes, and
backfillis the way to go if you need to backfill or reprocess parts of the DAG in time. Note thatbackfillcan be used after aclearcommand to fill in the holes created by clear.Say if you want to reprocess a specific task for a date range (that had succeeded but was some sort of false positive, or the business logic has changed) and all of the tasks downstream from this task, you probably want to start with a
clearcommand that clears what needs to be reprocessed (here a set of tasks and everything downstream from it), and then run a backfill to fill in the holes in the right order.Note that if you don't specify
--end_date(-e) in the backfill command it will just run what you specified instart_date.dirkgently007 commentedon Jun 22, 2015
Thanks @mistercrunch - that makes sense.
version update for 1.10.4+twtr_rc1 tag (apache#18)
Fixed an orphan session issue in the handle_callback (apache#18)
OpenLineage java model Generator (apache#18)
parent b22b1a6
parent b22b1a6
Merge pull request apache#18 from nestjs/renovate/nest-monorepo