-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Closed
Description
I have a flow that is linear and sequential as shown below :
- new data arrives in SQS
- wait for that data to start being written to postgres
- wait for the SQS queue to drain
- send an email that the flow completed
Eventually, we will have a pre-cursor step (e.g. step 0.) that kicks off a Spark job before waiting for new data in the SQS queue. For now, I do not have that, so I would like my flow to simply reset to the first step. In some ways, I am asking for a circular flow that never ends. This breaks the entire workflow abstraction. On another note, where is the "cron" functionality? Do I schedule airflow commands in cron outside of airflow?
Metadata
Metadata
Assignees
Labels
No labels