You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the DatasetDagRunQueue table is one record per target_dag_id+dataset_id.
This is because we don't care if a dataset is updated multiple times in between triggered dag runs (e.g. while waiting on another dataset to be updated -- once is enough information.
But a get-or-create operation like this is potentially sucsceptible to an integrity error if another process commits the queue record after "you" do get but before commit the "add".
Whether we should do anything about this i'm not sure. Since the queue records are committed almost immediately after they are added, the likelihood of this happening has got to be very small. You'd have to have two TIs that update the same dataset, which ran at basically the same time, and then you have to have T1 get, t2 create, t1 create, t2 commit, t1 commit. Possible, but not likely. Not sure what our tolerance is for very unlikely errosr like this -- the consequence is the transaction marking TI as success would fail.
If we want to do something about it, we either need to split TI "success" and queue-table-write into separate commits, or we need to loosen the grain of the queue table so that we can always add more queue records even when more won't make a difference. Or, thirdly, I guess we could try and figure out a way to do a safe insert-or-update.
Ash initially specified that success + queue should be in same transaction, so unless there's a good reason we should prioritize the other two options.
Activity
[-]Handle potential write conflict in queue table[/-][+]Maybe handle potential write conflict in queue table?[/+][-]Maybe handle potential write conflict in queue table?[/-][+]Maybe handle potential [unlikely] write conflict in queue table?[/+][-]Maybe handle potential [unlikely] write conflict in queue table?[/-][+][Won't do] Maybe handle potential [unlikely] write conflict in queue table?[/+]