Skip to content

[Won't do] Maybe handle potential [unlikely] write conflict in queue table? #24974

Closed as not planned

Description

@dstandish

Currently the DatasetDagRunQueue table is one record per target_dag_id+dataset_id.
This is because we don't care if a dataset is updated multiple times in between triggered dag runs (e.g. while waiting on another dataset to be updated -- once is enough information.

But a get-or-create operation like this is potentially sucsceptible to an integrity error if another process commits the queue record after "you" do get but before commit the "add".

Whether we should do anything about this i'm not sure. Since the queue records are committed almost immediately after they are added, the likelihood of this happening has got to be very small. You'd have to have two TIs that update the same dataset, which ran at basically the same time, and then you have to have T1 get, t2 create, t1 create, t2 commit, t1 commit. Possible, but not likely. Not sure what our tolerance is for very unlikely errosr like this -- the consequence is the transaction marking TI as success would fail.

If we want to do something about it, we either need to split TI "success" and queue-table-write into separate commits, or we need to loosen the grain of the queue table so that we can always add more queue records even when more won't make a difference. Or, thirdly, I guess we could try and figure out a way to do a safe insert-or-update.

Ash initially specified that success + queue should be in same transaction, so unless there's a good reason we should prioritize the other two options.

Activity

  1. changed the title [-]Handle potential write conflict in queue table[/-] [+]Maybe handle potential write conflict in queue table?[/+] on Jul 11, 2022
  2. changed the title [-]Maybe handle potential write conflict in queue table?[/-] [+]Maybe handle potential [unlikely] write conflict in queue table?[/+] on Jul 11, 2022
  3. changed the title [-]Maybe handle potential [unlikely] write conflict in queue table?[/-] [+][Won't do] Maybe handle potential [unlikely] write conflict in queue table?[/+] on Jul 12, 2022
  4. added
    area:data-aware-schedulingassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitions
    and removed on Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:data-aware-schedulingassets, datasets, AIP-48, AIP-76, AIP-74, Asset Partitions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions