Skip to content

Alternator Streams does does not properly distinguish the type of change event #6918

@nyh

Description

@nyh

https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_Record.html#DDB-Type-streams_Record-eventName explains that DynamoDB assigns to each change record one of three types:

  1. INSERT - a new item was added to the table.
  2. MODIFY - one or more of an existing item's attributes were modified.
  3. REMOVE - the item was deleted from the table

This description has some interesting implications:

  • Both PutItem or UpdateItem can be reported as either MODIFY or INSERT, depending on whether the item previously existed.
  • Moreover, in both PutItem and UpdateItem, if the pre-existing item is identical to the new item, no change event is produced at all!
  • DeleteItem of a non-existant item doesn't produce any change event at all.

Alternator does not currently implement any of these finer distinctions. All updates will be marked MODIFY, all deletes (whether or not an item existed) marked as REMOVE. This is a known issue (there is a comment about it in executor::get_records).

xfailing test/alternator test cases:

  • test_streams.py::test_streams_updateitem_keys_only
  • test_streams.py::test_streams_1_keys_only
  • test_streams.py::test_streams_1_new_image
  • test_streams.py::test_streams_1_old_image
  • test_streams.py::test_streams_1_new_and_old_images

Implementing these distinctions in CDC will require it to do a read-before-write. But there are two serious problems to consider:

  1. We also need these distinctions in the KEYS_ONLY case, where CDC doesn't do a read-before-write.
  2. Alternator already does it's own read-before-write, and does it more correctly (with LWT). How can we get CDC to use that read for these decisions (as well as the pre/post-image), and not do its own read which might be less consistent?

CC @elcallio

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions