Currently, Alternator deletes items in tables without a clustering key using partition tombstones. But CDC doesn't fetch a preimage if the mutation contains only a partition key, which results in an empty OldImage, even if the OldImage would be present for a table with both a pk and a ck.
A test that reproduces this:
def test_streams_no_ck(test_table_s_no_ck_new_and_old_images, dynamodb, dynamodbstreams):
def do_updates(table, p, _):
events = []
table.put_item(Item={'p': p, 'x': 2})
events.append(['INSERT', {'p': p}, None, {'p': p, 'x': 2}])
table.delete_item(Key={'p': p})
events.append(['REMOVE', {'p': p}, {'p': p, 'x': 2}, None])
return events
do_test(test_table_s_no_ck_new_and_old_images, dynamodb, dynamodbstreams, do_updates, 'NEW_AND_OLD_IMAGES')fails with:
else:
new_image = {x:deserializer.deserialize(y) for (x,y) in record['NewImage'].items()}
assert expected_new_image == new_image
if expected_old_image == None:
assert not 'OldImage' in record
else:
> old_image = {x:deserializer.deserialize(y) for (x,y) in record['OldImage'].items()}
E KeyError: 'OldImage'
test/alternator/test_streams.py:766: KeyError
PK and CK case (passes):
def test_streams_ck(test_table_ss_new_and_old_images, dynamodb, dynamodbstreams):
def do_updates(table, p, c):
events = []
table.put_item(Item={'p': p, 'c': c, 'x': 2})
events.append(['INSERT', {'p': p, 'c': c}, None, {'p': p, 'c': c, 'x': 2}])
table.delete_item(Key={'p': p, 'c': c})
events.append(['REMOVE', {'p': p, 'c': c}, {'p': p, 'c': c, 'x': 2}, None])
return events
do_test(test_table_ss_new_and_old_images, dynamodb, dynamodbstreams, do_updates, 'NEW_AND_OLD_IMAGES')This discrepancy between partition tombstones and row tombstones should be preserved if the table has a clustering key (i.e. deleting a partition shouldn't result in a new REMOVE event for each CK) - we only need to handle the pk+no_ck case.
Currently, Alternator deletes items in tables without a clustering key using partition tombstones. But CDC doesn't fetch a preimage if the mutation contains only a partition key, which results in an empty OldImage, even if the OldImage would be present for a table with both a pk and a ck.
A test that reproduces this:
fails with:
PK and CK case (passes):
This discrepancy between partition tombstones and row tombstones should be preserved if the table has a clustering key (i.e. deleting a partition shouldn't result in a new REMOVE event for each CK) - we only need to handle the pk+no_ck case.