-
Notifications
You must be signed in to change notification settings - Fork 0
Mismatch detector fixes #121
Description
- use Logging instead on system.out
- log row keys, values (they might be binary and large, check if Bigtable doesn't have any code for converting to binary and pruning long keys - we should do it)
- total number of entries read using next() call should be sent as a telemetry (not sure if it fits into Mismatch detector, maybe you can find a better place).
- Better scanner's next() verification.
- remove startingIndex from next() verificaiton, context and other places.
Creating a separate PR for each of those changes looks like a good idea.
scanner next verification:
scenario:
a single entry was added to secondary database that is not in the secondary database.
user performs a scan and will read 10 elements.
new element was read as 4th.
currently we report 7 errors - from 4th to 10th, even though there was only a single element which was missing from the secondary. This creates a lot of noise.
We want to be able to report only a single error and reduce noise.
Idea:
we will keep a queues with N (configurable, bounded) last entries read from both scanners. We will use it to detect detect at most N (-1?) inserts into one of the databases that are not present in the other database.
notes: we should perform verification in the same order as we perform the secondary operations - some locking on the queue is required to prevent reorderings.
the queue can contain only those entries, that did not match, if everything matches the queues can be empty, only when we will find a not matching entry we should keep it on the queue unitl we will make a decision that it is a value mismatch, not a spurious entry.
Example:
scanner1 : 1 2 3 4 5 6 7 8
scanner2 : 1 2 3 4 5 6 7 8
queues are empty
then
scanner1 : 9 10 11 12 13 14
scanner2 : 9 11 12 13 14 15
we should report that 10 is a entry not present in the secondary.
note that we do not have primary 15 to compare with, we should keep it on the queue.
queue1: empty
queue2: 15
next call:
scanner1 : 15 16 17 18
scanner2 : 16 17 18
the queue is consulted and no error is reported.
The queues should be flushed when the scanner is closed.