feat(spans-buffer): Add more flusher documentation#108029
Conversation
github-actions
bot
added
the
Scope: Backend
| This means the queue a segment lands in depends on both the trace ID and the | ||
| current set of assigned partitions. On a Kafka rebalance (triggered by | ||
| consumer deployment, scaling, or crashes), `assigned_shards` changes, possibly | ||
| in length or in values. Then, the same trace ID may route to a different queue. |
There was a problem hiding this comment.
Reading the code, the queue that a segment ends up on is based on the modulo of the number of assigned shards. So the queue that a segment is written too doesn't depend on the value of assigned shards. Just the queues that are read from are just the assigned shards.
shard = self.assigned_shards[
int(project_and_trace.split(":")[1], 16) % len(self.assigned_shards)
]
queue_key = self._get_queue_key(shard)
I think this means that assuming partitions are evenly spread (which they should be) we won't get incomplete segments even if the values themselves change.
There was a problem hiding this comment.
I don't think so, the shard (and therefore the queue) is the value at an index of self.assigned_shards.
Example with same length, different values:
Before: assigned_shards = [0,1,2,3,4], index=3 -> shard=3 -> span-buf:q:3
After: assigned_shards = [0,3,5,8,9], index=3 -> shard=8 -> span-buf:q:8
There was a problem hiding this comment.
Yep you're right. Sorry about that.
Documenting flusher behaviour on kafka rebalancing/consumer deployment.
2 participants
Documenting flusher behaviour on kafka rebalancing/consumer deployment.