feat(producer): add retry buffer tuning option to prevent OOM#3026
feat(producer): add retry buffer tuning option to prevent OOM#3026
Conversation
|
@dnwe @puellanivis @rtreffer @klauspost I’ve submitted a PR addressing the retry buffer OOM issue (#1358, #1372) by introducing an optional configuration to limit the retry buffer’s size. The changes aim to prevent unbounded memory growth while maintaining backward compatibility (unlimited buffer by default). Your review and feedback would be greatly appreciated! |
wanwenli
force-pushed
the
prevent-producer-retry-buffer-oom
branch
2 times, most recently
from
a0f5c5b to
fc4fc68
Compare
wanwenli
force-pushed
the
prevent-producer-retry-buffer-oom
branch
from
fc4fc68 to
cdba19d
Compare
This commit adds an optional configuration to Sarama's retry mechanism to limit the size of the retry buffer. The change addresses issues IBM#1358 and IBM#1372 by preventing unbounded memory growth when retries are backlogged or brokers are unresponsive. Key updates: - Added `Producer.Retry.MaxBufferLength` configuration to control the maximum number of messages stored in the retry buffer. - Implemented logic to handle overflow scenarios, ensuring non-flagged messages are either retried or sent to the errors channel, while flagged messages are re-queued. This enhancement provides a safeguard against OOM errors in high-throughput or unstable environments while maintaining backward compatibility (unlimited buffer by default). Signed-off-by: Wenli Wan <[email protected]>
wanwenli
force-pushed
the
prevent-producer-retry-buffer-oom
branch
from
cdba19d to
fbf1d5b
Compare
puellanivis
left a comment
puellanivis
left a comment
There was a problem hiding this comment.
I don’t see any concerns. Though, I don’t know how appropriate a minimum functional retry buffer of 4 KiB would be. (I just knew it should be a power of 2 for being more allocator friendly.)
FYI: I cannot actually “approve” or even “resolve conversation”.
dnwe
left a comment
dnwe
left a comment
There was a problem hiding this comment.
@wanwenli thanks for submitting your PR and thanks @puellanivis for the review and giving useful early feedback. Looking over the changes I'm happy with the implementation and it provides a useful tuning mechanism for backpressure. Thanks again!
dnwe
changed the title
3 participants
This commit adds an optional configuration to Sarama's retry mechanism to limit the size of the retry buffer. The change addresses issues #1358 and #1372 by preventing unbounded memory growth when retries are backlogged or brokers are unresponsive.
Key updates:
Producer.Retry.MaxBufferLengthconfiguration to control the maximum number of messages stored in the retry buffer.This enhancement provides a safeguard against OOM errors in high-throughput or unstable environments while maintaining backward compatibility (unlimited buffer by default).
Additionally, as demonstrated in the graph above, the retry buffer’s length is effectively constrained, confirming that the new feature functions as intended.