MaxBufferedRecords option works well if all Kafka messages have similar size, but that is not always the case. For example, we have messages that can differ by an order of magnitude: 1KB, 10KB, and 100KB.
What do you think about adding MaxBufferedBytes(number_of_bytes) option? The client would need to track the total size of all messages (size of Record.Value field) in the buffer and TryProduce would refuse to add more messages if the limit is reached.
MaxBufferedRecordsoption works well if all Kafka messages have similar size, but that is not always the case. For example, we have messages that can differ by an order of magnitude: 1KB, 10KB, and 100KB.What do you think about adding
MaxBufferedBytes(number_of_bytes)option? The client would need to track the total size of all messages (size ofRecord.Valuefield) in the buffer andTryProducewould refuse to add more messages if the limit is reached.