Description
I was reproducing with code in Rust using rust-rdkafka but that wrapper doesn't do much while reading committed offsets besides calling librdkafka itself. Because in rust-rdkafka it checks that metadata is a valid UTF-8 string it panics with errors like:
Metadata is not UTF-8: Utf8Error { valid_up_to: 3, error_len: Some(1) }
if it starts to return "random" data.
Verified also by implementing OffsetFetch and OffsetCommit in Rust that it's not an issue on the Kafka side - with pure Rust impl I couldn't reproduce issue with reading invalid data.
How to reproduce
Use byte array [10, 20, 0, 30, 40] as the commit metadata and commit for any partition. Then read committed offsets via rd_kafka_committed and in some cases metadata after \0 is just different than what was written.
Examples from other tests I've conducted where for the same metadata written we get random responses:
4 | 0 | 66 | 32 | 64 | 32 | 2 | 16 | 82 | 108 | 25 | 74 | 120 | 24 | 52 | 20 | 58 | 28 | 76 | 22 | 51 | 25 | 82 | 99 | 47 | 91 | 12 | 22 | 115 | 20 | 116 | 100 | 50 | 89 | 76 | 23 | 43 | 49 | 104 | 34 | 0 | 0 |
4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 90 | 84 | 85 | 77 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 160 | 32 | 0 | 0 | 0 | 0 | 0 | 0 | 90 | 84 | 85 | 77 | 0 | 0 |
4 | 0 | 66 | 32 | 64 | 32 | 2 | 16 | 93 | 108 | 25 | 74 | 120 | 24 | 52 | 20 | 58 | 28 | 76 | 22 | 51 | 25 | 82 | 99 | 47 | 91 | 12 | 22 | 115 | 20 | 116 | 100 | 50 | 89 | 76 | 23 | 43 | 49 | 104 | 34 | 0 | 0 |
4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 120 | 200 | 240 | 78 | 1 | 0 | 0 | 0 | 15 | 0 | 0 | 64 | 0 | 0 | 0 | 0 | 0 | 0 |
4 | 0 | 66 | 32 | 64 | 32 | 2 | 16 | 106 | 44 | 25 | 74 | 120 | 24 | 52 | 20 | 58 | 28 | 76 | 22 | 51 | 25 | 82 | 99 | 47 | 91 | 12 | 22 | 115 | 20 | 116 | 100 | 50 | 89 | 76 | 23 | 43 | 49 | 104 | 34 | 0 | 0 |
4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 160 | 62 | 84 | 232 | 1 | 0 | 0 | 0 | 239 | 179 | 223 | 191 | 254 | 255 | 255 | 255 | 17 | 76 | 32 | 64 | 1 | 0 | 0 | 0 | 3 | 25 |
rust-rdkafka used librdkafka 2.3.0 - fede1024/rust-rdkafka@87105bc.
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
this is all I set:
config: ClientConfig {
conf_map: {
"bootstrap.servers": "XXX",
"group.id": "bar",
},
log_level: Error,
}
Nothing is logged and everything seems to be working just fine.
Can't do it but no errors / warning on the broker side. Also as said above I've configured it isn't the issue purely on the Kafka side.
Description
I was reproducing with code in Rust using
rust-rdkafkabut that wrapper doesn't do much while reading committed offsets besides calling librdkafka itself. Because inrust-rdkafkait checks that metadata is a valid UTF-8 string it panics with errors like:if it starts to return "random" data.
Verified also by implementing
OffsetFetchandOffsetCommitin Rust that it's not an issue on the Kafka side - with pure Rust impl I couldn't reproduce issue with reading invalid data.How to reproduce
Use byte array
[10, 20, 0, 30, 40]as the commit metadata and commit for any partition. Then read committed offsets viard_kafka_committedand in some cases metadata after \0 is just different than what was written.Examples from other tests I've conducted where for the same metadata written we get random responses:
rust-rdkafkausedlibrdkafka2.3.0 - fede1024/rust-rdkafka@87105bc.Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
<REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">3.7.0<REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>this is all I set:
32-Ubuntu SMP Mon Jan 9 12:28:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linuxdebug=..as necessary) from librdkafkaNothing is logged and everything seems to be working just fine.
Can't do it but no errors / warning on the broker side. Also as said above I've configured it isn't the issue purely on the Kafka side.