Repository navigation
Description
Describe the bug
For a non-contiguous Buffer, to_io_slice() and the bytes::Buf::chunks_vectored() impl walk every underlying part to its physical end and ignore the logical size field. truncate / slice / split_to / split_off shrink size without dropping parts, so after any of them both methods hand back more bytes than remaining().
Two contracts are broken:
- The
bytes::Buf::chunks_vectoredcontract. Its docs state that "the sum of the lengths of all the buffers written todstwill be less than or equal toBuf::remaining()". - The type's own invariant.
Inner::NonContiguousis documented as "the logic view of the buffer starts atparts[idx][offset]and spanssizebytes across subsequentparts", and the sibling methodschunk()andIterator::nextboth honor it with.min(*size). These two methods destructure with..and dropsizeentirely.
to_io_slice is public and documented as "Convert buffer into a slice of [IoSlice] for vectored write", so a vectored write over a shortened view would emit the bytes that were supposed to be cut off.
Steps to Reproduce
use bytes::Buf;
use bytes::Bytes;
use opendal_core::Buffer;
// 6 physical bytes across two parts
let buf = Buffer::from(vec![Bytes::from("abc"), Bytes::from("def")]);
// logical view of the first 2 bytes
let view = buf.slice(0..2);
assert_eq!(view.remaining(), 2);
assert_eq!(view.chunk(), b"ab"); // correct
let total: usize = view.to_io_slice().iter().map(|s| s.len()).sum();
assert_eq!(total, 2); // fails: total is 6, and the bytes are "abcdef"All four ways of shortening a view are affected. Measured on main (65b3e5d), same buffer ["abc", "def"]:
| view | remaining() |
chunk() |
Iterator |
to_io_slice() |
chunks_vectored() |
|---|---|---|---|---|---|
| full | 6 | 3 | 6 | 6 | 6 |
slice(0..2) |
2 | 2 | 2 | 6 | 6 |
slice(1..4) |
3 | 2 | 3 | 5 | 5 |
split_to(2) |
2 | 2 | 2 | 6 | 6 |
split_off(2) head |
2 | 2 | 2 | 6 | 6 |
chunk() and Iterator are correct in every case; the two vectored methods over-report in every shortened case.
Expected Behavior
to_io_slice() and chunks_vectored() should expose exactly the logical view — the same bytes that chunk() and Iterator yield — so their total never exceeds remaining().
Additional Context
The same two methods were already corrected once for the sibling half of this problem: #4481 ("fix(core): Buffer offset misuse") fixed them to apply offset only to the first part. The size clamp was not added at that time, so the logical-length half of the invariant is still missing.
Impact, stated honestly: the only in-tree vectored write backend, compfs, obtains its bytes through Buffer::by_ref().collect() (the Iterator impl, which respects size), so it does not hit this path — there is no active data corruption in-tree today. This is a correctness defect at the public API and trait-contract level: to_io_slice is pub and documented for vectored writes, and chunks_vectored is a bytes::Buf trait impl, so downstream users and any future in-tree consumer that performs a vectored write over a shortened view would write the extra tail bytes.
I have a minimal fix ready that mirrors the existing chunk() clamp, plus regression tests covering slice, split_to and split_off. I will open the PR right away.
- Yes, I would like to submit a PR.
Activity