I have a crate implementing a custom b+tree with Layout API.
The CI is running ok for 2 months.
Recently CI fail in combination "LSAN + release" using nightly 1.99.0-nightly-2026-07-10 (always reproducible)
(edit: cargo test +nightly --release， cargo +nightly test has corruption)
Conditions of the CI pass (any of):
- every thing (including
cargo test, cargo test --release) stable toolchain, or nightly before "nightly-2026-07-09" are OK
- miri validation OK on nightly
background
This b+tree has fixed node size (2*64B),
The memory layout as decribe in the doc https://docs.rs/embed-btree/latest/embed_btree/.
For an InterNode, The first 64B is header + key array, the second 64B is child_ptr array. the whole page is allocated with
Layout::from_size_align(). use const fn to calculate Key/Value type size and alignment to determine capacity for the node.
key structs
NodeBase & NodeHeader
LeafNode
InterNode
The code and unittest are all single threaded, no atomic is ever used.
I have replace all read/write with reference to raw pointer access, in branch "btree_debug", to issolate the cause, according the suggestion of gemini. But that seems not related.
step to reproduce
git clone https://github.com/NaturalIO/embed-collections-rs.git
git checkout btree_debug
cd btree
make test-leak "test_alter_key_after_move_update_sep_height_2 -F trace_log,std"
on panic or assertion failure, will print a debug log to /tmp/emb_ring.log
the crate::trace_log! macro will add log to a memory ring which can be dump on panic
I have pinpoint the regression commit of rustc with cargo-bisect-rustc.
(Although cargo test --release without the LSAN flag cannot reproduce, but cargo-bisect-rustc can pinpoint to the same commit 4ccf0ea
either with or without RUSTFLAGS="-Zsanitizer=leak". I have no idea )
$ RUSTFLAGS="-Zsanitizer=leak" cargo bisect-rustc --start 2026-07-09 --end 2026-07-10 --by-commit --verbose --preserve -- test --release test_alter_key_after_move_update_sep_height_2
$ cargo bisect-rustc --start 2026-07-08 --end 2026-07-10 --by-commit --verbose --preserve -- test --release test_alter_key_after_move_update_sep_height_2
the resut:
Regression in https://github.com/rust-lang/rust/commit/4ccf0eadf7087640a20d0b5e09ea4b73b4d67033. Note that if it is a legacy rollup build, it might be available in https://github.com/rust-lang-ci/rust/commit/4ccf0eadf7087640a20d0b5e09ea4b73b4d67033.
The PR introducing the regression in this rollup is #158666: Carry the `b_offset` inside `BackendRepr::ScalarPair`
I have pinpoint failure to simple unitest construction steps during btree node initialization.
NaturalIO/embed-collections-rs@c53f626
#[logfn]
#[rstest]
fn test_alter_key_after_move_update_sep_height_2(setup_log: ()) {
reset_alive_count();
{
// TreeBuilder is just a helper to contruct test secenario
// CounterI32 is just a wrapper of i32 to test drop behavior
let mut builder = TreeBuilder::<CounterI32, CounterI32>::default();
// Construct Root -> [leaf0 | sep1 | leaf1 | sep2 | leaf2]
let mut leaf0 = builder.new_leaf();
let mut leaf1 = builder.new_leaf();
let mut leaf2 = builder.new_leaf();
builder.insert_leaf(&mut leaf0, CounterI32::from(10), CounterI32::from(100));
builder.insert_leaf(&mut leaf1, CounterI32::from(20), CounterI32::from(200));
builder.insert_leaf(&mut leaf2, CounterI32::from(30), CounterI32::from(300));
// leaf0 - leaf2 is unrelated, we just need it's addr to assign to the root node
let mut root: InterNode<CounterI32, CounterI32> = builder.new_inter(1);
assert_eq!(root.height(), 1, "alright");
root.set_left_ptr(leaf0.get_ptr_mut());
assert_eq!(root.height(), 1, "alright");
// it's all normal above.
unsafe {
let slot0_key = root.key_ptr(0); // just calcuate the key offset in the InterNode,
let slot0_child = root.child_ptr(0); // just calculate the first child offset in the InterNode
crate::trace_log!("root head addr {:p}", root.get_ptr());
crate::trace_log!("root key0 addr {slot0_key:p}");
crate::trace_log!("root child0 {slot0_child:p}");
}
// if removing the trace_log above, the root.height will be corrupted during insert_no_split
assert_eq!(root.height(), 1, "fail here"); // the height field in NodeHeader was wrong by unknown reason even before insertion
// expect insert_no_split at idx 0, because there no element for now
// insert should write to key address and child_ptr adddress and increase count in NodeHeader, but assertion already fail above.
root.insert_no_split(CounterI32::from(20), leaf1.get_ptr_mut());
assert_eq!(root.height(), 1, "fail");
root.insert_no_split(CounterI32::from(30), leaf2.get_ptr_mut());
assert_eq!(root.height(), 1, "fail");
// fail here
root.validate();
let mut map = builder.build(root.clone().into());
assert_eq!(map.len(), 3);
assert_eq!(map.leaf_count(), 3);
assert_eq!(map.inter_count(), 1);
map.dump();
map.validate();
// the following code is commented out because unrelated to the issue
}
}the log for the test:
make test-leak "test_alter_key_after_move_update_sep_height_2 -F trace_log,std"
in /tmp/emb_ring.log
[2026-07-11 10:59:35.666632][INFO][ThreadId(2)][entry_move.rs:510] <<< test_alter_key_after_move_update_sep_height_2 (setup_log = ()) enter <<<
[2026-07-11 10:59:35.666907][DEBUG][ThreadId(2)][entry_move.rs:534] root head addr 0x501000001800
[2026-07-11 10:59:35.666928][DEBUG][ThreadId(2)][entry_move.rs:535] root key0 addr 0x501000001808
[2026-07-11 10:59:35.666938][DEBUG][ThreadId(2)][entry_move.rs:536] root child0 0x501000001880
[2026-07-11 10:59:35.794741][ERROR][ThreadId(2)][entry_move.rs:538] panic occur: panicked at btree/src/btree/tests/entry_move.rs:538:9:
assertion `left == right` failed: fail here
left: 0
right: 1
trace: 0: captains_log::log_impl::log_panic
1: <captains_log::log_impl::panic_and_exit_hook as core::ops::function::Fn<(&std::panic::PanicHookInfo,)>>::call
2: std::panicking::panic_with_hook
3: std::panicking::panic_handler::{closure#0}
4: std::sys::backtrace::__rust_end_short_backtrace::<std::panicking::panic_handler::{closure#0}, !>
5: __rustc::rust_begin_unwind
6: core::panicking::panic_fmt
7: core::panicking::assert_failed_inner
8: core::panicking::assert_failed::<u32, u32>
9: embed_btree::btree::tests::entry_move::test_alter_key_after_move_update_sep_height_2
10: <embed_btree::btree::tests::entry_move::test_alter_key_after_move_update_sep_height_2::{closure#0} as core::ops::function::FnOnce<()>>::call_once
11: test::__rust_begin_short_backtrace::<core::result::Result<(), alloc::string::String>, fn() -> core::result::Result<(), alloc::string::String>>
12: test::run_test::{closure#0}
13: std::sys::backtrace::__rust_begin_short_backtrace::<test::run_test::{closure#1}, ()>
14: <std::thread::lifecycle::spawn_unchecked<test::run_test::{closure#1}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
15: <std::sys::thread::unix::Thread>::new::thread_start
16: _ZL15ThreadStartFuncILb0EEPvS0_
17: start_thread
at ./nptl/pthread_create.c:447:8
18: clone3
at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78:0
Meta
My local env is ubuntu-24.04.3 LTS
rustc +nightly --version --verbose
rustc 1.99.0-nightly (af3d95584 2026-07-09)
binary: rustc
commit-hash: af3d95584dbddcae597890340995509a7fb47a50
commit-date: 2026-07-09
host: x86_64-unknown-linux-gnu
release: 1.99.0-nightly
LLVM version: 22.1.8
uname -a
Linux x1 6.11.0-29-generic #29~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 26 14:16:59 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
github ci link
https://github.com/NaturalIO/embed-collections-rs/actions/runs/29080714864/job/86322665048
Chances of miminal code?
embed-btree has no dependencies outside, and the rstest is added to the cases after test failure, and log/captains-log is not turned on in the CI, just for debugging.
I have a crate implementing a custom b+tree with Layout API.
The CI is running ok for 2 months.
Recently CI fail in combination "LSAN + release" using nightly 1.99.0-nightly-2026-07-10 (always reproducible)
(edit:
cargo test +nightly --release，cargo +nightly testhas corruption)Conditions of the CI pass (any of):
cargo test,cargo test --release) stable toolchain, or nightly before "nightly-2026-07-09" are OKbackground
This b+tree has fixed node size (2*64B),
The memory layout as decribe in the doc https://docs.rs/embed-btree/latest/embed_btree/.
For an InterNode, The first 64B is header + key array, the second 64B is child_ptr array. the whole page is allocated with
Layout::from_size_align(). use const fn to calculate Key/Value type size and alignment to determine capacity for the node.key structs
NodeBase & NodeHeader
LeafNode
InterNode
The code and unittest are all single threaded, no atomic is ever used.
I have replace all read/write with reference to raw pointer access, in branch "btree_debug", to issolate the cause, according the suggestion of gemini. But that seems not related.
step to reproduce
on panic or assertion failure, will print a debug log to /tmp/emb_ring.log
the crate::trace_log! macro will add log to a memory ring which can be dump on panic
I have pinpoint the regression commit of rustc with cargo-bisect-rustc.
(Although cargo test --release without the LSAN flag cannot reproduce, but cargo-bisect-rustc can pinpoint to the same commit 4ccf0ea
either with or without
RUSTFLAGS="-Zsanitizer=leak". I have no idea )I have pinpoint failure to simple unitest construction steps during btree node initialization.
NaturalIO/embed-collections-rs@c53f626
the log for the test:
in /tmp/emb_ring.log
Meta
My local env is ubuntu-24.04.3 LTS
github ci link
https://github.com/NaturalIO/embed-collections-rs/actions/runs/29080714864/job/86322665048
Chances of miminal code?
embed-btree has no dependencies outside, and the rstest is added to the cases after test failure, and log/captains-log is not turned on in the CI, just for debugging.