Summary

A UBSan (UndefinedBehaviorSanitizer) error was found in test_rocksdb_read_only integration test when running under the combined ASan+UBSan build. The test fails intermittently (~4/10 runs).

This was surfaced by PR #99657 ("CI: combine ASan+UBSan into single builds"), which for the first time runs UBSan in integration tests (previously only ASan was used).

Error

/ClickHouse/src/IO/BufferBase.h:60:54: runtime error: addition of unsigned offset to 0x7bdf1faef183 overflowed to 0x7bdf1faef181
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /ClickHouse/src/IO/BufferBase.h:60:54

Stack trace:

#0  DB::BufferBase::BufferBase(char*, unsigned long, unsigned long)  src/IO/BufferBase.h:60
#1  DB::ReadBuffer::ReadBuffer(...)                                  src/IO/ReadBuffer.h:38
#2  DB::ReadBufferFromMemory::ReadBufferFromMemory<char>(...)        src/IO/ReadBufferFromMemory.h:30
#3  DB::ReadBufferFromString::ReadBufferFromString<rocksdb::Slice>   src/IO/ReadBufferFromString.h:14
#4  void DB::fillColumns<rocksdb::Slice>(...)                        src/Storages/KVStorageUtils.h:44
#5  DB::EmbeddedRocksDBSource::generateFullScan()                    src/Storages/RocksDB/StorageEmbeddedRocksDB.cpp:161
#6  DB::EmbeddedRocksDBSource::generate()                            src/Storages/RocksDB/StorageEmbeddedRocksDB.cpp:136

Root cause analysis

generateFullScan() iterates over the RocksDB table, passing each key/value rocksdb::Slice to fillColumns(), which wraps it in a ReadBufferFromString:

for (size_t rows = 0; iterator->Valid() && rows < max_block_size; ++rows, iterator->Next())
{
    fillColumns(iterator->key(), ...);   // <-- Slice passed here
    fillColumns(iterator->value(), ...);
}

// KVStorageUtils.h:44
ReadBufferFromString buffer(slice);   // -> ReadBufferFromMemory(slice.data(), slice.size())

// BufferBase.h:60
BufferBase(Position ptr, size_t size, size_t offset)
    : pos(ptr + offset), working_buffer(ptr, ptr + size), ...  // UBSan fires here: ptr + size overflows

The UBSan error fires because slice.size() returns 0xFFFFFFFFFFFFFFFE (i.e. (size_t)-2) — a clearly corrupted value. A valid rocksdb::Slice can never have this size. The loop correctly checks iterator->Valid() before accessing the slice, so this is not a simple logic error.

The most likely cause is memory corruption or iterator state corruption specific to RocksDB's read-only restart path (test_rocksdb_read_only stops the server and restarts it in read-only mode). The intermittency (~4/10 runs) is consistent with a race condition or filesystem/RocksDB state issue during the server restart.

Before PR #99657, this bug was silent: ASan-only builds did not abort on UBSan errors, so the corruption was undetected and the query either returned wrong data or crashed with a different error. Now that UBSan aborts on halt_on_error=1 (inherited from ASan's combined mode), the server crashes visibly.

CI report

• PR where this was found: CI: combine ASan+UBSan into single builds, remove UBSan #99657
• CI report: https://s3.amazonaws.com/clickhouse-test-reports/PRs/99657/25e970dd6ff97c27deb476e8244c8057f82f7321/result_pr.json
• Test: Integration tests (amd_asan_ubsan, targeted) → test_rocksdb_read_only/test.py::test_read_only
• Failure rate: ~4/10 runs

Temporary workaround

As a workaround to unblock CI, the following suppression was added to tests/ubsan_ignorelist.txt in PR #99657:

fun:*DB::BufferBase::BufferBase*

This suppresses the UBSan pointer-overflow check in BufferBase::BufferBase at compile time. It should be removed once this bug is fixed.

Suggested investigation

1. Look at how the RocksDB iterator is created and initialized in read-only mode restart in StorageEmbeddedRocksDB.cpp
2. Check whether the rocksdb::Iterator or its underlying data is being accessed after the underlying SST files have been closed/remapped during the read-only restart
3. Run test_rocksdb_read_only under Valgrind or with ROCKSDB_DISABLE_BACKGROUND_THREADS=1 to narrow down the race condition