chore(core): assert all munmaps are succesful, add if-guards around non-compliant code#6092
chore(core): assert all munmaps are succesful, add if-guards around non-compliant code#6092bluestreak01 merged 59 commits intomasterfrom
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughAdd guards to avoid munmap on zero/invalid addresses and tighten mmap/mremap/unmap validation and error handling across Files, MmapCache, and multiple callers; introduce errno helpers and adjust related exception checks. No public API signatures were changed. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Caller as Caller (various)
participant Mmap as Files / FilesFacade
participant Cache as MmapCache
Note over Caller,Mmap: Mapping path (mmap/mremap)
Caller->>Mmap: mmap(... len)
alt len invalid (<=0)
Mmap-->>Caller: throw CairoException (invalid len)
else success
Mmap-->>Caller: addr (or MAP_FAILED -> throw)
end
Note over Caller,Cache: Unmap path
Caller->>Cache: unmap(addr, len, tag)
alt addr == 0 or len <= 0
Cache-->>Caller: throw CairoException (invalid address/length)
else
Cache->>Cache: rc = munmap(addr,len)
alt rc == success
Cache->>Cache: recordMemAlloc(-len, tag)
Cache-->>Caller: return
else rc == failure
Cache-->>Caller: throw CairoException(errno)
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
core/src/main/java/io/questdb/std/MmapCache.java (1)
287-296: Asserting munmap failures is good; add a release-path signal and a len>0 assert.The early return on address==0 and conditional accounting are solid. With assertions disabled in production, a failed munmap silently does nothing and leaks accounting. Consider emitting a log/warn (or metrics counter) on failure, and assert len>0 to catch misuse earlier.
Example minimal tweak:
private static void unmap0(long address, long len, int memoryTag) { if (address == 0) { return; } + assert len > 0 : "munmap length must be > 0 [len=" + len + ']'; int result = Files.munmap0(address, len); if (result != -1) { Unsafe.recordMemAlloc(-len, memoryTag); } assert result != -1 : "failed to munmap non-zero address [address=" + address + ", len=" + len + ", memoryTag=" + memoryTag + ']'; + // Optional: emit a release-path signal when assertions are off. + // e.g., LOG.warn().$("munmap failed [address=").$(address).$(", len=").$(len).$(", tag=").$(memoryTag).I$(); }core/src/main/java/io/questdb/griffin/engine/table/parquet/MappedMemoryPartitionDescriptor.java (2)
55-59: Use overflow-safe sizing and add a sanity check before subtracting HEADER_SIZE.Casting symbolOffsetsSize to int can overflow silently. Also, subtracting HEADER_SIZE assumes symbolOffsetsAddr ≥ HEADER_SIZE.
- if (symbolOffsetsAddr != 0) { - final long symbolOffsetsSize = columnData.get(rawIndex + SYMBOL_OFFSET_SIZE_OFFSET); - final long offsetsMemSize = SymbolMapWriter.keyToOffset((int) symbolOffsetsSize + 1); - ff.munmap(symbolOffsetsAddr - SymbolMapWriter.HEADER_SIZE, offsetsMemSize, MemoryTag.MMAP_PARQUET_PARTITION_CONVERTER); - } + if (symbolOffsetsAddr != 0) { + final long symbolOffsetsSize = columnData.get(rawIndex + SYMBOL_OFFSET_SIZE_OFFSET); + // Ensure address math is valid and sizing is safe. + assert symbolOffsetsAddr >= SymbolMapWriter.HEADER_SIZE; + final int keyCapacity = Math.toIntExact(symbolOffsetsSize + 1); + final long offsetsMemSize = SymbolMapWriter.keyToOffset(keyCapacity); + ff.munmap(symbolOffsetsAddr - SymbolMapWriter.HEADER_SIZE, offsetsMemSize, MemoryTag.MMAP_PARQUET_PARTITION_CONVERTER); + }
55-59: Optional: reduce repetition of MemoryTag.Capture MemoryTag.MMAP_PARQUET_PARTITION_CONVERTER in a local final int tag to make the loop a touch cleaner.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
core/src/main/java/io/questdb/griffin/engine/table/parquet/MappedMemoryPartitionDescriptor.java(1 hunks)core/src/main/java/io/questdb/std/MmapCache.java(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (26)
- GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
- GitHub Check: New pull request (Hosted Running tests on windows-other)
- GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin)
- GitHub Check: New pull request (Hosted Running tests on mac-other)
- GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo)
- GitHub Check: New pull request (SelfHosted Other tests Start ARM Agent)
- GitHub Check: New pull request (SelfHosted Other tests Start X64Zfs Agent)
- GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
- GitHub Check: New pull request (Hosted Running tests on mac-griffin)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-other)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-pgwire)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-cairo)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-fuzz)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start X64Zfs Agent)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-griffin)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start ARM Agent)
- GitHub Check: New pull request (Check Changes Check changes)
🔇 Additional comments (1)
core/src/main/java/io/questdb/griffin/engine/table/parquet/MappedMemoryPartitionDescriptor.java (1)
50-53: Good zero-addr guard; verify if primary column can ever be zero.This prevents munmap(0) for columnSecondaryAddr. If there are scenarios where columnAddr (Line 45) can be 0, mirror the same guard there to avoid tripping the new assertion path.
core/src/main/java/io/questdb/griffin/engine/table/parquet/MappedMemoryPartitionDescriptor.java
Show resolved
Hide resolved
|
GitHub Actions - Rebuild Native Libraries seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (8)
core/src/main/java/io/questdb/std/MmapCache.java (3)
79-82: Unify failure checks to MAP_FAILED and fix misleading comment.Current check uses
-1and comment says “return 0”. UseFilesFacade.MAP_FAILEDand correct the comment.- if (address == -1) { - // mmap failed, return 0 - return address; - } + if (address == FilesFacade.MAP_FAILED) { + // mmap failed + return address; + }
185-191: Replace-1comparisons withFilesFacade.MAP_FAILEDin mremap path.Aligns with PR objective to unify MAP_FAILED checks.
- if (newAddress != -1) { + if (newAddress != FilesFacade.MAP_FAILED) { record.address = newAddress; record.length = newSize; record.memoryTag = memoryTag; mmapAddrCache.removeAt(addrMapIndex); mmapAddrCache.put(newAddress, record); } ... - if (newAddress != -1) { + if (newAddress != FilesFacade.MAP_FAILED) { record.count--; MmapCacheRecord newRecord = createMmapCacheRecord(fd, mmapCacheKey, newSize, newAddress, memoryTag); if (fdIndex != Integer.MAX_VALUE) { mmapFileCache.putAt(fdIndex, mmapCacheKey, newRecord); } else { mmapFileCache.put(mmapCacheKey, newRecord); } mmapAddrCache.put(newAddress, newRecord); }Also applies to: 198-214
50-91: Replace raw-1checks withFilesFacade.MAP_FAILEDand switch invalid‐parameter throws toerrnoInvalidParameter
- core/src/main/java/io/questdb/std/MmapCache.java still uses raw
if (address == -1)/if (newAddress != -1)incacheMmap,mmap0,mremap0, andunmap0; replace these with comparisons againstFilesFacade.MAP_FAILED(or the agreed facade constant) and propagate errors via the errno mechanism.- The initial len/offset validation in
cacheMmapthrowsCairoException.critical(0)…put("invalid len"); update this to useFiles.errnoInvalidParameter(…)instead.core/src/main/java/io/questdb/griffin/engine/functions/catalogue/WalTransactionsFunctionFactory.java (1)
116-131: Prevent busy-spin: rethrow non-file-read CairoExceptionsInside the retry loop, exceptions other than “file cannot read” are swallowed, causing an unbounded tight retry. Rethrow to avoid CPU spin and to surface genuine faults.
} catch (CairoException e) { Misc.free(cursor); if (e.isFileCannotRead()) { // Txn sequencer can have its parts deleted due to housekeeping // Need to keep scanning until we find a valid part if (txnLo == 0) { long writerTxn = executionContext.getCairoEngine().getTableSequencerAPI().getTxnTracker(tableToken).getWriterTxn(); if (writerTxn > 0) { txnLo = writerTxn; continue; } } throw e; } + // Not a transient "file cannot read" condition — propagate. + throw e; }core/src/main/java/io/questdb/cairo/TableWriter.java (3)
1719-1724: Add the same unmap guard here for consistency and safety.
convertPartitionParquetToNative()unmapsparquetAddrwithout validating it. Align this with other sites to avoid failing onaddr == 0.- ff.munmap(parquetAddr, parquetSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); + if (parquetAddr != 0) { + ff.munmap(parquetAddr, parquetSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); + }
3479-3483: Guard this munmap as well.If
mapROreturns 0 (or throws before assignment is reliable), unmapping 0 will now fail. Guard to match PR’s policy.- ff.munmap(mappedAddr, expectedFileSize, MemoryTag.MMAP_DEFAULT); + if (mappedAddr != 0) { + ff.munmap(mappedAddr, expectedFileSize, MemoryTag.MMAP_DEFAULT); + }
3543-3548: Guard munmap in symbol column check.Same rationale: avoid unmapping address 0.
- ff.munmap(address, fileSize, MemoryTag.MMAP_DEFAULT); + if (address != 0) { + ff.munmap(address, fileSize, MemoryTag.MMAP_DEFAULT); + }core/src/main/java/io/questdb/cairo/wal/CopyWalSegmentUtils.java (1)
264-285: EnsurenewAuxMemAddris unmapped on all pathsIf an exception is thrown between mapping and explicit unmap, the mapping can leak. Wrap in
try/finally.- final long newAuxMemSize = columnTypeDriver.getAuxVectorSize(rowCount); - final long newAuxMemAddr = TableUtils.mapRW(ff, secondaryFd, newAuxMemSize, MEMORY_TAG); - ff.madvise(newAuxMemAddr, newAuxMemSize, Files.POSIX_MADV_RANDOM); - - columnTypeDriver.shiftCopyAuxVector( - dataStartOffset, - auxMemAddr, - startRowNumber, - startRowNumber + rowCount - 1, // inclusive - newAuxMemAddr, - newAuxMemSize - ); - - columnRollSink.setSrcOffsets(dataStartOffset, columnTypeDriver.getAuxVectorSize(startRowNumber)); - columnRollSink.setDestSizes(dataSize, newAuxMemSize); - - if (commitMode != CommitMode.NOSYNC) { - ff.msync(newAuxMemAddr, newAuxMemSize, commitMode == CommitMode.ASYNC); - } - // All in memory calls, no need to unmap in finally - ff.munmap(newAuxMemAddr, newAuxMemSize, MEMORY_TAG); - return true; + final long newAuxMemSize = columnTypeDriver.getAuxVectorSize(rowCount); + final long newAuxMemAddr = TableUtils.mapRW(ff, secondaryFd, newAuxMemSize, MEMORY_TAG); + try { + ff.madvise(newAuxMemAddr, newAuxMemSize, Files.POSIX_MADV_RANDOM); + + columnTypeDriver.shiftCopyAuxVector( + dataStartOffset, + auxMemAddr, + startRowNumber, + startRowNumber + rowCount - 1, // inclusive + newAuxMemAddr, + newAuxMemSize + ); + + columnRollSink.setSrcOffsets(dataStartOffset, columnTypeDriver.getAuxVectorSize(startRowNumber)); + columnRollSink.setDestSizes(dataSize, newAuxMemSize); + + if (commitMode != CommitMode.NOSYNC) { + ff.msync(newAuxMemAddr, newAuxMemSize, commitMode == CommitMode.ASYNC); + } + return true; + } finally { + if (newAuxMemAddr > 0) { + ff.munmap(newAuxMemAddr, newAuxMemSize, MEMORY_TAG); + } + }
🧹 Nitpick comments (29)
core/src/main/c/share/dedup.cpp (1)
635-643: Optional: express the null-scan with std::any_of for clarity.Same behavior, slightly tighter and idiomatic.
- [&](int data_index, int64_t lo, int64_t hi) -> bool { - auto data = data_ptrs[data_index]; - for (int64_t i = lo; i < hi; i++) { - if (data[i] != null_value) { - return true; - } - } - return false; - } + [&](int data_index, int64_t lo, int64_t hi) -> bool { + const T* data = data_ptrs[data_index]; + return std::any_of(data + lo, data + hi, + [&](const T &v){ return v != null_value; }); + }core/src/main/java/io/questdb/cutlass/text/CopyTask.java (1)
1342-1344: Defensive reset after unmapSet the local to 0 after unmapping to prevent accidental reuse.
- if (mergeIndexAddr != 0) { - ff.munmap(mergeIndexAddr, mergedIndexSize, MemoryTag.MMAP_IMPORT); - } + if (mergeIndexAddr != 0) { + ff.munmap(mergeIndexAddr, mergedIndexSize, MemoryTag.MMAP_IMPORT); + mergeIndexAddr = 0; // defensive + }core/src/main/java/io/questdb/griffin/engine/functions/table/ReadParquetFunctionFactory.java (1)
100-102: Minor consistency/defenseReset addr after unmap.
- if (addr != 0) { - ff.munmap(addr, fileSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); - } + if (addr != 0) { + ff.munmap(addr, fileSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); + addr = 0; + }.github/workflows/rebuild_native_libs.yml (1)
33-35: Make cmake uninstall resilientAvoid failures if cmake is absent or has dependencies.
- # https://github.com/actions/runner-images/issues/12912 - # Temporary fix (mtopolnik 2025-09-02): GH Action pre-installs a custom cmake version, install then fails - brew uninstall cmake + # https://github.com/actions/runner-images/issues/12912 + # Temporary fix (mtopolnik 2025-09-02): GH Action pre-installs a custom cmake version, install then fails + if brew list --versions cmake >/dev/null 2>&1; then + brew uninstall --ignore-dependencies cmake || true + ficore/src/main/c/windows/files.c (1)
703-711: Also guard negative len/offset (paranoia, matches Java checks)Native layer can mirror Java validations for robustness.
if (len == 0) { // With len == 0, Windows will mmap the whole file. To be POSIX-compatible, return MAP_FAILED instead. // docs: https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffileex // dwNumberOfBytesToMap: "If this parameter is 0 (zero), the mapping extends from the // specified offset to the end of the file mapping." SetLastError(ERROR_INVALID_PARAMETER); SaveLastError(); return -1; } + if (len < 0 || offset < 0) { + SetLastError(ERROR_INVALID_PARAMETER); + SaveLastError(); + return -1; + }core/src/main/java/io/questdb/cairo/O3PartitionJob.java (2)
267-269: Reset address after unmapSmall defensive cleanup.
- if (parquetAddr != 0) { - ff.munmap(parquetAddr, parquetSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); - } + if (parquetAddr != 0) { + ff.munmap(parquetAddr, parquetSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); + parquetAddr = 0; + }
2570-2572: Nit: reset address; plus consider zero-size fast-path
- Reset local after unmap.
- Optionally skip mapping when newParquetSize == 0 to avoid needless exceptions.
- if (parquetAddr != 0) { - ff.munmap(parquetAddr, newParquetSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); - } + if (parquetAddr != 0) { + ff.munmap(parquetAddr, newParquetSize, MemoryTag.MMAP_PARQUET_PARTITION_DECODER); + parquetAddr = 0; + }Additional (outside this hunk) suggestion:
// early in updateParquetIndexes, before mapping if (newParquetSize == 0) { return; }core/src/main/java/io/questdb/std/MmapCache.java (8)
51-58: Prefer errnoInvalidParameter over errno=0 for invalidlen.Use the dedicated errno to improve diagnostics and align with cross-platform consistency goals.
- if (len < 1) { - throw CairoException.critical(0) + if (len < 1) { + throw CairoException.critical(Files.errnoInvalidParameter()) .put("could not mmap file, invalid len [len=").put(len) .put(", offset=").put(offset) .put(", fd=").put(fd) .put(", memoryTag=").put(memoryTag) .put(']'); }
112-121: Use errnoInvalidParameter for invalidnewSize.Same rationale as for
lenabove.- if (newSize < 1) { - throw CairoException.critical(0) + if (newSize < 1) { + throw CairoException.critical(Files.errnoInvalidParameter()) .put("could not remap file, invalid newSize [previousSize=").put(previousSize) .put(", newSize=").put(newSize) .put(", offset=").put(offset) .put(", fd=").put(fd) .put(", memoryTag=").put(memoryTag) .put(']'); }
231-235: Return EINVAL-like errno for invalid unmap arguments.Use the new errno helper instead of
0.- if (address <= 0 || len <= 0) { - throw CairoException.critical(0) + if (address <= 0 || len <= 0) { + throw CairoException.critical(Files.errnoInvalidParameter()) .put("unmap: invalid address or length [address=" + address + ", len=" + len + ']'); }
293-304: Unify MAP_FAILED check in mremap accounting.Mirror mmap path by using
FilesFacade.MAP_FAILED.- if (address != -1) { + if (address != FilesFacade.MAP_FAILED) { if (oldMemoryTag == memoryTag) { Unsafe.recordMemAlloc(newSize - previousSize, memoryTag); } else { Unsafe.recordMemAlloc(newSize, memoryTag); Unsafe.recordMemAlloc(-previousSize, oldMemoryTag); } }
306-316: Treat munmap success explicitly (result == 0); throw on non‑zero.This is clearer and avoids accidental success if return contract ever changes.
- int result = Files.munmap0(address, len); - if (result != -1) { + int result = Files.munmap0(address, len); + if (result == 0) { Unsafe.recordMemAlloc(-len, memoryTag); } else { throw CairoException.critical(Os.errno()) .put("munmap failed [address=").put(address) .put(", len=").put(len) .put(", memoryTag=").put(memoryTag).put(']'); }
281-283: Update comment: unmap is executed outside the synchronized section, not “offloaded”.The current comment suggests a background worker that doesn’t exist here.
- // offload the unmap to a single thread to not block everyone under synchronized section + // perform unmap outside the synchronized section to avoid blocking other threads unmap0(unmapPtr, unmapLen, unmapTag);
50-61: Optional: validateoffset >= 0for early parameter sanity.Negative offsets will fail in native anyway; failing early yields clearer errors and avoids syscalls.
122-124: Potential accounting mismatch when bypassing cache.When bypassing cache,
mremap0(..., oldMemoryTag=memoryTag, memoryTag)assumes unchanged tag. If callers ever change tags across remaps, memory accounting will be off.Would you like a follow-up patch to plumb the previous tag (or document the invariant that tags must not change across non-cached remaps)?
core/src/main/java/io/questdb/cairo/wal/CheckWalTransactionsJob.java (1)
106-109: Optional: log swallowed race for observabilityYou correctly gate on
isFileCannotRead(). Consider a low-level log to aid production triage when this race is frequent.- } catch (CairoException e) { - if (!e.isFileCannotRead()) { - throw e; - } // race, table is dropped, ApplyWal2TableJob is already deleting the files - } + } catch (CairoException e) { + if (!e.isFileCannotRead()) { + throw e; + } + // race, table is dropped, ApplyWal2TableJob is already deleting the files + if (LOG.isEnabled(LogLevel.DEBUG)) { + LOG.debug().$("txn file unreadable during check (likely drop in progress) [table=") + .$(tableToken).I$(); + } + }core/src/main/java/io/questdb/cairo/wal/WalPurgeJob.java (1)
427-433: Minor: capture errno once to keep logs consistentAvoid calling
ff.errno()twice; capture once after the failing call so the message reflects the same errno used in the condition.- if (!ff.rmdir(path, false) && !Files.isErrnoFileDoesNotExist(ff.errno())) { - LOG.debug() - .$("could not delete directory [path=").$(path) - .$(", errno=").$(ff.errno()) - .I$(); + final boolean removed = ff.rmdir(path, false); + final int errno = removed ? 0 : ff.errno(); + if (!removed && !Files.isErrnoFileDoesNotExist(errno)) { + LOG.debug() + .$("could not delete directory [path=").$(path) + .$(", errno=").$(errno) + .I$(); return false; }core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogV1.java (2)
372-379: Use ff.errno() for consistency with FilesFacade-backed callsFor map failures, prefer
ff.errno()(already used elsewhere) to keep errno sourcing consistent with the facade.- if (newAddr == FilesFacade.MAP_FAILED) { - throw CairoException.critical(Os.errno()).put("cannot mmap transaction log [path=").put(path).put(']'); + if (newAddr == FilesFacade.MAP_FAILED) { + throw CairoException.critical(ff.errno()).put("cannot mmap transaction log [path=").put(path).put(']'); }
391-395: Same here: prefer ff.errno() on remap failure- if (newAddr == FilesFacade.MAP_FAILED) { - throw CairoException.critical(Os.errno()).put("cannot remap transaction log [fd=").put(fd).put(']'); + if (newAddr == FilesFacade.MAP_FAILED) { + throw CairoException.critical(ff.errno()).put("cannot remap transaction log [fd=").put(fd).put(']'); }core/src/main/java/io/questdb/cairo/wal/WalWriter.java (1)
1759-1766: Minor nit: avoid repeating the async checkCache the async flag once to reduce duplication.
- if (commitMode != CommitMode.NOSYNC) { - ff.msync(auxMemAddr, auxMemSize, commitMode == CommitMode.ASYNC); - } + final boolean async = commitMode == CommitMode.ASYNC; + if (commitMode != CommitMode.NOSYNC) { + ff.msync(auxMemAddr, auxMemSize, async); + } ... - if (commitMode != CommitMode.NOSYNC) { - ff.msync(dataMemAddr, varColSize, commitMode == CommitMode.ASYNC); - } + final boolean async = commitMode == CommitMode.ASYNC; + if (commitMode != CommitMode.NOSYNC) { + ff.msync(dataMemAddr, varColSize, async); + }Also applies to: 1776-1783
core/src/main/java/io/questdb/cairo/TableWriter.java (1)
144-146: Remove redundant static imports or the wildcard to avoid duplication.You have both
import static io.questdb.cairo.TableUtils.*;and explicitopenAppend/openROstatic imports. Prefer one style; given the wildcard, the two explicit imports are redundant.import static io.questdb.cairo.TableUtils.*; -import static io.questdb.cairo.TableUtils.openAppend; -import static io.questdb.cairo.TableUtils.openRO;core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLog.java (1)
343-360: Unify errno source and enrich context in mmap failure pathPrefer
ff.errno()(used elsewhere in this PR) overOs.errno()and includetxnMetaOffsetin the error to aid diagnosis.- } else { - close(); - throw CairoException.critical(Os.errno()) - .put("cannot mmap table transaction log [path=").put(path) - .put(", txnMetaOffsetHi=").put(txnMetaOffsetHi) - .put(']'); + } else { + close(); + throw CairoException.critical(ff.errno()) + .put("cannot mmap table transaction log [path=").put(path) + .put(", txnMetaOffsetLo=").put(txnMetaOffset) + .put(", txnMetaOffsetHi=").put(txnMetaOffsetHi) + .put(']');core/src/main/java/io/questdb/cairo/wal/CopyWalSegmentUtils.java (2)
158-163: Guard fsync on optional FDs
srcVarFd/dstVarFdcan be-1for fixed-size targets. Guard to avoid unnecessary syscalls/errors.- if (commitMode != CommitMode.NOSYNC) { - ff.fsync(srcFixFd); - ff.fsync(srcVarFd); - ff.fsync(dstFixFd); - ff.fsync(dstVarFd); - } + if (commitMode != CommitMode.NOSYNC) { + ff.fsync(srcFixFd); + if (srcVarFd > -1) ff.fsync(srcVarFd); + ff.fsync(dstFixFd); + if (dstVarFd > -1) ff.fsync(dstVarFd); + }
224-231: Defensive unmap checkGiven munmap now fails on address == 0, add a guard to avoid spurious failures if the returned address ever equals 0.
- try { + try { Vect.flattenIndex(srcDataTimestampAddr, rowCount); if (commitMode != CommitMode.NOSYNC) { ff.msync(srcDataTimestampAddr, size, commitMode == CommitMode.ASYNC); } } finally { - ff.munmap(srcDataTimestampAddr, size, MEMORY_TAG); + if (srcDataTimestampAddr > 0) { + ff.munmap(srcDataTimestampAddr, size, MEMORY_TAG); + } }core/src/main/java/io/questdb/cairo/TableUtils.java (1)
1108-1110: Add runtime guard before munmapAn
assertmay be disabled in production. Sincemunmap(0, ...)now fails, add a guard to avoid false-positive failures.- long adjustedAddress = address - alignedExtraLen; - assert adjustedAddress > 0 : "address <= alignedExtraLen"; - ff.munmap(adjustedAddress, size + alignedExtraLen, memoryTag); + long adjustedAddress = address - alignedExtraLen; + assert adjustedAddress > 0 : "address <= alignedExtraLen"; + if (adjustedAddress > 0) { + ff.munmap(adjustedAddress, size + alignedExtraLen, memoryTag); + }core/src/main/java/io/questdb/std/Files.java (1)
287-295: Include POSIX EACCES in isErrnoFileCannotRead()
No existing POSIX “permission denied” constant—define, for example,// in CairoException public static final int ERRNO_ACCESS_DENIED_POSIX = 13;then update Files.isErrnoFileCannotRead() to also check
|| errno == CairoException.ERRNO_ACCESS_DENIED_POSIXcore/src/test/java/io/questdb/test/FilesTest.java (3)
768-793: Avoid masking the root failure by calling munmap with len=0 in the success path.If
mmaperroneously succeeds, the subsequentmunmap(..., 0, ...)throws a different error and can obscure the diagnosis. LetAssert.failfire immediately aftermmapinstead.- long mmapAddr = Files.mmap(fdro, 0, 0, Files.MAP_RO, MemoryTag.MMAP_DEFAULT); - Files.munmap(mmapAddr, 0, MemoryTag.MMAP_DEFAULT); + long mmapAddr = Files.mmap(fdro, 0, 0, Files.MAP_RO, MemoryTag.MMAP_DEFAULT); Assert.fail("mmap with zero len should have failed");
795-827: Don’t overwrite the original mapping and drop the zero-len munmap.Overwriting
mmapAddrcomplicates cleanup; the zero-lengthmunmapsimilarly hides the intended failure. Keep the original address for cleanup and fail immediately ifmremapdoesn’t throw.- mmapAddr = Files.mremap(fdro, mmapAddr, 64, 0, 0, Files.MAP_RO, MemoryTag.MMAP_DEFAULT); - Files.munmap(mmapAddr, 0, MemoryTag.MMAP_DEFAULT); + long ignored = Files.mremap(fdro, mmapAddr, 64, 0, 0, Files.MAP_RO, MemoryTag.MMAP_DEFAULT); Assert.fail("mremap with len 0 should have failed");
869-885: Assert FDs are valid after openCleanRW.Minor robustness: assert the returned FDs are positive before using/closing.
- long fd = Files.openCleanRW(path.$(), 1024); + long fd = Files.openCleanRW(path.$(), 1024); + Assert.assertTrue(fd > 0); Assert.assertTrue(Files.exists(path.$())); Assert.assertEquals(1024, Files.length(path.$())); - long fd2 = Files.openCleanRW(path.$(), 2048); + long fd2 = Files.openCleanRW(path.$(), 2048); + Assert.assertTrue(fd2 > 0); Assert.assertEquals(2048, Files.length(path.$()));
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (5)
core/src/main/resources/io/questdb/bin/darwin-aarch64/libquestdb.dylibis excluded by!**/*.dylibcore/src/main/resources/io/questdb/bin/darwin-x86-64/libquestdb.dylibis excluded by!**/*.dylibcore/src/main/resources/io/questdb/bin/linux-aarch64/libquestdb.sois excluded by!**/*.socore/src/main/resources/io/questdb/bin/linux-x86-64/libquestdb.sois excluded by!**/*.socore/src/main/resources/io/questdb/bin/windows-x86-64/libquestdb.dllis excluded by!**/*.dll
📒 Files selected for processing (26)
.github/workflows/rebuild_native_libs.yml(1 hunks)core/src/main/c/share/dedup.cpp(1 hunks)core/src/main/c/windows/files.c(1 hunks)core/src/main/java/io/questdb/cairo/CairoException.java(2 hunks)core/src/main/java/io/questdb/cairo/O3PartitionJob.java(2 hunks)core/src/main/java/io/questdb/cairo/TableNameRegistryStore.java(2 hunks)core/src/main/java/io/questdb/cairo/TableUtils.java(5 hunks)core/src/main/java/io/questdb/cairo/TableWriter.java(5 hunks)core/src/main/java/io/questdb/cairo/wal/CheckWalTransactionsJob.java(1 hunks)core/src/main/java/io/questdb/cairo/wal/CopyWalSegmentUtils.java(2 hunks)core/src/main/java/io/questdb/cairo/wal/WalPurgeJob.java(1 hunks)core/src/main/java/io/questdb/cairo/wal/WalWriter.java(2 hunks)core/src/main/java/io/questdb/cairo/wal/seq/TableSequencerAPI.java(1 hunks)core/src/main/java/io/questdb/cairo/wal/seq/TableSequencerImpl.java(1 hunks)core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLog.java(2 hunks)core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogV1.java(2 hunks)core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogV2.java(2 hunks)core/src/main/java/io/questdb/cutlass/text/CopyTask.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/functions/catalogue/WalTableListFunctionFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/functions/catalogue/WalTransactionsFunctionFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/functions/table/ReadParquetFunctionFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/table/parquet/MappedMemoryPartitionDescriptor.java(1 hunks)core/src/main/java/io/questdb/std/Files.java(2 hunks)core/src/main/java/io/questdb/std/FilesFacadeImpl.java(1 hunks)core/src/main/java/io/questdb/std/MmapCache.java(6 hunks)core/src/test/java/io/questdb/test/FilesTest.java(5 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- core/src/main/java/io/questdb/griffin/engine/table/parquet/MappedMemoryPartitionDescriptor.java
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (27)
- GitHub Check: aarch64
- GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Other tests Start ARM Agent)
- GitHub Check: New pull request (SelfHosted Other tests Start X64Zfs Agent)
- GitHub Check: New pull request (Hosted Running tests on windows-other)
- GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin)
- GitHub Check: New pull request (Hosted Running tests on mac-other)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-other)
- GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-pgwire)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-cairo)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-fuzz)
- GitHub Check: New pull request (Hosted Running tests on mac-griffin)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start X64Zfs Agent)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-griffin)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start ARM Agent)
- GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
- GitHub Check: New pull request (Check Changes Check changes)
🔇 Additional comments (39)
core/src/main/c/share/dedup.cpp (1)
635-643: Good fix: add explicit false return to close the non-null scan.Prevents UB/compile error from falling off the end of a non-void lambda and aligns semantics with other is_not_null_lambda implementations in this file.
core/src/main/java/io/questdb/cutlass/text/CopyTask.java (1)
1342-1344: Guarded munmap looks goodAvoids invalid munmap on zero address.
core/src/main/java/io/questdb/griffin/engine/functions/table/ReadParquetFunctionFactory.java (1)
100-102: Guarded unmap in finally is correctPrevents calling munmap on zero address.
.github/workflows/rebuild_native_libs.yml (1)
33-35: Thanks for linking the runner-images issueThe inline link addresses a prior nit about discoverability.
core/src/main/c/windows/files.c (1)
703-711: Windows mmap: POSIX-aligned handling of len==0Early-return with ERROR_INVALID_PARAMETER for len==0 matches cross-platform behavior.
core/src/main/java/io/questdb/cairo/O3PartitionJob.java (2)
267-269: Guarded unmap in parquet read: goodPrevents invalid munmap calls.
2570-2572: Guarded unmap in index updater: goodConsistent with new munmap contract.
core/src/main/java/io/questdb/std/MmapCache.java (1)
287-289: LGTM: MAP_FAILED check for mmap accounting is consistent.core/src/main/java/io/questdb/griffin/engine/functions/catalogue/WalTableListFunctionFactory.java (1)
316-320: LGTM: switched toisFileCannotRead()for clearer intent.Consistent with the repo-wide move away from errno-based checks.
core/src/main/java/io/questdb/cairo/wal/seq/TableSequencerAPI.java (1)
152-161: LGTM: useisFileCannotRead()/isTableDropped()in catch.Matches the new CairoException helpers and preserves previous behavior.
core/src/main/java/io/questdb/griffin/engine/functions/catalogue/WalTransactionsFunctionFactory.java (1)
118-118: Switch to CairoException.isFileCannotRead(): LGTMConsistent with the PR-wide predicate rename; behavior preserved.
core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogV1.java (1)
372-379: MAP_FAILED checks: LGTMUsing
FilesFacade.MAP_FAILEDformmap/mremapfailure detection aligns with cross-platform semantics introduced in this PR.Also applies to: 391-395
core/src/main/java/io/questdb/cairo/wal/seq/TableSequencerImpl.java (2)
128-135: Open-path errno predicate migration: LGTMReplacing
errnoFileCannotRead()withisFileCannotRead()keeps the intended drop-detection logic intact.
120-141: No stale predicates or legacy mmap checks detected
Repository-wide scan found zero occurrences oferrnoFileCannotReador manualmmap/mremapcomparisons against –1.core/src/main/java/io/questdb/std/FilesFacadeImpl.java (1)
429-431: Rename-consistency: switched to Files.isErrnoFileDoesNotExist() — LGTMMatches the new helper naming and preserves semantics.
core/src/main/java/io/questdb/cairo/TableNameRegistryStore.java (3)
359-365: Use of CairoException.isFileCannotRead(): correct behavior retainedProperly treats transient file-read/rename/drop scenarios as non-fatal during root dir scan.
433-441: RO-mode reload: correct switch to isFileCannotRead()Keeps the RO retry/continue logic intact when tables.d.* is swapped by RW instance.
359-365: No remaining deprecated helper usages found
Ran the suggested RG scan across the codebase (excluding target/build); no occurrences oferrnoFileCannotReadorerrnoFileDoesNotExistdetected.core/src/main/java/io/questdb/cairo/wal/WalWriter.java (2)
1759-1766: Always unmap in finally: good catchMoving munmap() to finally guarantees cleanup even on exceptions and aligns with stricter munmap preconditions.
1776-1783: Symmetric fix for data vector: LGTMSame robustness applied to the data file path; matches the aux path change.
core/src/main/java/io/questdb/cairo/TableWriter.java (4)
5670-5673: Good: guarded munmap for parquet mapping.The
parquetAddr != 0check prevents invalid unmap calls and matches the new strictermunmapsemantics.
8556-8558: Good: parquet unmap guard.Matches the new invariant and avoids spurious failures.
8668-8670: Good: parquet unmap guard here too.Consistent with other call sites.
8738-8739: Verified API rename; no deprecated calls remain
rgconfirms noerrnoFileCannotRead()usages in TableWriter.java, and allff.munmap()calls are properly guarded.core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLog.java (1)
41-41: Import looks good
Osis used in this file; no action needed.core/src/main/java/io/questdb/cairo/wal/CopyWalSegmentUtils.java (1)
27-43: Import changes are fineExplicit imports improve clarity; no action needed.
core/src/main/java/io/questdb/cairo/CairoException.java (2)
45-47: Adding platform errno constants is correct
ERRNO_INVALID_PARAMETERand Windows equivalent are accurate and useful for cross-platform checks.
249-251: API rename aligns with Files helpers
isFileCannotRead()delegating toFiles.isErrnoFileCannotRead(errno)is consistent with the rest of the PR.core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogV2.java (2)
486-493: Safe munmap guard in closePart()Only unmapping when
address > 0matches the new failure semantics.
524-532: MAP_FAILED check and detailed exception are spot-onCapturing
mmapresult, checkingMAP_FAILED, and throwing withpathandmapSizeis correct and consistent.core/src/main/java/io/questdb/cairo/TableUtils.java (3)
858-871: Switch toex.isFileCannotRead()looks goodMatches the new CairoException API and Files helpers.
1191-1199: Use of MAP_FAILED sentinel is correctReplacing
addr > -1withaddr != FilesFacade.MAP_FAILEDis the right failure check.
1297-1314: Error classification updated correctlySwitch to
Files.isErrnoFileCannotRead(errno)keeps the logic consistent with renamed helpers.core/src/main/java/io/questdb/std/Files.java (1)
128-131: OS-specific invalid-parameter errno helper: LGTM.Clearer than sprinkling conditional constants at call sites.
core/src/test/java/io/questdb/test/FilesTest.java (5)
28-28: Import CairoException: LGTM.Needed for the new exception-based mmap/mremap/munmap assertions.
65-65: Static importassertContains: LGTM.Keeps intent concise in failure assertions.
830-839: Zero address munmap test: LGTM.Accurately asserts the new guard behavior and message.
842-866: Zero length munmap test: LGTM.Covers the “address or length” validation path and cleans up properly.
886-898: openCleanRW on directory negative FD test: LGTM.Validates error propagation without leaking resources.
jerrinot
left a comment
There was a problem hiding this comment.
let the fuzzing commence!
core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogV2.java
Outdated
Show resolved
Hide resolved
[PR Coverage check]😍 pass : 112 / 141 (79.43%) file detail
|
mmap
PR makes every call site checks the return value of
mmap, and captureserrnoinCairoException.On Windows, the
mmapequivalent acceptslen == 0with the meaning "map the whole file". In order to maintain cross-platform consistency, make ourmmap0native function checklen == 0and return -1 instead, settingerrno = ERROR_INVALID_PARAMETER.PR adds an early check for
len < 0, throwingCairoExceptionif so.mremap
Prior to this PR,
mremap()was already checking fornewSize < 0and throwingCairoException. PR adds a proper test for this.PR also pushes the check deeper down the call stack, into
MmapCache, in order to ensure all calls toMmapCache.mremap()get the check.munmap
Prior to this PR,
munmapwould return silently when called withaddress == 0, masking potential bugs. The PR makesmunmapfail in that case, and ensures all call sites checkaddressfirst.General
MAP_FAILED. At various places there were== -1and< 0checks.munmapintofinallyto ensure cleanupex.errnoFileCannotRead->ex.isFileCannotReadFollowups
ff.close(fd)