fix(core): breaking change 💥 - remove tx scoreboard v1#6449
fix(core): breaking change 💥 - remove tx scoreboard v1#6449bluestreak01 merged 8 commits intomasterfrom
Conversation
This change removes transaction scoreboard v1. Deployments explicitly configured with cairo.txn.scoreboard.format=1 will continue to function, but the option will be ignore and the server will use scoreboard v2.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThe pull request removes TxnScoreboardPoolV1 and TxnScoreboardPoolFactory, consolidating transaction scoreboard handling to use only TxnScoreboardPoolV2. Configuration properties and tests that supported multiple scoreboard format versions are eliminated. Checkpoint-based locking guards on partition purge and squash operations are removed, allowing these operations to proceed during checkpoints. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai please review |
|
@jerrinot: I'll review the changes in this PR for you. ✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (6)
core/src/test/java/io/questdb/test/cairo/TxnScoreboardTest.java (6)
51-52: Remove unused Parameterized imports.These imports are no longer needed since the test class was converted from parameterized tests to V2-only tests.
Apply this diff:
-import org.junit.runner.RunWith; -import org.junit.runners.Parameterized;
494-500: Simplify by removing dead V1 branch.Since only
TxnScoreboardV2instances are created in this test, the V1 branch is unreachable dead code.Apply this diff:
private static long getActiveReaderCount(TxnScoreboard scoreboard, long txn) { - if (scoreboard instanceof TxnScoreboardV2) { - return ((TxnScoreboardV2) scoreboard).getActiveReaderCount(txn); - } else { - return ((TxnScoreboardV1) scoreboard).getActiveReaderCount(txn); - } + return ((TxnScoreboardV2) scoreboard).getActiveReaderCount(txn); }
504-510: Simplify by removing dead V1 branch.Since only
TxnScoreboardV2instances are created in this test, the V1 branch is unreachable dead code.Apply this diff:
private static long getMin(TxnScoreboard scoreboard) { - if (scoreboard instanceof TxnScoreboardV2) { - return ((TxnScoreboardV2) scoreboard).getMin(); - } else { - return ((TxnScoreboardV1) scoreboard).getMin(); - } + return ((TxnScoreboardV2) scoreboard).getMin(); }
512-518: Simplify by removing dead V1 branch.Since only
TxnScoreboardV2instances are created in this test, the V1 branch is unreachable dead code.Apply this diff:
private void assertScoreboardMinOrNoLocks(TxnScoreboard scoreboard, long txn) { - if (scoreboard instanceof TxnScoreboardV2) { - Assert.assertEquals(-1, getMin(scoreboard)); - } else { - Assert.assertEquals(txn, getMin(scoreboard)); - } + Assert.assertEquals(-1, getMin(scoreboard)); }
619-635: Remove unused version field from Writer class.The
versionfield is always set to2(line 538) and only gates V1-specific logic that is now dead code.Apply this diff to the Writer constructor and field:
private static class Writer extends Thread { private final AtomicInteger anomaly; private final CyclicBarrier barrier; private final int iterations; private final CountDownLatch latch; private final TxnScoreboard scoreboard; - private final int version; - private Writer(TxnScoreboard scoreboard, CyclicBarrier barrier, CountDownLatch latch, AtomicInteger anomaly, int iterations, int version) { + private Writer(TxnScoreboard scoreboard, CyclicBarrier barrier, CountDownLatch latch, AtomicInteger anomaly, int iterations) { this.scoreboard = scoreboard; this.barrier = barrier; this.latch = latch; this.anomaly = anomaly; this.iterations = iterations; - this.version = version; }
637-696: Remove V1-specific dead code from Writer.run().Lines 647-669 contain V1-specific logic that never executes since
versionis always2. This wait logic for slow readers is specific to V1 scoreboard limitations.Apply this diff to remove the dead code:
barrier.await(); for (int i = 0; i < iterations; i++) { - if (version == 1) { - for (int sleepCount = 0; sleepCount < 50 && txn - getMin(scoreboard) > publishWaitBarrier; sleepCount++) { - // Some readers are slow and haven't released transaction yet. Give them a bit more time - LOG.infoW().$("slow reader release, waiting... [txn=") - .$(txn) - .$(", min=").$(getMin(scoreboard)) - .$(", size=").$(scoreboard.getEntryCount()) - .I$(); - Os.sleep(100); - } - - if (txn - getMin(scoreboard) > publishWaitBarrier) { - // Wait didn't help. Abort the test. - anomaly.addAndGet(1000); - LOG.errorW().$("slow reader release, abort [txn=") - .$(txn) - .$(", min=").$(getMin(scoreboard)) - .$(", size=").$(scoreboard.getEntryCount()) - .I$(); - txn = iterations; - return; - } - } - // This is the only writerAlso update the Writer instantiation at line 538:
-Writer writer = new Writer(scoreboard, barrier, latch, anomaly, iterations, 2); +Writer writer = new Writer(scoreboard, barrier, latch, anomaly, iterations);
🧹 Nitpick comments (2)
core/src/test/java/io/questdb/test/cutlass/http/IODispatcherTest.java (1)
7575-7583: Directly constructing TxnScoreboardPoolV2 for TableReader is functionally soundBoth helper methods now pass a
TxnScoreboardPoolV2instance intoTableReader, which correctly satisfies theTxnScoreboardPooldependency and mirrors the behavior previously obtained viaTxnScoreboardPoolFactory.createPool(configuration). This keeps the tests aligned with the new v2‑only scoreboard implementation.If you ever want to micro‑optimize, you could reuse a single
TxnScoreboardPoolV2perconfigurationinstead of allocating a new one per helper call, but for tests this is purely optional.Also applies to: 7592-7602
core/src/main/java/io/questdb/PropServerConfiguration.java (1)
231-232: SQL HTTP context path sets aligned with existing patternDefining dedicated
httpContextPathSqlExecuteandhttpContextPathSqlValidatesets here matches how other HTTP context paths are handled and keeps the configuration surface consistent. No functional issues spotted. One minor design point: the validation set is only populated with a hard-coded default later in the ctor; if you ever want to make the SQL validation endpoint configurable like the others, you’d likely want agetUrls(...)call for it as well.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (25)
benchmarks/src/main/java/org/questdb/TableReaderReloadBenchmark.java(2 hunks)core/src/main/java/io/questdb/PropServerConfiguration.java(3 hunks)core/src/main/java/io/questdb/cairo/CairoConfiguration.java(0 hunks)core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java(0 hunks)core/src/main/java/io/questdb/cairo/CairoEngine.java(1 hunks)core/src/main/java/io/questdb/cairo/ColumnPurgeJob.java(0 hunks)core/src/main/java/io/questdb/cairo/DatabaseCheckpointAgent.java(1 hunks)core/src/main/java/io/questdb/cairo/DatabaseCheckpointStatus.java(0 hunks)core/src/main/java/io/questdb/cairo/DefaultCairoConfiguration.java(0 hunks)core/src/main/java/io/questdb/cairo/O3PartitionPurgeJob.java(0 hunks)core/src/main/java/io/questdb/cairo/TableWriter.java(0 hunks)core/src/main/java/io/questdb/cairo/TxnScoreboardPoolFactory.java(0 hunks)core/src/main/java/io/questdb/cairo/TxnScoreboardPoolV1.java(0 hunks)core/src/main/resources/io/questdb/site/conf/server.conf(0 hunks)core/src/test/java/io/questdb/test/PropServerConfigurationTest.java(0 hunks)core/src/test/java/io/questdb/test/ServerMainTest.java(0 hunks)core/src/test/java/io/questdb/test/cairo/TableReaderTest.java(0 hunks)core/src/test/java/io/questdb/test/cairo/TxnScoreboardPoolTest.java(0 hunks)core/src/test/java/io/questdb/test/cairo/TxnScoreboardTest.java(3 hunks)core/src/test/java/io/questdb/test/cairo/fuzz/CheckpointFuzzTest.java(0 hunks)core/src/test/java/io/questdb/test/cairo/o3/O3PartitionPurgeTest.java(0 hunks)core/src/test/java/io/questdb/test/cutlass/http/IODispatcherTest.java(3 hunks)core/src/test/java/io/questdb/test/griffin/CheckpointTest.java(0 hunks)core/src/test/resources/server.conf(0 hunks)pkg/ami/marketplace/assets/server.conf(0 hunks)
💤 Files with no reviewable changes (19)
- core/src/test/resources/server.conf
- core/src/main/java/io/questdb/cairo/CairoConfiguration.java
- core/src/main/java/io/questdb/cairo/O3PartitionPurgeJob.java
- core/src/main/java/io/questdb/cairo/DefaultCairoConfiguration.java
- core/src/test/java/io/questdb/test/cairo/TableReaderTest.java
- core/src/test/java/io/questdb/test/cairo/o3/O3PartitionPurgeTest.java
- core/src/test/java/io/questdb/test/ServerMainTest.java
- core/src/main/java/io/questdb/cairo/TxnScoreboardPoolFactory.java
- core/src/test/java/io/questdb/test/PropServerConfigurationTest.java
- core/src/test/java/io/questdb/test/cairo/fuzz/CheckpointFuzzTest.java
- core/src/main/java/io/questdb/cairo/TxnScoreboardPoolV1.java
- pkg/ami/marketplace/assets/server.conf
- core/src/test/java/io/questdb/test/cairo/TxnScoreboardPoolTest.java
- core/src/test/java/io/questdb/test/griffin/CheckpointTest.java
- core/src/main/java/io/questdb/cairo/DatabaseCheckpointStatus.java
- core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java
- core/src/main/resources/io/questdb/site/conf/server.conf
- core/src/main/java/io/questdb/cairo/ColumnPurgeJob.java
- core/src/main/java/io/questdb/cairo/TableWriter.java
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-11-07T00:59:31.522Z
Learnt from: bluestreak01
Repo: questdb/questdb PR: 0
File: :0-0
Timestamp: 2025-11-07T00:59:31.522Z
Learning: In QuestDB's Cairo engine, transaction (_txn) files have a strong invariant: they are never truncated below TX_BASE_HEADER_SIZE. Once created, they are either fully formed (size >= header size) or completely removed along with the entire table directory when the table is dropped.
Applied to files:
core/src/test/java/io/questdb/test/cutlass/http/IODispatcherTest.java
🧬 Code graph analysis (4)
core/src/main/java/io/questdb/cairo/DatabaseCheckpointAgent.java (2)
core/src/main/c/share/txn_board.cpp (8)
txn(55-79)txn(55-55)txn(104-114)txn(104-104)txn(117-161)txn(117-117)txn(164-170)txn(164-164)core/src/main/java/io/questdb/std/Numbers.java (1)
Numbers(44-3523)
benchmarks/src/main/java/org/questdb/TableReaderReloadBenchmark.java (1)
core/src/main/java/io/questdb/cairo/TxnScoreboardPoolV2.java (1)
TxnScoreboardPoolV2(36-131)
core/src/test/java/io/questdb/test/cairo/TxnScoreboardTest.java (1)
core/src/main/java/io/questdb/cairo/TxnScoreboardPoolV2.java (1)
TxnScoreboardPoolV2(36-131)
core/src/test/java/io/questdb/test/cutlass/http/IODispatcherTest.java (1)
core/src/main/java/io/questdb/cairo/TxnScoreboardPoolV2.java (1)
TxnScoreboardPoolV2(36-131)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (35)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-other)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-pgwire)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-cairo-sub)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-cairo-root)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-fuzz2)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-fuzz1)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-griffin-sub)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x86-graal)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-griffin-root)
- GitHub Check: New pull request (SelfHosted Cairo tests on linux-x86-graal)
- GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
- GitHub Check: New pull request (Hosted Running tests on windows-other-2)
- GitHub Check: New pull request (Hosted Running tests on windows-other-1)
- GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo-2)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo-1)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin-sub)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin-base)
- GitHub Check: New pull request (Hosted Running tests on mac-other)
- GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
- GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo)
- GitHub Check: New pull request (Hosted Running tests on mac-griffin)
- GitHub Check: New pull request (SelfHosted Cairo tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Griffin tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Cairo tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Griffin tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Griffin tests on linux-x86-graal)
- GitHub Check: New pull request (Check Changes Check changes)
- GitHub Check: build
- GitHub Check: build
🔇 Additional comments (7)
core/src/main/java/io/questdb/cairo/CairoEngine.java (1)
196-196: LGTM - Clean migration to V2 complete.The direct instantiation of
TxnScoreboardPoolV2is correct and aligns with the PR objective to remove scoreboard V1. Verification confirms:
- Factory and V1 classes completely removed
- Configuration property properly registered as deprecated (silently ignored, not erroring)
- No production code reads the deprecated format value
- Tests appropriately parameterized for compatibility testing
- EmptyTxnScoreboardPool is a separate stub implementation for COPY SQL operations (unrelated to V1/V2 migration)
core/src/test/java/io/questdb/test/cairo/TxnScoreboardTest.java (1)
75-75: LGTM - Correctly migrated to V2.The direct instantiation of TxnScoreboardPoolV2 correctly implements the V1 removal objective.
core/src/test/java/io/questdb/test/cutlass/http/IODispatcherTest.java (1)
43-43: TxnScoreboardPoolV2 import is correct and consistent with the PR directionThe new import matches the updated usages below and aligns with removing the v1/factory path. No issues here.
benchmarks/src/main/java/org/questdb/TableReaderReloadBenchmark.java (1)
36-36: LGTM! Import correctly updated to V2.The import change from
TxnScoreboardPoolFactorytoTxnScoreboardPoolV2aligns with the PR objective to consolidate transaction scoreboard handling to use only V2.core/src/main/java/io/questdb/PropServerConfiguration.java (2)
2638-2685: Deprecation ofcairo.txn.scoreboard.formatcorrectly preserves compatibilityRegistering
PropertyKey.CAIRO_TXN_SCOREBOARD_FORMATas deprecated (with no replacements) causes it to be recognized and reported as “No longer used” without being treated as obsolete or invalid. GivenvalidateProperties()only throws whenisErroris true (i.e., for incorrect/obsolete keys), this ensures:
- Existing configs using
cairo.txn.scoreboard.formatcontinue to start even withCONFIG_VALIDATION_STRICT=true.- Operators still get an advisory log telling them the setting is no longer used.
This matches the PR intent to ignore the v1/v2 switch while remaining backward compatible.
4688-4691: NewgetContextPathSqlValidation()override cleanly exposes validation endpointsWiring
getContextPathSqlValidation()tohttpContextPathSqlValidatebrings the SQL-validation paths in line with the other HTTP context path accessors and makes the contract explicit onHttpFullFatServerConfiguration. Implementation is straightforward and consistent with the surrounding getters.core/src/main/java/io/questdb/cairo/DatabaseCheckpointAgent.java (1)
316-328: LGTM! Scoreboard locking now unconditionally applied.The removal of the conditional check ensures that checkpoint operations always use scoreboard v2 transaction locking, which aligns with the PR objective. The logic correctly:
- Acquires a txn lock via
scoreboard.incrementTxn()- Stores the txn and associated metadata for later release
- Uses proper encoding (
Numbers.encodeLowHighInts) that matches the decoding inreleaseScoreboardTxns()The error handling at line 318-320 ensures that if the scoreboard lock cannot be acquired, the checkpoint fails gracefully and resources are cleaned up via the catch block at line 378.
core/src/test/java/io/questdb/test/cairo/TxnScoreboardTest.java
Outdated
Show resolved
Hide resolved
|
@jerrinot please submit Ent tandem PR and link in the description |
|
@jerrinot we can probably remove |
|
@bluestreak01 there is a dummy impl use in SQL COPY: |
|
it is subclassing a class i think, not the interface |
|
@jerrinot you're right, copy is using it |
[PR Coverage check]😍 pass : 14 / 15 (93.33%) file detail
|
This change removes transaction scoreboard v1.
Deployments explicitly configured with cairo.txn.scoreboard.format=1 will continue to function, but the option will be ignore and the server will use scoreboard v2.
tandem: https://github.com/questdb/questdb-enterprise/pull/798