Skip to content

feat(sql): add exporting PARQUET functionality to COPY#6008

Merged
ideoma merged 226 commits intomasterfrom
feat-copy-parquet
Oct 20, 2025
Merged

feat(sql): add exporting PARQUET functionality to COPY#6008
ideoma merged 226 commits intomasterfrom
feat-copy-parquet

Conversation

@nwoolmer
Copy link
Copy Markdown
Contributor

@nwoolmer nwoolmer commented Jul 31, 2025

Related to questdb/roadmap#97

Closes #6278

Currently, to export PARQUET files, you must convert partitions in-place from NATIVE to PARQUET format. This is inflexible, and blocks writes whilst it occurs.

This PR adds support for exporting parquet files via COPY, improving ergonomics and making it easier for users to extract data as parquet files.

Syntax

-- export an entire table to parquet, with same partitioning as original
COPY 'table_name' TO 'folder_name' WITH FORMAT PARQUET;

-- export a subquery to parquet
COPY (SELECT * FROM trades WHERE timestamp IN today()) TO 'folder_name' WITH FORMAT PARQUET;

Changelist

  • add new TO option to COPY
  • add job to handle export requests
    • add new copy_export_log table
  • Add single-threaded parquet partition writer.
  • COPY 'table_name' TO 'folder_name' WITH FORMAT PARQUET
    • whole-table copy
  • COPY (SELECT * FROM x) TO 'folder_name' WITH FORMAT PARQUET
    • arbitrary query copy
  • COPY options:
    • FORMAT PARQUET
    • PARTITION_BY DAY
      • output partitioning, if desired (for subquery variant)
    • COMPRESSION_CODEC UNCOMPRESSED/ZSTD/etc.
    • COMPRESSION_LEVEL N
    • ROW_GROUP_SIZE N
    • DATA_PAGE_SIZE N
    • STATISTICS_ENABLED true
    • PARQUET_VERSION N
    • RAW_ARRAY_ENCODING true
  • support exporting parquet over HTTP via /exp
    • /exp?query=<query>&fmt=parquet provides a parquet file in response instead of csv
    • &partition_by= not supported yet
    • &compression_codec=LZ4_RAW
    • &compression_level=3
    • &row_group_size=100000
    • &data_page_size=1048576
    • &statistics_enabled=true
    • &parquet_version=1
    • &raw_array_encoding=true
  • unit tests
  • fuzz tests
  • docs
  • syntax highlighting
  • refactor copy import (text import) for consistency
  • tandem PR
@coderabbitai ignore

@nwoolmer nwoolmer added SQL Issues or changes relating to SQL execution Core Related to storage, data type, etc. labels Jul 31, 2025
@coderabbitai

This comment was marked as spam.

@ideoma
Copy link
Copy Markdown
Collaborator

ideoma commented Oct 20, 2025

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Oct 20, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

ideoma
ideoma previously approved these changes Oct 20, 2025
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 26

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (9)
core/src/test/java/io/questdb/test/cutlass/http/HttpConnectionCountTest.java (1)

1-1: Add test coverage for /exp (export) endpoint returning 429 on connection limit breach

The export processor has connection limit infrastructure (HTTP_EXPORT_CONNECTION_LIMIT config and ActiveConnectionTracker enforcement), but lacks test coverage. Add a test method similar to testIlpConnectionLimit() and testQueryConnectionLimit() to verify that /exp returns 429 when the export connection limit is exceeded.

core/src/main/java/io/questdb/cairo/sql/NetworkSqlExecutionCircuitBreaker.java (1)

55-73: Add @NotNull annotation and runtime null-check for engine parameter to enforce fail-fast semantics

The engine field is dereferenced in checkIfTripped(), getState(), and testCancelled() methods without null-safety guards. While all current call sites provide non-null values, the constructor parameter lacks @NotNull annotation and runtime validation. Apply defensive programming:

-    public NetworkSqlExecutionCircuitBreaker(CairoEngine engine, @NotNull SqlExecutionCircuitBreakerConfiguration configuration, int memoryTag) {
+    public NetworkSqlExecutionCircuitBreaker(@NotNull CairoEngine engine, @NotNull SqlExecutionCircuitBreakerConfiguration configuration, int memoryTag) {
         this.configuration = configuration;
         this.nf = configuration.getNetworkFacade();
         this.throttle = configuration.getCircuitBreakerThrottle();
         this.bufferSize = configuration.getBufferSize();
         this.memoryTag = memoryTag;
         this.buffer = Unsafe.malloc(this.bufferSize, this.memoryTag);
         this.clock = configuration.getClock();
         long timeout = configuration.getQueryTimeout();
         if (timeout > 0) {
             this.timeout = timeout;
         } else if (timeout == TIMEOUT_FAIL_ON_FIRST_CHECK) {
             this.timeout = -100;
         } else {
             this.timeout = Long.MAX_VALUE;
         }
         this.defaultMaxTime = this.timeout;
-        this.engine = engine;
+        this.engine = java.util.Objects.requireNonNull(engine, "engine");
     }
core/src/main/java/io/questdb/griffin/engine/ops/CopyImportFactory.java (1)

99-121: Always release SPSequence slot; handle queue-full (avoid stuck import queue).

If an exception occurs between next() and done(), the slot is leaked. Also, asserting on the cursor (> -1) is not a runtime guard. Wrap publish in try/finally and error on -1.

-            try {
-                circuitBreaker.reset();
-                final SPSequence copyRequestPubSeq = messageBus.getCopyImportRequestPubSeq();
-                long processingCursor = copyRequestPubSeq.next();
-                assert processingCursor > -1;
-                final CopyImportRequestTask task = copyImportRequestQueue.get(processingCursor);
-                task.of(
+            try {
+                circuitBreaker.reset();
+                final SPSequence copyRequestPubSeq = messageBus.getCopyImportRequestPubSeq();
+                long processingCursor = -1;
+                try {
+                    processingCursor = copyRequestPubSeq.next();
+                    if (processingCursor == -1) {
+                        throw SqlException.$(0, "unable to process the import request - import queue is full");
+                    }
+                    final CopyImportRequestTask task = copyImportRequestQueue.get(processingCursor);
+                    task.of(
                         executionContext.getSecurityContext(),
                         copyID,
                         tableName,
                         fileName,
                         headerFlag,
                         timestampColumn,
                         delimiter,
                         timestampFormat,
                         partitionBy,
                         atomicity
-                );
-                copyRequestPubSeq.done(processingCursor);
+                    );
+                } finally {
+                    if (processingCursor >= 0) {
+                        copyRequestPubSeq.done(processingCursor);
+                    }
+                }
                 record.setValue(importIdSink);
                 cursor.toTop();
                 return cursor;
core/src/main/java/io/questdb/cutlass/http/HttpConnectionContext.java (1)

906-910: Connection-limit accounting sticks across keep‑alive; switch slot when processor changes.

A keep‑alive connection may serve different processors over time. Because connectionCounted stays true and processorName isn’t updated, counts can be held under the wrong processor and limits not enforced for the new one.

-                if (!connectionCounted && !processor.ignoreConnectionLimitCheck()) {
-                    processor = checkConnectionLimit(processor);
-                    connectionCounted = true;
-                }
+                if (!processor.ignoreConnectionLimitCheck()) {
+                    final String currentName = processor.getName();
+                    // If we already counted but processor changed, release old slot first.
+                    if (connectionCounted && processorName != null && !java.util.Objects.equals(processorName, currentName)) {
+                        decrementActiveConnections(getFd()); // sets connectionCounted=false and clears processorName
+                    }
+                    if (!connectionCounted) {
+                        processor = checkConnectionLimit(processor);
+                        connectionCounted = true;
+                    }
+                }
core/src/main/java/io/questdb/cutlass/text/CopyImportRequestJob.java (1)

171-179: Fix status logging: wrong variable used for status.

Uses getStatusName(phase) instead of status; log shows wrong status.

-                        .$(", phase=").$(getPhaseName(phase))
-                        .$(", status=").$(getStatusName(phase))
+                        .$(", phase=").$(getPhaseName(phase))
+                        .$(", status=").$(getStatusName(status))
core/src/main/java/io/questdb/cairo/CairoEngine.java (2)

471-493: Engine shutdown: signal and free COPY contexts; prevent leaks.

close() never signals shutdown nor frees copyExportContext/copyImportContext. Risk of thread/queue/resource leaks.

 public void close() {
+        // Signal shutdown early so dependent services/jobs stop cleanly
+        signalClose();
         Misc.free(sqlCompilerPool);
@@
         Misc.free(settingsStore);
         Misc.free(frameFactory);
+        // Release COPY contexts
+        Misc.free(copyExportContext);
+        Misc.free(copyImportContext);
 }

451-472: clear(): also clear the import context.

Only copyExportContext.clear() is called.

-        copyExportContext.clear();
+        copyExportContext.clear();
+        copyImportContext.clear();
core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessorState.java (1)

124-129: close() should release parquet resources too.

Currently only cursor/factory are freed. Call cleanupParquetState() to avoid fd/mapping/temp path leaks when callers invoke close() without clear().

     public void close() {
         cursor = Misc.free(cursor);
         recordCursorFactory = Misc.free(recordCursorFactory);
+        cleanupParquetState();
     }
core/src/main/java/io/questdb/griffin/SqlCompilerImpl.java (1)

663-729: Inconsistent progress reporting frequency for string timestamp columns.

The copyOrderedBatchedStrTimestamp and copyOrderedBatchedVarcharTimestamp methods hardcode the report frequency to 50000 (lines 688, 723), while copyOrderedBatched0 uses a configurable reportFrequency parameter. This creates inconsistent behavior where progress reporting frequency cannot be configured when the timestamp column is a string or varchar type.

Consider accepting reportFrequency as a parameter in these methods for consistency:

 private static long copyOrderedBatchedStrTimestamp(
         TableWriterAPI writer,
         RecordCursor cursor,
         RecordToRowCopier copier,
         int cursorTimestampIndex,
         long batchSize,
         long o3MaxLag,
         SqlExecutionCircuitBreaker circuitBreaker,
-        CopyDataProgressReporter reporter
+        CopyDataProgressReporter reporter,
+        int reportFrequency
 ) {
     // ... existing code ...
-    if (rowCount % 50000 == 0) {
+    if (reportFrequency > 0 && rowCount % reportFrequency == 0) {
         reporter.onProgress(CopyDataProgressReporter.Stage.Inserting, rowCount);
     }

Apply the same pattern to copyOrderedBatchedVarcharTimestamp.

♻️ Duplicate comments (7)
core/src/main/java/io/questdb/griffin/engine/table/parquet/ParquetCompression.java (1)

42-43: GZIP max level off-by-one.

GZIP compression supports levels 0–9, not 0–10. This issue was already flagged in a previous review.

core/src/main/java/io/questdb/griffin/CopyDataProgressReporter.java (1)

34-38: Enum constants declared with () will not compile; remove parentheses.

This is a hard compile error unless the enum declares a matching constructor. Also, this was flagged earlier and remains unresolved. Fix to plain constants.

Apply:

-    enum Stage {
-        Start(),
-        Inserting(),
-        Finish(),
-    }
+    enum Stage {
+        Start,
+        Inserting,
+        Finish
+    }
core/src/main/java/io/questdb/ServerMain.java (1)

356-361: Read-only gating for QueryTracingJob acknowledged

This aligns with prior discussion to allow startup in read-only mode.

core/src/main/java/io/questdb/cutlass/parquet/SerialParquetExporter.java (1)

382-389: mkdirs on file path ('.parquet') creates a directory named like the file.

Single-file export calls mkdirs on the file path, which can break rename.

-        this.exportPath.put(toParquet.$());
-        createDirsOrFail(ff, toParquet, configuration.getMkDirMode());
+        this.exportPath.put(toParquet.$());
+        // ensure parent directory exists, not the file path
+        toParquet.slash();
+        createDirsOrFail(ff, toParquet, configuration.getMkDirMode());
+        toParquet.chopZ();
core/src/main/java/io/questdb/cutlass/text/CopyExportContext.java (2)

147-165: Fix authorization target and avoid calling external code under the read lock.

Authorize against the task owner’s SecurityContext and perform authorization/cancel outside the lock. Current code self‑authorizes and calls into external code while holding the read lock.

-    public boolean cancel(long id, SecurityContext securityContext) {
-        lock.readLock().lock();
-        try {
-            ExportTaskEntry e = activeExports.get(id);
-            if (e != null) {
-                var cb = e.getCircuitBreaker();
-                if (cb != null) {
-                    if (securityContext != null) {
-                        securityContext.authorizeCopyCancel(securityContext);
-                    }
-                    cb.cancel();
-                }
-                return true;
-            }
-            return false;
-        } finally {
-            lock.readLock().unlock();
-        }
-    }
+    public boolean cancel(long id, SecurityContext securityContext) {
+        SecurityContext owner = null;
+        SqlExecutionCircuitBreaker cb = null;
+        lock.readLock().lock();
+        try {
+            ExportTaskEntry e = activeExports.get(id);
+            if (e == null) {
+                return false;
+            }
+            owner = e.getSecurityContext();
+            cb = e.getCircuitBreaker();
+        } finally {
+            lock.readLock().unlock();
+        }
+        if (securityContext != null && owner != null) {
+            owner.authorizeCopyCancel(securityContext);
+        }
+        if (cb != null) {
+            cb.cancel();
+        }
+        return true;
+    }

267-279: Prevent use‑after‑release: release to pool after fields are fully reset, not inside releaseEntry().

Move pool release into ExportTaskEntry.clear() after zeroing all fields; releaseEntry() should only unregister from registries under the write lock. This avoids handing out a stale instance to another thread.

@@
     public void releaseEntry(ExportTaskEntry entry) {
         lock.writeLock().lock();
         try {
             activeExports.remove(entry.id);
-            exportBySqlText.remove(entry.sqlText);
+            exportBySqlText.remove(entry.sqlText);
             if (!entry.fileName.isEmpty()) {
                 exportEntriesByFileName.remove(entry.fileName);
             }
-            exportTaskEntryPools.release(entry);
         } finally {
             lock.writeLock().unlock();
         }
     }
@@ public static class ExportTaskEntry implements Mutable {
-        @Override
-        public void clear() {
+        @Override
+        public void clear() {
             this.securityContext = null;
             if (atomicBooleanCircuitBreaker != null) {
                 atomicBooleanCircuitBreaker.clear();
             }
             this.id = INACTIVE_COPY_ID;
             this.sqlText.clear();
             this.fileName.clear();
             this.phase = null;
             this.startTime = Numbers.LONG_NULL;
             this.workerId = -1;
             this.populatedRowCount = 0;
             this.finishedPartitionCount = 0;
             realCircuitBreaker = null;
             this.totalPartitionCount = 0;
             this.totalRowCount = 0;
             this.trigger = CopyTrigger.NONE;
+            // Return to pool last to ensure a fully reset instance is reused.
+            // Requires enclosing context to hold write lock while calling clear().
+            // If clear() is called outside context lock, introduce a dedicated release() that acquires it.
+            // noinspection DataFlowIssue
+            // (exportTaskEntryPools is an outer class field)
+            CopyExportContext.this.exportTaskEntryPools.release(this);
         }

Also applies to: 493-511

core/src/main/java/io/questdb/griffin/SqlCompilerImpl.java (1)

1630-1636: Fail early when source table doesn't exist for COPY TO operations.

For COPY TO operations, the source table must exist. However, if the table doesn't exist, authorization is silently skipped (line 1633-1635) and the operation will fail later with a potentially confusing error. This was flagged in previous reviews but remains unresolved.

Apply this diff to fail early with a clear error message:

 private void authorizeSelectForCopy(SecurityContext securityContext, ExportModel model) {
     final CharSequence tableName = unquote(model.getTableName());
     final TableToken tt = engine.verifyTableName(tableName);
     if (tt != null) {
         securityContext.authorizeSelectOnAnyColumn(tt);
     } else {
-        // silently skip authorization
+        throw SqlException.tableDoesNotExist(model.getTableNameExpr().position, tableName);
     }
 }
🧹 Nitpick comments (42)
core/src/test/java/io/questdb/test/cutlass/pgwire/PGErrorHandlingTest.java (1)

66-66: Remove unused counter variable.

The counter variable is no longer used after refactoring from a counter-based trigger to a filename-based trigger.

Apply this diff to remove the unused variable:

-        final AtomicInteger counter = new AtomicInteger(0);
         final Bootstrap bootstrap = new Bootstrap(
core/src/test/java/io/questdb/test/cairo/RssMemoryLimitTest.java (2)

71-71: Good improvement to test specificity.

The change from waitFor to waitForRegex with an explicit memory tag value makes the test more precise and helps catch regressions in memory tagging behavior.

Consider adding a brief comment explaining what memory tag 45 represents (e.g., which allocation category or operation it corresponds to) to aid future maintainers.


125-125: Consistent improvement to test specificity.

The regex-based assertion correctly validates the memory tag for SELECT query operations. The pattern matches the change at line 71, maintaining consistency across the test suite.

As with line 71, a comment explaining the significance of memory tag 30 would improve maintainability.

core/src/main/java/io/questdb/cairo/mv/MatViewRefreshJob.java (1)

814-835: Excellent improvements to error handling and diagnostics.

This change adds several valuable enhancements:

  • Special handling for shutdown interruptions logs at INFO level (not ERROR) and returns gracefully—appropriate since cancellation during shutdown is not a failure.
  • Captures and conditionally logs errno from CairoException for better diagnostics, consistent with similar logging elsewhere in the file (lines 998–1005, 1419–1423).
  • Uses LogRecord for structured error logging.

Minor: the comment on line 817 says "because a questdb shutdown"; consider "due to questdb shutdown" for clarity.

Apply this diff to improve the comment:

-                    // The query was cancelled, because a questdb shutdown.
+                    // The query was cancelled due to questdb shutdown.
core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursorFactory.java (1)

42-51: Consider adding @NotNull annotation to the engine parameter for consistency.

All other object parameters in this constructor are annotated with @NotNull, but the new engine parameter is not. Adding the annotation would maintain consistency, improve static analysis, and make the contract explicit.

Apply this diff:

 public LatestByAllIndexedRecordCursorFactory(
-        CairoEngine engine,
+        @NotNull CairoEngine engine,
         @NotNull CairoConfiguration configuration,
         @NotNull RecordMetadata metadata,
         @NotNull PartitionFrameCursorFactory partitionFrameCursorFactory,
core/src/main/resources/io/questdb/site/conf/server.conf (2)

548-575: Consider consolidating duplicate Parquet configuration blocks.

The Parquet export settings (lines 548-575) and partition encoder settings (lines 579-600) are nearly identical, differing only in their prefix (cairo.parquet.export.* vs cairo.partition.encoder.parquet.*). This duplication creates maintenance overhead—any future Parquet configuration updates must be applied to both sections.

If these two use cases require different defaults, consider whether a shared base configuration with per-feature overrides would reduce duplication. If the duplication is intentional and necessary, adding a comment explaining why two separate configuration blocks exist would help future maintainers.

Also applies to: 579-600


576-577: Consider relocating HTTP export timeout setting.

The http.export.timeout setting is placed between two Parquet configuration blocks, which disrupts the logical grouping of Parquet-specific settings. Consider moving it to:

  • Near the http.export.connection.limit setting (line 76-78), or
  • To the beginning of the COPY export section (after line 533)

This would improve configuration discoverability and maintain better logical grouping.

core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursor.java (1)

70-70: Add @NotNull annotation to the engine parameter for consistency.

The engine parameter is used to construct AtomicBooleanCircuitBreaker and would cause a NullPointerException if null. For consistency with other constructor parameters (configuration, metadata, rows, prefixes) that are annotated with @NotNull, consider annotating the engine parameter as well.

Apply this diff to add the annotation:

     public LatestByAllIndexedRecordCursor(
-            CairoEngine engine,
+            @NotNull CairoEngine engine,
             @NotNull CairoConfiguration configuration,
             @NotNull @Transient RecordMetadata metadata,
             int columnIndex,
             @NotNull DirectLongList rows,
             @NotNull DirectLongList prefixes
     ) {
core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)

462-467: Consider inverting the condition for better readability.

The logic is correct but confusing: the comment mentions "special handling for access control errors," yet the if block handles NON-access errors (due to the negation), and access errors are in the else block.

Apply this diff to make the control flow match the comment and improve readability:

-                // Special handling for access control errors: to distinguish them from other parsing errors for better diagnostics
-                if (!Chars.startsWith(e.getFlyweightMessage(), "Access")) {
-                    LOG.error().$safe(e.getFlyweightMessage()).I$();
-                } else {
+                // Special handling for access control errors: to distinguish them from other parsing errors for better diagnostics
+                if (Chars.startsWith(e.getFlyweightMessage(), "Access")) {
                     LOG.error().$("failed to parse message [err: `").$safe(e.getFlyweightMessage()).$("`]").$();
+                } else {
+                    LOG.error().$safe(e.getFlyweightMessage()).I$();
                 }

This makes the if block directly handle the case mentioned in the comment (access errors), improving code clarity.

core/rust/qdb-core/src/col_type.rs (1)

290-290: Add space before colon in type annotation.

Rust style convention prefers a space before the colon in type annotations.

Apply this diff:

-        let flag_shifted:i32 = flag << 8;
+        let flag_shifted: i32 = flag << 8;
core/src/main/java/io/questdb/cutlass/http/processors/LineHttpTudCache.java (1)

225-226: Table kind passed on create: LGTM

Passing TABLE_KIND_REGULAR_TABLE aligns with the new engine API.

Optionally call securityContext.authorizeTableCreate(TableUtils.TABLE_KIND_REGULAR_TABLE) before create to fail fast at the edge (engine still rechecks).

core/src/test/java/io/questdb/test/cutlass/http/HttpConnectionCountTest.java (2)

258-261: Bound the active-connection spin to avoid test hangs

Add a timeout to the while-loop waiting for connection count to drop to prevent indefinite hangs under flakiness.

-while (serverMain.getActiveConnectionCount(ActiveConnectionTracker.PROCESSOR_ILP) > numOfThreads) {
-    Os.sleep(50);
-}
+long deadline = System.currentTimeMillis() + 10_000; // 10s
+while (serverMain.getActiveConnectionCount(ActiveConnectionTracker.PROCESSOR_ILP) > numOfThreads) {
+    if (System.currentTimeMillis() > deadline) {
+        Assert.fail("Timed out waiting for ILP active connections to drop");
+    }
+    Os.sleep(50);
+}

490-513: Helper duplication: consider consolidating contains vs equals asserts

assertResponseContains largely duplicates assertResponse. Consider a single helper with a flag or predicate to reduce duplication.

-private void assertResponseContains(HttpClient.ResponseHeaders responseHeaders, int expectedHttpStatusCode, String expectedHttpResponse) {
+private void assertResponseMaybeContains(HttpClient.ResponseHeaders responseHeaders, int expectedHttpStatusCode, String expectedHttpResponse, boolean contains) {
     responseHeaders.clear();
     responseHeaders.await();
     final Utf8StringSink sink = new Utf8StringSink();
     Fragment fragment;
     final Response response = responseHeaders.getResponse();
     while ((fragment = response.recv()) != null) {
         Utf8s.strCpy(fragment.lo(), fragment.hi(), sink);
     }
-    if (!Utf8s.containsAscii(sink, expectedHttpResponse)) {
-        Assert.fail("Expected response to contain: " + expectedHttpResponse + ", actual: " + sink);
-    }
+    if (contains ? !Utf8s.containsAscii(sink, expectedHttpResponse) : !Chars.equalsNc(expectedHttpResponse, sink.toString())) {
+        Assert.fail("Unexpected response body: " + sink);
+    }
     sink.clear();
     TestUtils.assertEquals(String.valueOf(expectedHttpStatusCode), responseHeaders.getStatusCode());
 }

Usage: pass contains=true where partial match is needed.

core/src/main/java/io/questdb/griffin/engine/functions/catalogue/FilesRecordCursor.java (1)

90-100: Prefer FilesFacade for find operations to improve testability.*

You inject FilesFacade but call static Files.findFirst/findNext/findClose/findType/findName. Prefer ff.* where available to ease mocking and keep a consistent abstraction.

Also applies to: 131-141, 153-165

core/src/test/java/io/questdb/test/cutlass/http/TestHttpClient.java (2)

387-394: Mark queryParams as nullable to match call sites.

This parameter is often passed as null; add nullability to aid static analysis.

-    protected String reqToSink(
+    protected String reqToSink(
             HttpClient.Request req,
             Utf8StringSink sink,
             @Nullable CharSequence username,
             @Nullable CharSequence password,
             @Nullable CharSequence token,
-            CharSequenceObjHashMap<String> queryParams
+            @Nullable CharSequenceObjHashMap<String> queryParams
     ) {

452-474: Make status check tolerant of reason-phrase differences.

Responses may be "200 OK" while callers pass "200". Compare by exact match or prefix with space.

-        if (expectedStatus != null) {
-            if (!expectedStatus.equals(respCode)) {
+        if (expectedStatus != null) {
+            final String exp = expectedStatus.toString();
+            if (!(respCode.equals(exp) || respCode.startsWith(exp + " "))) {
                 LOG.error().$("unexpected status code received, expected ").$(expectedStatus)
                         .$(", but was ").$(respCode).$(". Response: ").$safe(sink).$();
                 Assert.fail("expected response code " + expectedStatus + " but got " + respCode);
             }
         }
core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpsSenderTest.java (1)

90-92: Align expected text with rounded SQL result.

Query now uses round(avg(value), 3) but the expected string concatenates the unrounded expectedAvg. Depending on formatting (e.g., trailing zeros), this can flake.

Option A (robust): format the expected to 3dp using locale‑neutral formatting and keep or strip trailing zeros to match engine output.

+import java.math.BigDecimal;
+import java.math.RoundingMode;
@@
-                TestUtils.assertEventually(() -> serverMain.assertSql(
-                        "select sum(value), max(value), min(value), round(avg(value), 3) from " + tableName,
-                        "sum\tmax\tmin\tround\n"
-                                + expectedSum + "\t" + count + "\t1\t" + expectedAvg + "\n"
-                ));
+                TestUtils.assertEventually(() -> {
+                    String expectedAvgStr = new BigDecimal(expectedAvg).setScale(3, RoundingMode.HALF_UP).stripTrailingZeros().toPlainString();
+                    return serverMain.assertSql(
+                            "select sum(value), max(value), min(value), round(avg(value), 3) from " + tableName,
+                            "sum\tmax\tmin\tround\n" + expectedSum + "\t" + count + "\t1\t" + expectedAvgStr + "\n"
+                    );
+                });

If the engine always omits trailing zeros for round, only .stripTrailingZeros() is needed. Please confirm which representation your engine returns.

core/src/main/java/io/questdb/cairo/O3PartitionJob.java (1)

409-414: Path handling cleanup looks good.

Using createDirsOrFail(ff, path, ...) (no trailing slash) is consistent and avoids double‑slash mutations.

Minor nit: the nearby debug log still calls path.slash$() which mutates the Path temporarily; harmless, but consider logging without mutating for clarity.

Also applies to: 1096-1101

core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessor.java (1)

257-263: Propagate proper HTTP status from HttpException (enables 429 for connection limit).

Hardcoding HTTP_BAD_REQUEST here blocks returning 429 Too Many Requests for connection‑limit errors (issue #6278). Prefer deriving the status from the exception (or mapping a typed connection‑limit exception to 429) and falling back to 400 only when unknown.

Would you confirm whether HttpException exposes a status (e.g., getStatus()/code)? If yes, I can provide a precise diff; otherwise we should introduce a typed exception or a mapper used here and in other HTTP processors.

core/src/main/java/io/questdb/mp/WorkerPoolUtils.java (1)

68-71: Make Rnd seeds per‑worker deterministic and unique.

Mix the worker index into seeds to avoid potential collisions when workers start on the same tick.

-                final PageFrameReduceJob pageFrameReduceJob = new PageFrameReduceJob(
-                        cairoEngine,
-                        messageBus,
-                        new Rnd(microsecondClock.getTicks(), nanosecondClock.getTicks())
-                );
+                final long seedLo = microsecondClock.getTicks() ^ (i * 0x9E3779B97F4A7C15L);
+                final long seedHi = nanosecondClock.getTicks() ^ ((long) i << 32);
+                final PageFrameReduceJob pageFrameReduceJob = new PageFrameReduceJob(
+                        cairoEngine,
+                        messageBus,
+                        new Rnd(seedLo, seedHi)
+                );
core/src/test/java/io/questdb/test/cairo/SecurityContextTest.java (1)

68-73: Updated single‑arg dispatch matches new API.

Passing sc, new ObjHashSet<>(), and TABLE_KIND_REGULAR_TABLE is appropriate for the respective methods.

Optionally, reuse a shared static ObjHashSet test instance to avoid per‑call allocations; not critical for tests.

Also applies to: 105-110, 153-158

core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java (1)

100-125: Consider adding @NotNull annotation for consistency.

The engine parameter is used immediately to construct the circuit breaker (line 125), so null would cause an NPE. For consistency with other factories in this PR (e.g., AsyncJitFilteredRecordCursorFactory, AsyncGroupByRecordCursorFactory), consider annotating the parameter as @NotNull.

 public GroupByRecordCursorFactory(
+        @NotNull CairoEngine engine,
-        CairoEngine engine,
         CairoConfiguration configuration,
core/src/main/java/io/questdb/cairo/SecurityContext.java (1)

103-114: LGTM! Consider adding an inline comment for clarity.

The table-kind-based authorization correctly delegates regular table creation to the existing method and allows parquet export in read-only mode. The comment at lines 109-110 explains the intent, but an inline comment explaining why parquet export is allowed in read-only mode would improve maintainability.

         case TableUtils.TABLE_KIND_TEMP_PARQUET_EXPORT:
-            // Allowed even in read-only mode
+            // Allowed in read-only mode because parquet export uses temporary tables
+            // that don't persist data modifications
             return;
core/src/main/java/io/questdb/cutlass/line/udp/LineUdpParserImpl.java (1)

405-431: Switch expression refactor preserves semantics.

Logic matches prior validation; side-effect assignment for geoHash bits is intact. Optional: add parentheses for clarity around the final OR clause.

-                    case ColumnType.STRING, ColumnType.VARCHAR -> columnTypeTag == ColumnType.STRING ||
-                            columnTypeTag == ColumnType.VARCHAR ||
-                            columnTypeTag == ColumnType.CHAR ||
-                            columnTypeTag == ColumnType.IPv4 ||
-                            isForField &&
-                                    (geoHashBits = ColumnType.getGeoHashBits(columnType)) != 0;
+                    case ColumnType.STRING, ColumnType.VARCHAR -> 
+                            (columnTypeTag == ColumnType.STRING
+                             || columnTypeTag == ColumnType.VARCHAR
+                             || columnTypeTag == ColumnType.CHAR
+                             || columnTypeTag == ColumnType.IPv4)
+                            || (isForField && (geoHashBits = ColumnType.getGeoHashBits(columnType)) != 0);
core/src/main/java/io/questdb/cutlass/http/processors/LineHttpPingProcessor.java (1)

44-46: Add @OverRide for getName().

Minor polish: annotate to ensure interface contract is enforced by the compiler.

-    public String getName() {
+    @Override
+    public String getName() {
         return ActiveConnectionTracker.PROCESSOR_ILP;
     }
core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperation.java (1)

42-43: Nullability for CopyDataProgressReporter

If reporter is optional, mark accessor and setter param as @nullable to aid static analysis.

Apply:

@@
-    CopyDataProgressReporter getCopyDataProgressReporter();
+    @org.jetbrains.annotations.Nullable
+    CopyDataProgressReporter getCopyDataProgressReporter();
@@
-    void setCopyDataProgressReporter(CopyDataProgressReporter reporter);
+    void setCopyDataProgressReporter(@org.jetbrains.annotations.Nullable CopyDataProgressReporter reporter);

Also applies to: 67-70

core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestJob.java (2)

102-112: Clarify status transition: WAITING + FINISHED

Setting Phase.WAITING with Status.FINISHED is counterintuitive. If this is a “queued → started” marker, consider a STARTED/IN_PROGRESS status, or document the convention.


95-101: Cancellation check upfront is good; consider also checking after of()

A quick second check after serialExporter.of(localTaskCopy) avoids doing work if cancellation happens between the two calls.

@@
-                serialExporter.of(localTaskCopy);
-                phase = serialExporter.process(); // throws CopyExportException
+                serialExporter.of(localTaskCopy);
+                if (circuitBreaker.checkIfTripped()) {
+                    throw CopyExportException.instance(phase, -1).put("cancelled by user").setInterruption(true).setCancellation(true);
+                }
+                phase = serialExporter.process(); // throws CopyExportException
core/src/main/java/io/questdb/ServerMain.java (1)

392-415: Export worker pool wiring: OK; add tests for “export disabled” advisory path

When workerCount == 0, log/advisory path is exercised; add a small test asserting the advisory log to prevent regressions.

core/src/test/java/io/questdb/test/griffin/CopyExportTest.java (1)

325-328: Avoid asserting exact diskSizeHuman values.

Exact sizes are brittle across platforms/encodings. Prefer checking presence of expected paths and positive sizes, or assert num_exported_files.

Also applies to: 576-580, 614-616, 992-995, 1409-1413, 1440-1443, 1482-1488

core/src/main/java/io/questdb/griffin/engine/ops/CopyExportFactory.java (3)

158-165: Add tiny backoff in MPSequence spin to reduce CPU burn under contention.

-            do {
-                processingCursor = copyRequestPubSeq.next();
-            } while (processingCursor == -2);
+            do {
+                processingCursor = copyRequestPubSeq.next();
+                if (processingCursor == -2) {
+                    io.questdb.std.Os.pause();
+                }
+            } while (processingCursor == -2);

67-69: Avoid coupling to CopyImportFactory.CopyRecord; define a local record.

-    private final CopyImportFactory.CopyRecord record = new CopyImportFactory.CopyRecord();
+    private final CopyRecord record = new CopyRecord();
@@
-    static {
+    public static final class CopyRecord implements io.questdb.cairo.sql.Record {
+        private CharSequence value;
+        @Override public CharSequence getStrA(int col) { return value; }
+        @Override public CharSequence getStrB(int col) { return value; }
+        @Override public int getStrLen(int col) { return value.length(); }
+        public void setValue(CharSequence value) { this.value = value; }
+    }
+    static {
         METADATA.add(new TableColumnMetadata("id", ColumnType.STRING));
     }

Also applies to: 249-252


86-95: General: ensure Parquet libs are on a patched 1.15.1+ (avoid 1.15.0).

Parquet 1.15.0 had deserialization issues; prefer 1.15.1+ in the export path dependencies. Based on learnings.

core/src/main/java/io/questdb/cutlass/text/CopyExportResult.java (1)

51-56: Make cleanup idempotent by resetting the flag after deletion.

     public void cleanUpTempPath(FilesFacade ff) {
         if (needCleanUp) {
             path.clear(cleanUpFileLength, path.isAscii());
             TableUtils.cleanupDirQuiet(ff, path);
+            needCleanUp = false;
         }
     }
core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ImportFilesFunctionFactory.java (2)

70-79: Gate listing behind an auth check (avoid leaking filesystem info).

Consider restricting to admins (or a dedicated permission) before exposing import_files():

     ) throws SqlException {
         if (Chars.isBlank(configuration.getSqlCopyInputRoot())) {
             throw SqlException.$(position, "import_files() is disabled ['cairo.sql.copy.root' is not set?]");
         }
+        final io.questdb.cairo.SecurityContext sc = sqlExecutionContext.getSecurityContext();
+        if (sc == null || !sc.isSystemAdmin()) {
+            throw SqlException.$(position, "import_files() requires admin privileges");
+        }
         return new CursorFunction(new ImportFilesCursorFactory(configuration)) {

109-114: Prefer Misc.free(cursor) to ensure native cleanup via our utility.

     @Override
     protected void _close() {
-        Misc.free(importPath);
-        cursor.close();
+        Misc.free(importPath);
+        Misc.free(cursor);
     }
core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestTask.java (1)

50-63: Defensive null-safety and reset missing in clear().

  • getCircuitBreaker()/getSecurityContext() dereference entry; guard against null or document precondition.
  • clear() doesn’t reset rawArrayEncoding; reset to a safe default.

Apply:

@@ public void clear() {
-        this.statisticsEnabled = true;
+        this.statisticsEnabled = true;
+        this.rawArrayEncoding = false;
@@
-    public SqlExecutionCircuitBreaker getCircuitBreaker() {
-        return entry.getCircuitBreaker();
+    public SqlExecutionCircuitBreaker getCircuitBreaker() {
+        assert entry != null : "entry must be set via of() before use";
+        return entry.getCircuitBreaker();
@@
-    public SecurityContext getSecurityContext() {
-        return entry.getSecurityContext();
+    public SecurityContext getSecurityContext() {
+        assert entry != null : "entry must be set via of() before use";
+        return entry.getSecurityContext();

Also applies to: 65-67, 109-111

core/src/main/java/io/questdb/cutlass/text/CopyImportRequestJob.java (1)

238-253: Redundant setStatusReporter() call (nit).

Setting the reporter twice around of()/process() is unnecessary. Keep one.

-                    parallelImporter.setStatusReporter(updateStatusRef);
                     parallelImporter.of(
@@
-                    parallelImporter.setStatusReporter(updateStatusRef);
+                    parallelImporter.setStatusReporter(updateStatusRef);

Also applies to: 251-253

core/src/main/java/io/questdb/cutlass/parquet/SerialParquetExporter.java (1)

305-311: Avoid class-wide synchronization on mkdirs (throughput).

createDirsOrFail synchronizes on SerialParquetExporter.class, throttling concurrent exports.

  • Remove synchronized block; rely on mkdirs idempotency and handle ENOENT/EEXIST gracefully.
  • If needed, synchronize per-target directory or use ff.mkdirs return code only.
core/src/main/java/io/questdb/cutlass/text/CopyExportContext.java (1)

477-492: Ensure visibility of progress fields across threads.

Fields written by worker threads and read under locks elsewhere may still be stale without a proper happens‑before; mark them volatile or write under the same lock.

-        int finishedPartitionCount = 0;
-        long id = INACTIVE_COPY_ID;
-        CopyExportRequestTask.Phase phase;
-        long populatedRowCount = 0;
+        volatile int finishedPartitionCount = 0;
+        volatile long id = INACTIVE_COPY_ID;
+        volatile CopyExportRequestTask.Phase phase;
+        volatile long populatedRowCount = 0;
@@
-        long startTime = Numbers.LONG_NULL;
-        int totalPartitionCount = 0;
-        long totalRowCount = 0;
+        volatile long startTime = Numbers.LONG_NULL;
+        volatile int totalPartitionCount = 0;
+        volatile long totalRowCount = 0;
@@
-        CopyTrigger trigger = CopyTrigger.NONE;
-        int workerId = -1;
+        volatile CopyTrigger trigger = CopyTrigger.NONE;
+        volatile int workerId = -1;
core/src/test/java/io/questdb/test/cutlass/http/ExpParquetExportTest.java (2)

873-1040: Add an assertion that over‑limit requests get 429 (issue #6278).

The limit test exercises concurrency but doesn’t verify 429. Add a small probe that opens (limit+1) connections and asserts one returns 429.

@@
-                try {
+                try {
                     params.clear();
@@
-                    try {
-                        for (int i = 0; i < requestLimit; i++) {
+                    try {
+                        for (int i = 0; i < requestLimit; i++) {
                             HttpClient client = HttpClientFactory.newPlainTextInstance();
                             clients.add(client);
                             HttpClient.ResponseHeaders resp = startExport(client, serverMain, params, "multiple_options_test");
                             respHeaders.add(resp);
                         }
                     } finally {
-                        for (int i = 0; i < respHeaders.size(); i++) {
-                            respHeaders.get(i).await();
-                        }
+                        boolean saw429 = false;
+                        for (int i = 0; i < respHeaders.size(); i++) {
+                            respHeaders.get(i).await();
+                            if ("429".contentEquals(respHeaders.get(i).getStatusCode())) {
+                                saw429 = true;
+                            }
+                        }
+                        Assert.assertTrue("Expected at least one 429 Too Many Requests when exceeding export limit", saw429);
                         for (int i = 0; i < clients.size(); i++) {
                             clients.get(i).close();
                         }

372-387: Param name consistency.

Tests sometimes use “format=csv” while the endpoint reads “fmt”; relying on default CSV works, but use “fmt” for clarity.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 51a6039 and 3f6022f.

⛔ Files ignored due to path filters (5)
  • core/src/main/resources/io/questdb/bin/darwin-aarch64/libquestdbr.dylib is excluded by !**/*.dylib
  • core/src/main/resources/io/questdb/bin/darwin-x86-64/libquestdbr.dylib is excluded by !**/*.dylib
  • core/src/main/resources/io/questdb/bin/linux-aarch64/libquestdbr.so is excluded by !**/*.so
  • core/src/main/resources/io/questdb/bin/linux-x86-64/libquestdbr.so is excluded by !**/*.so
  • core/src/main/resources/io/questdb/bin/windows-x86-64/questdbr.dll is excluded by !**/*.dll
📒 Files selected for processing (107)
  • compat/src/test/java/io/questdb/compat/ParquetTest.java (4 hunks)
  • core/rust/qdb-core/src/col_type.rs (1 hunks)
  • core/rust/qdbr/Cargo.toml (1 hunks)
  • core/rust/qdbr/parquet2/Cargo.toml (1 hunks)
  • core/rust/qdbr/src/parquet_read/decode.rs (1 hunks)
  • core/rust/qdbr/src/parquet_read/meta.rs (1 hunks)
  • core/rust/qdbr/src/parquet_write/schema.rs (1 hunks)
  • core/src/main/java/io/questdb/DynamicPropServerConfiguration.java (3 hunks)
  • core/src/main/java/io/questdb/PropHttpServerConfiguration.java (5 hunks)
  • core/src/main/java/io/questdb/PropServerConfiguration.java (22 hunks)
  • core/src/main/java/io/questdb/PropertyKey.java (5 hunks)
  • core/src/main/java/io/questdb/ServerMain.java (6 hunks)
  • core/src/main/java/io/questdb/cairo/CairoConfiguration.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/CairoEngine.java (11 hunks)
  • core/src/main/java/io/questdb/cairo/DdlListener.java (1 hunks)
  • core/src/main/java/io/questdb/cairo/DefaultCairoConfiguration.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/DefaultDdlListener.java (1 hunks)
  • core/src/main/java/io/questdb/cairo/O3PartitionJob.java (2 hunks)
  • core/src/main/java/io/questdb/cairo/SecurityContext.java (1 hunks)
  • core/src/main/java/io/questdb/cairo/TableUtils.java (4 hunks)
  • core/src/main/java/io/questdb/cairo/TableWriter.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/mv/MatViewRefreshJob.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/sql/AtomicBooleanCircuitBreaker.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/sql/NetworkSqlExecutionCircuitBreaker.java (8 hunks)
  • core/src/main/java/io/questdb/cairo/sql/SqlExecutionCircuitBreaker.java (1 hunks)
  • core/src/main/java/io/questdb/cairo/sql/SqlExecutionCircuitBreakerWrapper.java (2 hunks)
  • core/src/main/java/io/questdb/cairo/sql/async/PageFrameReduceJob.java (3 hunks)
  • core/src/main/java/io/questdb/cairo/sql/async/PageFrameSequence.java (3 hunks)
  • core/src/main/java/io/questdb/cutlass/http/HttpConnectionContext.java (16 hunks)
  • core/src/main/java/io/questdb/cutlass/http/HttpConstants.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/http/HttpRequestProcessor.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/http/HttpResponseSink.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/http/HttpServer.java (6 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/ActiveConnectionTracker.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessor.java (22 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessorState.java (3 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessor.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessorState.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/LineHttpPingProcessor.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/LineHttpProcessorImpl.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/LineHttpTudCache.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/http/processors/StaticContentProcessor.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/line/tcp/LineTcpMeasurementScheduler.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/line/udp/LineUdpParserImpl.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestJob.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestTask.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/parquet/SerialParquetExporter.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (3 hunks)
  • core/src/main/java/io/questdb/cutlass/pgwire/PGServer.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/text/CairoTextWriter.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/text/CopyExportContext.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/text/CopyExportResult.java (1 hunks)
  • core/src/main/java/io/questdb/cutlass/text/CopyImportContext.java (2 hunks)
  • core/src/main/java/io/questdb/cutlass/text/CopyImportRequestJob.java (7 hunks)
  • core/src/main/java/io/questdb/griffin/CompiledQuery.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/CompiledQueryImpl.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/CopyDataProgressReporter.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/SqlCodeGenerator.java (11 hunks)
  • core/src/main/java/io/questdb/griffin/SqlCompilerImpl.java (28 hunks)
  • core/src/main/java/io/questdb/griffin/SqlExecutionContextImpl.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/SqlKeywords.java (4 hunks)
  • core/src/main/java/io/questdb/griffin/SqlParser.java (16 hunks)
  • core/src/main/java/io/questdb/griffin/engine/functions/activity/ExportActivityFunctionFactory.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ExportFilesFunctionFactory.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/engine/functions/catalogue/FilesRecordCursor.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ImportFilesFunctionFactory.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByNotKeyedVectorRecordCursorFactory.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/ops/CopyExportFactory.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/engine/ops/CopyImportFactory.java (6 hunks)
  • core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperation.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperationBuilderImpl.java (7 hunks)
  • core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperationImpl.java (17 hunks)
  • core/src/main/java/io/questdb/griffin/engine/ops/InsertAsSelectOperationImpl.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncFilteredRecordCursorFactory.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByNotKeyedRecordCursorFactory.java (4 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByRecordCursor.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByRecordCursorFactory.java (4 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncJitFilteredRecordCursorFactory.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncTopKRecordCursorFactory.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursor.java (3 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursorFactory.java (2 hunks)
  • core/src/main/java/io/questdb/griffin/engine/table/parquet/ParquetCompression.java (1 hunks)
  • core/src/main/java/io/questdb/griffin/model/ExportModel.java (1 hunks)
  • core/src/main/java/io/questdb/log/AbstractLogRecord.java (1 hunks)
  • core/src/main/java/io/questdb/mp/WorkerPoolUtils.java (1 hunks)
  • core/src/main/resources/io/questdb/site/conf/server.conf (4 hunks)
  • core/src/test/java/io/questdb/test/AbstractCairoTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/PropServerConfigurationTest.java (7 hunks)
  • core/src/test/java/io/questdb/test/ServerMainTest.java (7 hunks)
  • core/src/test/java/io/questdb/test/cairo/RssMemoryLimitTest.java (2 hunks)
  • core/src/test/java/io/questdb/test/cairo/SecurityContextTest.java (4 hunks)
  • core/src/test/java/io/questdb/test/cairo/fuzz/ParallelGroupByFuzzTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/cairo/mv/MatViewTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/cutlass/http/ExpParquetExportTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/cutlass/http/HttpConnectionCountTest.java (9 hunks)
  • core/src/test/java/io/questdb/test/cutlass/http/HttpErrorHandlingTest.java (2 hunks)
  • core/src/test/java/io/questdb/test/cutlass/http/HttpQueryTestBuilder.java (5 hunks)
  • core/src/test/java/io/questdb/test/cutlass/http/TestHttpClient.java (11 hunks)
  • core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpsSenderTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/cutlass/pgwire/PGErrorHandlingTest.java (2 hunks)
  • core/src/test/java/io/questdb/test/cutlass/pgwire/PGJobContextTest.java (3 hunks)
  • core/src/test/java/io/questdb/test/griffin/AlterTableConvertPartitionTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/griffin/CheckpointTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/griffin/CopyExportTest.java (1 hunks)
  • core/src/test/java/io/questdb/test/griffin/CopyImportTest.java (10 hunks)
⛔ Files not processed due to max files limit (2)
  • core/src/test/java/io/questdb/test/griffin/OrderByTimeoutTest.java
  • core/src/test/java/io/questdb/test/griffin/ParallelFilterTest.java
✅ Files skipped from review due to trivial changes (5)
  • core/rust/qdbr/parquet2/Cargo.toml
  • core/src/main/java/io/questdb/log/AbstractLogRecord.java
  • core/src/test/java/io/questdb/test/cairo/mv/MatViewTest.java
  • core/rust/qdbr/src/parquet_read/decode.rs
  • core/rust/qdbr/src/parquet_read/meta.rs
🚧 Files skipped from review as they are similar to previous changes (11)
  • core/src/main/java/io/questdb/DynamicPropServerConfiguration.java
  • core/src/test/java/io/questdb/test/griffin/CopyImportTest.java
  • core/src/test/java/io/questdb/test/PropServerConfigurationTest.java
  • core/src/main/java/io/questdb/griffin/CompiledQueryImpl.java
  • core/src/main/java/io/questdb/griffin/SqlExecutionContextImpl.java
  • core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ExportFilesFunctionFactory.java
  • core/src/test/java/io/questdb/test/griffin/AlterTableConvertPartitionTest.java
  • core/src/test/java/io/questdb/test/cutlass/pgwire/PGJobContextTest.java
  • core/src/test/java/io/questdb/test/AbstractCairoTest.java
  • core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java
  • core/src/main/java/io/questdb/PropertyKey.java
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (27)
  • GitHub Check: New pull request (Coverage Report Coverage Report)
  • GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
  • GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-x64-zfs)
  • GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
  • GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-arm64)
  • GitHub Check: New pull request (Hosted Running tests on windows-other)
  • GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
  • GitHub Check: New pull request (Hosted Running tests on windows-cairo)
  • GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
  • GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
  • GitHub Check: New pull request (Hosted Running tests on windows-griffin)
  • GitHub Check: New pull request (Hosted Running tests on mac-other)
  • GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
  • GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
  • GitHub Check: New pull request (Hosted Running tests on mac-cairo)
  • GitHub Check: New pull request (Hosted Running tests on mac-griffin)
  • GitHub Check: New pull request (SelfHosted Other tests Start X64Zfs Agent)
  • GitHub Check: New pull request (SelfHosted Other tests Start ARM Agent)
  • GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start X64Zfs Agent)
  • GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
  • GitHub Check: New pull request (Hosted Running tests with cover on linux-other)
  • GitHub Check: New pull request (Hosted Running tests with cover on linux-pgwire)
  • GitHub Check: New pull request (Hosted Running tests with cover on linux-cairo)
  • GitHub Check: New pull request (Hosted Running tests with cover on linux-fuzz)
  • GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start ARM Agent)
  • GitHub Check: New pull request (Hosted Running tests with cover on linux-griffin)
  • GitHub Check: New pull request (Check Changes Check changes)

ideoma
ideoma previously approved these changes Oct 20, 2025
@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 2282 / 2585 (88.28%)

file detail

path covered line new line coverage
🔵 io/questdb/network/NetworkError.java 0 1 00.00%
🔵 io/questdb/griffin/model/ExecutionModel.java 0 1 00.00%
🔵 io/questdb/cutlass/http/DefaultHttpServerConfiguration.java 0 1 00.00%
🔵 io/questdb/cairo/CairoException.java 0 1 00.00%
🔵 io/questdb/griffin/engine/table/parquet/PartitionEncoder.java 1 5 20.00%
🔵 io/questdb/griffin/CompiledQueryImpl.java 4 8 50.00%
🔵 io/questdb/DefaultServerConfiguration.java 1 2 50.00%
🔵 io/questdb/cutlass/pgwire/PGConnectionContext.java 2 3 66.67%
🔵 io/questdb/cutlass/text/SerialCsvFileImporter.java 5 7 71.43%
🔵 io/questdb/cairo/mv/MatViewRefreshJob.java 10 14 71.43%
🔵 io/questdb/cutlass/parquet/CopyExportRequestJob.java 58 76 76.32%
🔵 io/questdb/cutlass/parquet/SerialParquetExporter.java 209 272 76.84%
🔵 io/questdb/griffin/engine/functions/catalogue/FilesRecordCursor.java 108 135 80.00%
🔵 io/questdb/griffin/SqlExecutionContextImpl.java 4 5 80.00%
🔵 io/questdb/ServerMain.java 22 27 81.48%
🔵 io/questdb/griffin/SqlCodeGenerator.java 9 11 81.82%
🔵 io/questdb/cairo/SecurityContext.java 5 6 83.33%
🔵 io/questdb/griffin/engine/ops/CopyCancelFactory.java 20 24 83.33%
🔵 io/questdb/cutlass/text/CopyExportResult.java 28 33 84.85%
🔵 io/questdb/cairo/TableUtils.java 21 25 84.00%
🔵 io/questdb/cutlass/http/processors/ExportQueryProcessor.java 242 285 84.91%
🔵 io/questdb/griffin/engine/ops/CopyImportFactory.java 21 25 84.00%
🔵 io/questdb/cutlass/http/HttpConnectionContext.java 35 41 85.37%
🔵 io/questdb/griffin/engine/functions/activity/ExportActivityFunctionFactory.java 78 89 87.64%
🔵 io/questdb/cutlass/text/CopyExportContext.java 240 271 88.56%
🔵 io/questdb/griffin/engine/ops/CreateTableOperationImpl.java 47 53 88.68%
🔵 io/questdb/griffin/engine/ops/CopyExportFactory.java 77 87 88.51%
🔵 io/questdb/griffin/SqlParser.java 97 107 90.65%
🔵 io/questdb/cutlass/http/ActiveConnectionTracker.java 32 35 91.43%
🔵 io/questdb/griffin/SqlCompilerImpl.java 106 115 92.17%
🔵 io/questdb/cutlass/line/udp/LineUdpParserImpl.java 13 14 92.86%
🔵 io/questdb/std/LongStack.java 68 73 93.15%
🔵 io/questdb/cairo/CairoEngine.java 17 18 94.44%
🔵 io/questdb/PropServerConfiguration.java 71 74 95.95%
🔵 io/questdb/griffin/model/ExportModel.java 134 140 95.71%
🔵 io/questdb/griffin/engine/functions/catalogue/ExportFilesFunctionFactory.java 21 22 95.45%
🔵 io/questdb/griffin/engine/table/parquet/ParquetCompression.java 26 27 96.30%
🔵 io/questdb/griffin/engine/functions/catalogue/ImportFilesFunctionFactory.java 28 29 96.55%
🔵 io/questdb/cutlass/text/ParallelCsvFileImporter.java 67 68 98.53%
🔵 io/questdb/cutlass/http/DefaultHttpContextConfiguration.java 1 1 100.00%
🔵 io/questdb/cairo/MetadataCache.java 2 2 100.00%
🔵 io/questdb/cutlass/http/HttpConstants.java 10 10 100.00%
🔵 io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursor.java 1 1 100.00%
🔵 io/questdb/MessageBusImpl.java 27 27 100.00%
🔵 io/questdb/cairo/sql/AtomicBooleanCircuitBreaker.java 5 5 100.00%
🔵 io/questdb/cutlass/line/tcp/LineTcpMeasurementScheduler.java 1 1 100.00%
🔵 io/questdb/cutlass/text/CopyImportTask.java 2 2 100.00%
🔵 io/questdb/cairo/sql/async/PageFrameSequence.java 1 1 100.00%
🔵 io/questdb/cutlass/text/CopyImportJob.java 3 3 100.00%
🔵 io/questdb/cutlass/http/HttpRequestProcessor.java 3 3 100.00%
🔵 io/questdb/WorkerPoolManager.java 21 21 100.00%
🔵 io/questdb/log/AbstractLogRecord.java 1 1 100.00%
🔵 io/questdb/griffin/engine/groupby/vect/GroupByNotKeyedVectorRecordCursorFactory.java 1 1 100.00%
🔵 parquet_read/decode.rs 1 1 100.00%
🔵 io/questdb/cutlass/http/processors/LineHttpPingProcessor.java 1 1 100.00%
🔵 io/questdb/griffin/engine/ops/CreateTableOperationBuilderImpl.java 5 5 100.00%
🔵 io/questdb/cutlass/http/processors/LineHttpTudCache.java 1 1 100.00%
🔵 io/questdb/cutlass/parquet/CopyExportRequestTask.java 64 64 100.00%
🔵 io/questdb/cutlass/http/processors/JsonQueryProcessor.java 3 3 100.00%
🔵 io/questdb/cutlass/http/HttpResponseSink.java 1 1 100.00%
🔵 io/questdb/cutlass/http/HttpServer.java 5 5 100.00%
🔵 io/questdb/cairo/sql/async/PageFrameReduceJob.java 1 1 100.00%
🔵 io/questdb/cutlass/text/CopyImportRequestTask.java 1 1 100.00%
🔵 io/questdb/cairo/DefaultCairoConfiguration.java 13 13 100.00%
🔵 io/questdb/DynamicPropServerConfiguration.java 1 1 100.00%
🔵 io/questdb/mp/WorkerPoolUtils.java 1 1 100.00%
🔵 io/questdb/cutlass/text/CopyImportRequestJob.java 15 15 100.00%
🔵 io/questdb/cairo/O3PartitionJob.java 2 2 100.00%
🔵 io/questdb/cutlass/http/processors/StaticContentProcessor.java 1 1 100.00%
🔵 io/questdb/cairo/sql/SqlExecutionCircuitBreakerWrapper.java 2 2 100.00%
🔵 io/questdb/cutlass/http/processors/ExportQueryProcessorState.java 39 39 100.00%
🔵 io/questdb/PropertyKey.java 20 20 100.00%
🔵 io/questdb/griffin/engine/table/AsyncGroupByRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/cutlass/http/processors/LineHttpProcessorImpl.java 1 1 100.00%
🔵 io/questdb/cutlass/parquet/CopyExportException.java 15 15 100.00%
🔵 io/questdb/std/MemoryTag.java 1 1 100.00%
🔵 io/questdb/std/LongObjHashMap.java 1 1 100.00%
🔵 io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncGroupByRecordCursor.java 1 1 100.00%
🔵 io/questdb/griffin/engine/ops/CreateMatViewOperationImpl.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/SqlException.java 1 1 100.00%
🔵 io/questdb/cairo/sql/NetworkSqlExecutionCircuitBreaker.java 6 6 100.00%
🔵 io/questdb/griffin/CopyDataProgressReporter.java 6 6 100.00%
🔵 io/questdb/cairo/TableWriter.java 3 3 100.00%
🔵 io/questdb/cairo/CairoConfigurationWrapper.java 11 11 100.00%
🔵 io/questdb/std/str/Utf8StringSink.java 3 3 100.00%
🔵 io/questdb/cutlass/http/processors/JsonQueryProcessorState.java 1 1 100.00%
🔵 io/questdb/griffin/SqlKeywords.java 30 30 100.00%
🔵 parquet_write/schema.rs 1 1 100.00%
🔵 io/questdb/cutlass/text/CopyImportContext.java 12 12 100.00%
🔵 io/questdb/PropHttpServerConfiguration.java 2 2 100.00%

@ideoma ideoma merged commit b56f96f into master Oct 20, 2025
37 checks passed
@ideoma ideoma deleted the feat-copy-parquet branch October 20, 2025 18:47
ideoma added a commit that referenced this pull request Oct 21, 2025
Co-authored-by: victor <[email protected]>
Co-authored-by: Vlad Ilyushchenko <[email protected]>
Co-authored-by: ideoma <[email protected]>
@jmthomas
Copy link
Copy Markdown

@nwoolmer excited to see this merge! Are there new docs associated with this? I assume this will be the preferred way to remove data partitions for long term storage and backup.

@nwoolmer
Copy link
Copy Markdown
Contributor Author

Yes indeed, for OSS users. There is a draft docs PR that I will get merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Core Related to storage, data type, etc. SQL Issues or changes relating to SQL execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

change the response code for a connection limit error to 429 instead of 400

9 participants