feat(sql): add exporting PARQUET functionality to COPY#6008
Conversation
This comment was marked as spam.
This comment was marked as spam.
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 26
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (9)
core/src/test/java/io/questdb/test/cutlass/http/HttpConnectionCountTest.java (1)
1-1: Add test coverage for /exp (export) endpoint returning 429 on connection limit breachThe export processor has connection limit infrastructure (HTTP_EXPORT_CONNECTION_LIMIT config and ActiveConnectionTracker enforcement), but lacks test coverage. Add a test method similar to testIlpConnectionLimit() and testQueryConnectionLimit() to verify that /exp returns 429 when the export connection limit is exceeded.
core/src/main/java/io/questdb/cairo/sql/NetworkSqlExecutionCircuitBreaker.java (1)
55-73: Add @NotNull annotation and runtime null-check for engine parameter to enforce fail-fast semanticsThe
enginefield is dereferenced incheckIfTripped(),getState(), andtestCancelled()methods without null-safety guards. While all current call sites provide non-null values, the constructor parameter lacks@NotNullannotation and runtime validation. Apply defensive programming:- public NetworkSqlExecutionCircuitBreaker(CairoEngine engine, @NotNull SqlExecutionCircuitBreakerConfiguration configuration, int memoryTag) { + public NetworkSqlExecutionCircuitBreaker(@NotNull CairoEngine engine, @NotNull SqlExecutionCircuitBreakerConfiguration configuration, int memoryTag) { this.configuration = configuration; this.nf = configuration.getNetworkFacade(); this.throttle = configuration.getCircuitBreakerThrottle(); this.bufferSize = configuration.getBufferSize(); this.memoryTag = memoryTag; this.buffer = Unsafe.malloc(this.bufferSize, this.memoryTag); this.clock = configuration.getClock(); long timeout = configuration.getQueryTimeout(); if (timeout > 0) { this.timeout = timeout; } else if (timeout == TIMEOUT_FAIL_ON_FIRST_CHECK) { this.timeout = -100; } else { this.timeout = Long.MAX_VALUE; } this.defaultMaxTime = this.timeout; - this.engine = engine; + this.engine = java.util.Objects.requireNonNull(engine, "engine"); }core/src/main/java/io/questdb/griffin/engine/ops/CopyImportFactory.java (1)
99-121: Always release SPSequence slot; handle queue-full (avoid stuck import queue).If an exception occurs between next() and done(), the slot is leaked. Also, asserting on the cursor (> -1) is not a runtime guard. Wrap publish in try/finally and error on -1.
- try { - circuitBreaker.reset(); - final SPSequence copyRequestPubSeq = messageBus.getCopyImportRequestPubSeq(); - long processingCursor = copyRequestPubSeq.next(); - assert processingCursor > -1; - final CopyImportRequestTask task = copyImportRequestQueue.get(processingCursor); - task.of( + try { + circuitBreaker.reset(); + final SPSequence copyRequestPubSeq = messageBus.getCopyImportRequestPubSeq(); + long processingCursor = -1; + try { + processingCursor = copyRequestPubSeq.next(); + if (processingCursor == -1) { + throw SqlException.$(0, "unable to process the import request - import queue is full"); + } + final CopyImportRequestTask task = copyImportRequestQueue.get(processingCursor); + task.of( executionContext.getSecurityContext(), copyID, tableName, fileName, headerFlag, timestampColumn, delimiter, timestampFormat, partitionBy, atomicity - ); - copyRequestPubSeq.done(processingCursor); + ); + } finally { + if (processingCursor >= 0) { + copyRequestPubSeq.done(processingCursor); + } + } record.setValue(importIdSink); cursor.toTop(); return cursor;core/src/main/java/io/questdb/cutlass/http/HttpConnectionContext.java (1)
906-910: Connection-limit accounting sticks across keep‑alive; switch slot when processor changes.A keep‑alive connection may serve different processors over time. Because connectionCounted stays true and processorName isn’t updated, counts can be held under the wrong processor and limits not enforced for the new one.
- if (!connectionCounted && !processor.ignoreConnectionLimitCheck()) { - processor = checkConnectionLimit(processor); - connectionCounted = true; - } + if (!processor.ignoreConnectionLimitCheck()) { + final String currentName = processor.getName(); + // If we already counted but processor changed, release old slot first. + if (connectionCounted && processorName != null && !java.util.Objects.equals(processorName, currentName)) { + decrementActiveConnections(getFd()); // sets connectionCounted=false and clears processorName + } + if (!connectionCounted) { + processor = checkConnectionLimit(processor); + connectionCounted = true; + } + }core/src/main/java/io/questdb/cutlass/text/CopyImportRequestJob.java (1)
171-179: Fix status logging: wrong variable used for status.Uses getStatusName(phase) instead of status; log shows wrong status.
- .$(", phase=").$(getPhaseName(phase)) - .$(", status=").$(getStatusName(phase)) + .$(", phase=").$(getPhaseName(phase)) + .$(", status=").$(getStatusName(status))core/src/main/java/io/questdb/cairo/CairoEngine.java (2)
471-493: Engine shutdown: signal and free COPY contexts; prevent leaks.close() never signals shutdown nor frees copyExportContext/copyImportContext. Risk of thread/queue/resource leaks.
public void close() { + // Signal shutdown early so dependent services/jobs stop cleanly + signalClose(); Misc.free(sqlCompilerPool); @@ Misc.free(settingsStore); Misc.free(frameFactory); + // Release COPY contexts + Misc.free(copyExportContext); + Misc.free(copyImportContext); }
451-472: clear(): also clear the import context.Only copyExportContext.clear() is called.
- copyExportContext.clear(); + copyExportContext.clear(); + copyImportContext.clear();core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessorState.java (1)
124-129: close() should release parquet resources too.Currently only cursor/factory are freed. Call cleanupParquetState() to avoid fd/mapping/temp path leaks when callers invoke close() without clear().
public void close() { cursor = Misc.free(cursor); recordCursorFactory = Misc.free(recordCursorFactory); + cleanupParquetState(); }core/src/main/java/io/questdb/griffin/SqlCompilerImpl.java (1)
663-729: Inconsistent progress reporting frequency for string timestamp columns.The
copyOrderedBatchedStrTimestampandcopyOrderedBatchedVarcharTimestampmethods hardcode the report frequency to 50000 (lines 688, 723), whilecopyOrderedBatched0uses a configurablereportFrequencyparameter. This creates inconsistent behavior where progress reporting frequency cannot be configured when the timestamp column is a string or varchar type.Consider accepting
reportFrequencyas a parameter in these methods for consistency:private static long copyOrderedBatchedStrTimestamp( TableWriterAPI writer, RecordCursor cursor, RecordToRowCopier copier, int cursorTimestampIndex, long batchSize, long o3MaxLag, SqlExecutionCircuitBreaker circuitBreaker, - CopyDataProgressReporter reporter + CopyDataProgressReporter reporter, + int reportFrequency ) { // ... existing code ... - if (rowCount % 50000 == 0) { + if (reportFrequency > 0 && rowCount % reportFrequency == 0) { reporter.onProgress(CopyDataProgressReporter.Stage.Inserting, rowCount); }Apply the same pattern to
copyOrderedBatchedVarcharTimestamp.
♻️ Duplicate comments (7)
core/src/main/java/io/questdb/griffin/engine/table/parquet/ParquetCompression.java (1)
42-43: GZIP max level off-by-one.GZIP compression supports levels 0–9, not 0–10. This issue was already flagged in a previous review.
core/src/main/java/io/questdb/griffin/CopyDataProgressReporter.java (1)
34-38: Enum constants declared with()will not compile; remove parentheses.This is a hard compile error unless the enum declares a matching constructor. Also, this was flagged earlier and remains unresolved. Fix to plain constants.
Apply:
- enum Stage { - Start(), - Inserting(), - Finish(), - } + enum Stage { + Start, + Inserting, + Finish + }core/src/main/java/io/questdb/ServerMain.java (1)
356-361: Read-only gating for QueryTracingJob acknowledgedThis aligns with prior discussion to allow startup in read-only mode.
core/src/main/java/io/questdb/cutlass/parquet/SerialParquetExporter.java (1)
382-389: mkdirs on file path ('.parquet') creates a directory named like the file.Single-file export calls mkdirs on the file path, which can break rename.
- this.exportPath.put(toParquet.$()); - createDirsOrFail(ff, toParquet, configuration.getMkDirMode()); + this.exportPath.put(toParquet.$()); + // ensure parent directory exists, not the file path + toParquet.slash(); + createDirsOrFail(ff, toParquet, configuration.getMkDirMode()); + toParquet.chopZ();core/src/main/java/io/questdb/cutlass/text/CopyExportContext.java (2)
147-165: Fix authorization target and avoid calling external code under the read lock.Authorize against the task owner’s SecurityContext and perform authorization/cancel outside the lock. Current code self‑authorizes and calls into external code while holding the read lock.
- public boolean cancel(long id, SecurityContext securityContext) { - lock.readLock().lock(); - try { - ExportTaskEntry e = activeExports.get(id); - if (e != null) { - var cb = e.getCircuitBreaker(); - if (cb != null) { - if (securityContext != null) { - securityContext.authorizeCopyCancel(securityContext); - } - cb.cancel(); - } - return true; - } - return false; - } finally { - lock.readLock().unlock(); - } - } + public boolean cancel(long id, SecurityContext securityContext) { + SecurityContext owner = null; + SqlExecutionCircuitBreaker cb = null; + lock.readLock().lock(); + try { + ExportTaskEntry e = activeExports.get(id); + if (e == null) { + return false; + } + owner = e.getSecurityContext(); + cb = e.getCircuitBreaker(); + } finally { + lock.readLock().unlock(); + } + if (securityContext != null && owner != null) { + owner.authorizeCopyCancel(securityContext); + } + if (cb != null) { + cb.cancel(); + } + return true; + }
267-279: Prevent use‑after‑release: release to pool after fields are fully reset, not inside releaseEntry().Move pool release into ExportTaskEntry.clear() after zeroing all fields; releaseEntry() should only unregister from registries under the write lock. This avoids handing out a stale instance to another thread.
@@ public void releaseEntry(ExportTaskEntry entry) { lock.writeLock().lock(); try { activeExports.remove(entry.id); - exportBySqlText.remove(entry.sqlText); + exportBySqlText.remove(entry.sqlText); if (!entry.fileName.isEmpty()) { exportEntriesByFileName.remove(entry.fileName); } - exportTaskEntryPools.release(entry); } finally { lock.writeLock().unlock(); } } @@ public static class ExportTaskEntry implements Mutable { - @Override - public void clear() { + @Override + public void clear() { this.securityContext = null; if (atomicBooleanCircuitBreaker != null) { atomicBooleanCircuitBreaker.clear(); } this.id = INACTIVE_COPY_ID; this.sqlText.clear(); this.fileName.clear(); this.phase = null; this.startTime = Numbers.LONG_NULL; this.workerId = -1; this.populatedRowCount = 0; this.finishedPartitionCount = 0; realCircuitBreaker = null; this.totalPartitionCount = 0; this.totalRowCount = 0; this.trigger = CopyTrigger.NONE; + // Return to pool last to ensure a fully reset instance is reused. + // Requires enclosing context to hold write lock while calling clear(). + // If clear() is called outside context lock, introduce a dedicated release() that acquires it. + // noinspection DataFlowIssue + // (exportTaskEntryPools is an outer class field) + CopyExportContext.this.exportTaskEntryPools.release(this); }Also applies to: 493-511
core/src/main/java/io/questdb/griffin/SqlCompilerImpl.java (1)
1630-1636: Fail early when source table doesn't exist for COPY TO operations.For COPY TO operations, the source table must exist. However, if the table doesn't exist, authorization is silently skipped (line 1633-1635) and the operation will fail later with a potentially confusing error. This was flagged in previous reviews but remains unresolved.
Apply this diff to fail early with a clear error message:
private void authorizeSelectForCopy(SecurityContext securityContext, ExportModel model) { final CharSequence tableName = unquote(model.getTableName()); final TableToken tt = engine.verifyTableName(tableName); if (tt != null) { securityContext.authorizeSelectOnAnyColumn(tt); } else { - // silently skip authorization + throw SqlException.tableDoesNotExist(model.getTableNameExpr().position, tableName); } }
🧹 Nitpick comments (42)
core/src/test/java/io/questdb/test/cutlass/pgwire/PGErrorHandlingTest.java (1)
66-66: Remove unused counter variable.The
countervariable is no longer used after refactoring from a counter-based trigger to a filename-based trigger.Apply this diff to remove the unused variable:
- final AtomicInteger counter = new AtomicInteger(0); final Bootstrap bootstrap = new Bootstrap(core/src/test/java/io/questdb/test/cairo/RssMemoryLimitTest.java (2)
71-71: Good improvement to test specificity.The change from
waitFortowaitForRegexwith an explicit memory tag value makes the test more precise and helps catch regressions in memory tagging behavior.Consider adding a brief comment explaining what memory tag 45 represents (e.g., which allocation category or operation it corresponds to) to aid future maintainers.
125-125: Consistent improvement to test specificity.The regex-based assertion correctly validates the memory tag for SELECT query operations. The pattern matches the change at line 71, maintaining consistency across the test suite.
As with line 71, a comment explaining the significance of memory tag 30 would improve maintainability.
core/src/main/java/io/questdb/cairo/mv/MatViewRefreshJob.java (1)
814-835: Excellent improvements to error handling and diagnostics.This change adds several valuable enhancements:
- Special handling for shutdown interruptions logs at INFO level (not ERROR) and returns gracefully—appropriate since cancellation during shutdown is not a failure.
- Captures and conditionally logs
errnofromCairoExceptionfor better diagnostics, consistent with similar logging elsewhere in the file (lines 998–1005, 1419–1423).- Uses
LogRecordfor structured error logging.Minor: the comment on line 817 says "because a questdb shutdown"; consider "due to questdb shutdown" for clarity.
Apply this diff to improve the comment:
- // The query was cancelled, because a questdb shutdown. + // The query was cancelled due to questdb shutdown.core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursorFactory.java (1)
42-51: Consider adding@NotNullannotation to theengineparameter for consistency.All other object parameters in this constructor are annotated with
@NotNull, but the newengineparameter is not. Adding the annotation would maintain consistency, improve static analysis, and make the contract explicit.Apply this diff:
public LatestByAllIndexedRecordCursorFactory( - CairoEngine engine, + @NotNull CairoEngine engine, @NotNull CairoConfiguration configuration, @NotNull RecordMetadata metadata, @NotNull PartitionFrameCursorFactory partitionFrameCursorFactory,core/src/main/resources/io/questdb/site/conf/server.conf (2)
548-575: Consider consolidating duplicate Parquet configuration blocks.The Parquet export settings (lines 548-575) and partition encoder settings (lines 579-600) are nearly identical, differing only in their prefix (
cairo.parquet.export.*vscairo.partition.encoder.parquet.*). This duplication creates maintenance overhead—any future Parquet configuration updates must be applied to both sections.If these two use cases require different defaults, consider whether a shared base configuration with per-feature overrides would reduce duplication. If the duplication is intentional and necessary, adding a comment explaining why two separate configuration blocks exist would help future maintainers.
Also applies to: 579-600
576-577: Consider relocating HTTP export timeout setting.The
http.export.timeoutsetting is placed between two Parquet configuration blocks, which disrupts the logical grouping of Parquet-specific settings. Consider moving it to:
- Near the
http.export.connection.limitsetting (line 76-78), or- To the beginning of the COPY export section (after line 533)
This would improve configuration discoverability and maintain better logical grouping.
core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursor.java (1)
70-70: Add @NotNull annotation to theengineparameter for consistency.The
engineparameter is used to constructAtomicBooleanCircuitBreakerand would cause a NullPointerException if null. For consistency with other constructor parameters (configuration,metadata,rows,prefixes) that are annotated with@NotNull, consider annotating theengineparameter as well.Apply this diff to add the annotation:
public LatestByAllIndexedRecordCursor( - CairoEngine engine, + @NotNull CairoEngine engine, @NotNull CairoConfiguration configuration, @NotNull @Transient RecordMetadata metadata, int columnIndex, @NotNull DirectLongList rows, @NotNull DirectLongList prefixes ) {core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)
462-467: Consider inverting the condition for better readability.The logic is correct but confusing: the comment mentions "special handling for access control errors," yet the
ifblock handles NON-access errors (due to the negation), and access errors are in theelseblock.Apply this diff to make the control flow match the comment and improve readability:
- // Special handling for access control errors: to distinguish them from other parsing errors for better diagnostics - if (!Chars.startsWith(e.getFlyweightMessage(), "Access")) { - LOG.error().$safe(e.getFlyweightMessage()).I$(); - } else { + // Special handling for access control errors: to distinguish them from other parsing errors for better diagnostics + if (Chars.startsWith(e.getFlyweightMessage(), "Access")) { LOG.error().$("failed to parse message [err: `").$safe(e.getFlyweightMessage()).$("`]").$(); + } else { + LOG.error().$safe(e.getFlyweightMessage()).I$(); }This makes the
ifblock directly handle the case mentioned in the comment (access errors), improving code clarity.core/rust/qdb-core/src/col_type.rs (1)
290-290: Add space before colon in type annotation.Rust style convention prefers a space before the colon in type annotations.
Apply this diff:
- let flag_shifted:i32 = flag << 8; + let flag_shifted: i32 = flag << 8;core/src/main/java/io/questdb/cutlass/http/processors/LineHttpTudCache.java (1)
225-226: Table kind passed on create: LGTMPassing TABLE_KIND_REGULAR_TABLE aligns with the new engine API.
Optionally call securityContext.authorizeTableCreate(TableUtils.TABLE_KIND_REGULAR_TABLE) before create to fail fast at the edge (engine still rechecks).
core/src/test/java/io/questdb/test/cutlass/http/HttpConnectionCountTest.java (2)
258-261: Bound the active-connection spin to avoid test hangsAdd a timeout to the while-loop waiting for connection count to drop to prevent indefinite hangs under flakiness.
-while (serverMain.getActiveConnectionCount(ActiveConnectionTracker.PROCESSOR_ILP) > numOfThreads) { - Os.sleep(50); -} +long deadline = System.currentTimeMillis() + 10_000; // 10s +while (serverMain.getActiveConnectionCount(ActiveConnectionTracker.PROCESSOR_ILP) > numOfThreads) { + if (System.currentTimeMillis() > deadline) { + Assert.fail("Timed out waiting for ILP active connections to drop"); + } + Os.sleep(50); +}
490-513: Helper duplication: consider consolidating contains vs equals assertsassertResponseContains largely duplicates assertResponse. Consider a single helper with a flag or predicate to reduce duplication.
-private void assertResponseContains(HttpClient.ResponseHeaders responseHeaders, int expectedHttpStatusCode, String expectedHttpResponse) { +private void assertResponseMaybeContains(HttpClient.ResponseHeaders responseHeaders, int expectedHttpStatusCode, String expectedHttpResponse, boolean contains) { responseHeaders.clear(); responseHeaders.await(); final Utf8StringSink sink = new Utf8StringSink(); Fragment fragment; final Response response = responseHeaders.getResponse(); while ((fragment = response.recv()) != null) { Utf8s.strCpy(fragment.lo(), fragment.hi(), sink); } - if (!Utf8s.containsAscii(sink, expectedHttpResponse)) { - Assert.fail("Expected response to contain: " + expectedHttpResponse + ", actual: " + sink); - } + if (contains ? !Utf8s.containsAscii(sink, expectedHttpResponse) : !Chars.equalsNc(expectedHttpResponse, sink.toString())) { + Assert.fail("Unexpected response body: " + sink); + } sink.clear(); TestUtils.assertEquals(String.valueOf(expectedHttpStatusCode), responseHeaders.getStatusCode()); }Usage: pass contains=true where partial match is needed.
core/src/main/java/io/questdb/griffin/engine/functions/catalogue/FilesRecordCursor.java (1)
90-100: Prefer FilesFacade for find operations to improve testability.*You inject FilesFacade but call static Files.findFirst/findNext/findClose/findType/findName. Prefer ff.* where available to ease mocking and keep a consistent abstraction.
Also applies to: 131-141, 153-165
core/src/test/java/io/questdb/test/cutlass/http/TestHttpClient.java (2)
387-394: MarkqueryParamsas nullable to match call sites.This parameter is often passed as null; add nullability to aid static analysis.
- protected String reqToSink( + protected String reqToSink( HttpClient.Request req, Utf8StringSink sink, @Nullable CharSequence username, @Nullable CharSequence password, @Nullable CharSequence token, - CharSequenceObjHashMap<String> queryParams + @Nullable CharSequenceObjHashMap<String> queryParams ) {
452-474: Make status check tolerant of reason-phrase differences.Responses may be "200 OK" while callers pass "200". Compare by exact match or prefix with space.
- if (expectedStatus != null) { - if (!expectedStatus.equals(respCode)) { + if (expectedStatus != null) { + final String exp = expectedStatus.toString(); + if (!(respCode.equals(exp) || respCode.startsWith(exp + " "))) { LOG.error().$("unexpected status code received, expected ").$(expectedStatus) .$(", but was ").$(respCode).$(". Response: ").$safe(sink).$(); Assert.fail("expected response code " + expectedStatus + " but got " + respCode); } }core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpsSenderTest.java (1)
90-92: Align expected text with rounded SQL result.Query now uses
round(avg(value), 3)but the expected string concatenates the unroundedexpectedAvg. Depending on formatting (e.g., trailing zeros), this can flake.Option A (robust): format the expected to 3dp using locale‑neutral formatting and keep or strip trailing zeros to match engine output.
+import java.math.BigDecimal; +import java.math.RoundingMode; @@ - TestUtils.assertEventually(() -> serverMain.assertSql( - "select sum(value), max(value), min(value), round(avg(value), 3) from " + tableName, - "sum\tmax\tmin\tround\n" - + expectedSum + "\t" + count + "\t1\t" + expectedAvg + "\n" - )); + TestUtils.assertEventually(() -> { + String expectedAvgStr = new BigDecimal(expectedAvg).setScale(3, RoundingMode.HALF_UP).stripTrailingZeros().toPlainString(); + return serverMain.assertSql( + "select sum(value), max(value), min(value), round(avg(value), 3) from " + tableName, + "sum\tmax\tmin\tround\n" + expectedSum + "\t" + count + "\t1\t" + expectedAvgStr + "\n" + ); + });If the engine always omits trailing zeros for
round, only.stripTrailingZeros()is needed. Please confirm which representation your engine returns.core/src/main/java/io/questdb/cairo/O3PartitionJob.java (1)
409-414: Path handling cleanup looks good.Using
createDirsOrFail(ff, path, ...)(no trailing slash) is consistent and avoids double‑slash mutations.Minor nit: the nearby debug log still calls
path.slash$()which mutates thePathtemporarily; harmless, but consider logging without mutating for clarity.Also applies to: 1096-1101
core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessor.java (1)
257-263: Propagate proper HTTP status from HttpException (enables 429 for connection limit).Hardcoding HTTP_BAD_REQUEST here blocks returning 429 Too Many Requests for connection‑limit errors (issue #6278). Prefer deriving the status from the exception (or mapping a typed connection‑limit exception to 429) and falling back to 400 only when unknown.
Would you confirm whether HttpException exposes a status (e.g., getStatus()/code)? If yes, I can provide a precise diff; otherwise we should introduce a typed exception or a mapper used here and in other HTTP processors.
core/src/main/java/io/questdb/mp/WorkerPoolUtils.java (1)
68-71: Make Rnd seeds per‑worker deterministic and unique.Mix the worker index into seeds to avoid potential collisions when workers start on the same tick.
- final PageFrameReduceJob pageFrameReduceJob = new PageFrameReduceJob( - cairoEngine, - messageBus, - new Rnd(microsecondClock.getTicks(), nanosecondClock.getTicks()) - ); + final long seedLo = microsecondClock.getTicks() ^ (i * 0x9E3779B97F4A7C15L); + final long seedHi = nanosecondClock.getTicks() ^ ((long) i << 32); + final PageFrameReduceJob pageFrameReduceJob = new PageFrameReduceJob( + cairoEngine, + messageBus, + new Rnd(seedLo, seedHi) + );core/src/test/java/io/questdb/test/cairo/SecurityContextTest.java (1)
68-73: Updated single‑arg dispatch matches new API.Passing sc, new ObjHashSet<>(), and TABLE_KIND_REGULAR_TABLE is appropriate for the respective methods.
Optionally, reuse a shared static ObjHashSet test instance to avoid per‑call allocations; not critical for tests.
Also applies to: 105-110, 153-158
core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java (1)
100-125: Consider adding @NotNull annotation for consistency.The
engineparameter is used immediately to construct the circuit breaker (line 125), so null would cause an NPE. For consistency with other factories in this PR (e.g., AsyncJitFilteredRecordCursorFactory, AsyncGroupByRecordCursorFactory), consider annotating the parameter as@NotNull.public GroupByRecordCursorFactory( + @NotNull CairoEngine engine, - CairoEngine engine, CairoConfiguration configuration,core/src/main/java/io/questdb/cairo/SecurityContext.java (1)
103-114: LGTM! Consider adding an inline comment for clarity.The table-kind-based authorization correctly delegates regular table creation to the existing method and allows parquet export in read-only mode. The comment at lines 109-110 explains the intent, but an inline comment explaining why parquet export is allowed in read-only mode would improve maintainability.
case TableUtils.TABLE_KIND_TEMP_PARQUET_EXPORT: - // Allowed even in read-only mode + // Allowed in read-only mode because parquet export uses temporary tables + // that don't persist data modifications return;core/src/main/java/io/questdb/cutlass/line/udp/LineUdpParserImpl.java (1)
405-431: Switch expression refactor preserves semantics.Logic matches prior validation; side-effect assignment for geoHash bits is intact. Optional: add parentheses for clarity around the final OR clause.
- case ColumnType.STRING, ColumnType.VARCHAR -> columnTypeTag == ColumnType.STRING || - columnTypeTag == ColumnType.VARCHAR || - columnTypeTag == ColumnType.CHAR || - columnTypeTag == ColumnType.IPv4 || - isForField && - (geoHashBits = ColumnType.getGeoHashBits(columnType)) != 0; + case ColumnType.STRING, ColumnType.VARCHAR -> + (columnTypeTag == ColumnType.STRING + || columnTypeTag == ColumnType.VARCHAR + || columnTypeTag == ColumnType.CHAR + || columnTypeTag == ColumnType.IPv4) + || (isForField && (geoHashBits = ColumnType.getGeoHashBits(columnType)) != 0);core/src/main/java/io/questdb/cutlass/http/processors/LineHttpPingProcessor.java (1)
44-46: Add @OverRide forgetName().Minor polish: annotate to ensure interface contract is enforced by the compiler.
- public String getName() { + @Override + public String getName() { return ActiveConnectionTracker.PROCESSOR_ILP; }core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperation.java (1)
42-43: Nullability for CopyDataProgressReporterIf reporter is optional, mark accessor and setter param as @nullable to aid static analysis.
Apply:
@@ - CopyDataProgressReporter getCopyDataProgressReporter(); + @org.jetbrains.annotations.Nullable + CopyDataProgressReporter getCopyDataProgressReporter(); @@ - void setCopyDataProgressReporter(CopyDataProgressReporter reporter); + void setCopyDataProgressReporter(@org.jetbrains.annotations.Nullable CopyDataProgressReporter reporter);Also applies to: 67-70
core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestJob.java (2)
102-112: Clarify status transition: WAITING + FINISHEDSetting Phase.WAITING with Status.FINISHED is counterintuitive. If this is a “queued → started” marker, consider a STARTED/IN_PROGRESS status, or document the convention.
95-101: Cancellation check upfront is good; consider also checking after of()A quick second check after serialExporter.of(localTaskCopy) avoids doing work if cancellation happens between the two calls.
@@ - serialExporter.of(localTaskCopy); - phase = serialExporter.process(); // throws CopyExportException + serialExporter.of(localTaskCopy); + if (circuitBreaker.checkIfTripped()) { + throw CopyExportException.instance(phase, -1).put("cancelled by user").setInterruption(true).setCancellation(true); + } + phase = serialExporter.process(); // throws CopyExportExceptioncore/src/main/java/io/questdb/ServerMain.java (1)
392-415: Export worker pool wiring: OK; add tests for “export disabled” advisory pathWhen workerCount == 0, log/advisory path is exercised; add a small test asserting the advisory log to prevent regressions.
core/src/test/java/io/questdb/test/griffin/CopyExportTest.java (1)
325-328: Avoid asserting exact diskSizeHuman values.Exact sizes are brittle across platforms/encodings. Prefer checking presence of expected paths and positive sizes, or assert num_exported_files.
Also applies to: 576-580, 614-616, 992-995, 1409-1413, 1440-1443, 1482-1488
core/src/main/java/io/questdb/griffin/engine/ops/CopyExportFactory.java (3)
158-165: Add tiny backoff in MPSequence spin to reduce CPU burn under contention.- do { - processingCursor = copyRequestPubSeq.next(); - } while (processingCursor == -2); + do { + processingCursor = copyRequestPubSeq.next(); + if (processingCursor == -2) { + io.questdb.std.Os.pause(); + } + } while (processingCursor == -2);
67-69: Avoid coupling to CopyImportFactory.CopyRecord; define a local record.- private final CopyImportFactory.CopyRecord record = new CopyImportFactory.CopyRecord(); + private final CopyRecord record = new CopyRecord(); @@ - static { + public static final class CopyRecord implements io.questdb.cairo.sql.Record { + private CharSequence value; + @Override public CharSequence getStrA(int col) { return value; } + @Override public CharSequence getStrB(int col) { return value; } + @Override public int getStrLen(int col) { return value.length(); } + public void setValue(CharSequence value) { this.value = value; } + } + static { METADATA.add(new TableColumnMetadata("id", ColumnType.STRING)); }Also applies to: 249-252
86-95: General: ensure Parquet libs are on a patched 1.15.1+ (avoid 1.15.0).Parquet 1.15.0 had deserialization issues; prefer 1.15.1+ in the export path dependencies. Based on learnings.
core/src/main/java/io/questdb/cutlass/text/CopyExportResult.java (1)
51-56: Make cleanup idempotent by resetting the flag after deletion.public void cleanUpTempPath(FilesFacade ff) { if (needCleanUp) { path.clear(cleanUpFileLength, path.isAscii()); TableUtils.cleanupDirQuiet(ff, path); + needCleanUp = false; } }core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ImportFilesFunctionFactory.java (2)
70-79: Gate listing behind an auth check (avoid leaking filesystem info).Consider restricting to admins (or a dedicated permission) before exposing import_files():
) throws SqlException { if (Chars.isBlank(configuration.getSqlCopyInputRoot())) { throw SqlException.$(position, "import_files() is disabled ['cairo.sql.copy.root' is not set?]"); } + final io.questdb.cairo.SecurityContext sc = sqlExecutionContext.getSecurityContext(); + if (sc == null || !sc.isSystemAdmin()) { + throw SqlException.$(position, "import_files() requires admin privileges"); + } return new CursorFunction(new ImportFilesCursorFactory(configuration)) {
109-114: Prefer Misc.free(cursor) to ensure native cleanup via our utility.@Override protected void _close() { - Misc.free(importPath); - cursor.close(); + Misc.free(importPath); + Misc.free(cursor); }core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestTask.java (1)
50-63: Defensive null-safety and reset missing in clear().
- getCircuitBreaker()/getSecurityContext() dereference entry; guard against null or document precondition.
- clear() doesn’t reset rawArrayEncoding; reset to a safe default.
Apply:
@@ public void clear() { - this.statisticsEnabled = true; + this.statisticsEnabled = true; + this.rawArrayEncoding = false; @@ - public SqlExecutionCircuitBreaker getCircuitBreaker() { - return entry.getCircuitBreaker(); + public SqlExecutionCircuitBreaker getCircuitBreaker() { + assert entry != null : "entry must be set via of() before use"; + return entry.getCircuitBreaker(); @@ - public SecurityContext getSecurityContext() { - return entry.getSecurityContext(); + public SecurityContext getSecurityContext() { + assert entry != null : "entry must be set via of() before use"; + return entry.getSecurityContext();Also applies to: 65-67, 109-111
core/src/main/java/io/questdb/cutlass/text/CopyImportRequestJob.java (1)
238-253: Redundant setStatusReporter() call (nit).Setting the reporter twice around of()/process() is unnecessary. Keep one.
- parallelImporter.setStatusReporter(updateStatusRef); parallelImporter.of( @@ - parallelImporter.setStatusReporter(updateStatusRef); + parallelImporter.setStatusReporter(updateStatusRef);Also applies to: 251-253
core/src/main/java/io/questdb/cutlass/parquet/SerialParquetExporter.java (1)
305-311: Avoid class-wide synchronization on mkdirs (throughput).createDirsOrFail synchronizes on SerialParquetExporter.class, throttling concurrent exports.
- Remove synchronized block; rely on mkdirs idempotency and handle ENOENT/EEXIST gracefully.
- If needed, synchronize per-target directory or use ff.mkdirs return code only.
core/src/main/java/io/questdb/cutlass/text/CopyExportContext.java (1)
477-492: Ensure visibility of progress fields across threads.Fields written by worker threads and read under locks elsewhere may still be stale without a proper happens‑before; mark them volatile or write under the same lock.
- int finishedPartitionCount = 0; - long id = INACTIVE_COPY_ID; - CopyExportRequestTask.Phase phase; - long populatedRowCount = 0; + volatile int finishedPartitionCount = 0; + volatile long id = INACTIVE_COPY_ID; + volatile CopyExportRequestTask.Phase phase; + volatile long populatedRowCount = 0; @@ - long startTime = Numbers.LONG_NULL; - int totalPartitionCount = 0; - long totalRowCount = 0; + volatile long startTime = Numbers.LONG_NULL; + volatile int totalPartitionCount = 0; + volatile long totalRowCount = 0; @@ - CopyTrigger trigger = CopyTrigger.NONE; - int workerId = -1; + volatile CopyTrigger trigger = CopyTrigger.NONE; + volatile int workerId = -1;core/src/test/java/io/questdb/test/cutlass/http/ExpParquetExportTest.java (2)
873-1040: Add an assertion that over‑limit requests get 429 (issue #6278).The limit test exercises concurrency but doesn’t verify 429. Add a small probe that opens (limit+1) connections and asserts one returns 429.
@@ - try { + try { params.clear(); @@ - try { - for (int i = 0; i < requestLimit; i++) { + try { + for (int i = 0; i < requestLimit; i++) { HttpClient client = HttpClientFactory.newPlainTextInstance(); clients.add(client); HttpClient.ResponseHeaders resp = startExport(client, serverMain, params, "multiple_options_test"); respHeaders.add(resp); } } finally { - for (int i = 0; i < respHeaders.size(); i++) { - respHeaders.get(i).await(); - } + boolean saw429 = false; + for (int i = 0; i < respHeaders.size(); i++) { + respHeaders.get(i).await(); + if ("429".contentEquals(respHeaders.get(i).getStatusCode())) { + saw429 = true; + } + } + Assert.assertTrue("Expected at least one 429 Too Many Requests when exceeding export limit", saw429); for (int i = 0; i < clients.size(); i++) { clients.get(i).close(); }
372-387: Param name consistency.Tests sometimes use “format=csv” while the endpoint reads “fmt”; relying on default CSV works, but use “fmt” for clarity.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (5)
core/src/main/resources/io/questdb/bin/darwin-aarch64/libquestdbr.dylibis excluded by!**/*.dylibcore/src/main/resources/io/questdb/bin/darwin-x86-64/libquestdbr.dylibis excluded by!**/*.dylibcore/src/main/resources/io/questdb/bin/linux-aarch64/libquestdbr.sois excluded by!**/*.socore/src/main/resources/io/questdb/bin/linux-x86-64/libquestdbr.sois excluded by!**/*.socore/src/main/resources/io/questdb/bin/windows-x86-64/questdbr.dllis excluded by!**/*.dll
📒 Files selected for processing (107)
compat/src/test/java/io/questdb/compat/ParquetTest.java(4 hunks)core/rust/qdb-core/src/col_type.rs(1 hunks)core/rust/qdbr/Cargo.toml(1 hunks)core/rust/qdbr/parquet2/Cargo.toml(1 hunks)core/rust/qdbr/src/parquet_read/decode.rs(1 hunks)core/rust/qdbr/src/parquet_read/meta.rs(1 hunks)core/rust/qdbr/src/parquet_write/schema.rs(1 hunks)core/src/main/java/io/questdb/DynamicPropServerConfiguration.java(3 hunks)core/src/main/java/io/questdb/PropHttpServerConfiguration.java(5 hunks)core/src/main/java/io/questdb/PropServerConfiguration.java(22 hunks)core/src/main/java/io/questdb/PropertyKey.java(5 hunks)core/src/main/java/io/questdb/ServerMain.java(6 hunks)core/src/main/java/io/questdb/cairo/CairoConfiguration.java(3 hunks)core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java(3 hunks)core/src/main/java/io/questdb/cairo/CairoEngine.java(11 hunks)core/src/main/java/io/questdb/cairo/DdlListener.java(1 hunks)core/src/main/java/io/questdb/cairo/DefaultCairoConfiguration.java(3 hunks)core/src/main/java/io/questdb/cairo/DefaultDdlListener.java(1 hunks)core/src/main/java/io/questdb/cairo/O3PartitionJob.java(2 hunks)core/src/main/java/io/questdb/cairo/SecurityContext.java(1 hunks)core/src/main/java/io/questdb/cairo/TableUtils.java(4 hunks)core/src/main/java/io/questdb/cairo/TableWriter.java(3 hunks)core/src/main/java/io/questdb/cairo/mv/MatViewRefreshJob.java(3 hunks)core/src/main/java/io/questdb/cairo/sql/AtomicBooleanCircuitBreaker.java(3 hunks)core/src/main/java/io/questdb/cairo/sql/NetworkSqlExecutionCircuitBreaker.java(8 hunks)core/src/main/java/io/questdb/cairo/sql/SqlExecutionCircuitBreaker.java(1 hunks)core/src/main/java/io/questdb/cairo/sql/SqlExecutionCircuitBreakerWrapper.java(2 hunks)core/src/main/java/io/questdb/cairo/sql/async/PageFrameReduceJob.java(3 hunks)core/src/main/java/io/questdb/cairo/sql/async/PageFrameSequence.java(3 hunks)core/src/main/java/io/questdb/cutlass/http/HttpConnectionContext.java(16 hunks)core/src/main/java/io/questdb/cutlass/http/HttpConstants.java(2 hunks)core/src/main/java/io/questdb/cutlass/http/HttpRequestProcessor.java(2 hunks)core/src/main/java/io/questdb/cutlass/http/HttpResponseSink.java(2 hunks)core/src/main/java/io/questdb/cutlass/http/HttpServer.java(6 hunks)core/src/main/java/io/questdb/cutlass/http/processors/ActiveConnectionTracker.java(1 hunks)core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessor.java(22 hunks)core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessorState.java(3 hunks)core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessor.java(2 hunks)core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessorState.java(2 hunks)core/src/main/java/io/questdb/cutlass/http/processors/LineHttpPingProcessor.java(1 hunks)core/src/main/java/io/questdb/cutlass/http/processors/LineHttpProcessorImpl.java(1 hunks)core/src/main/java/io/questdb/cutlass/http/processors/LineHttpTudCache.java(1 hunks)core/src/main/java/io/questdb/cutlass/http/processors/StaticContentProcessor.java(1 hunks)core/src/main/java/io/questdb/cutlass/line/tcp/LineTcpMeasurementScheduler.java(1 hunks)core/src/main/java/io/questdb/cutlass/line/udp/LineUdpParserImpl.java(2 hunks)core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestJob.java(1 hunks)core/src/main/java/io/questdb/cutlass/parquet/CopyExportRequestTask.java(1 hunks)core/src/main/java/io/questdb/cutlass/parquet/SerialParquetExporter.java(1 hunks)core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java(3 hunks)core/src/main/java/io/questdb/cutlass/pgwire/PGServer.java(2 hunks)core/src/main/java/io/questdb/cutlass/text/CairoTextWriter.java(1 hunks)core/src/main/java/io/questdb/cutlass/text/CopyExportContext.java(1 hunks)core/src/main/java/io/questdb/cutlass/text/CopyExportResult.java(1 hunks)core/src/main/java/io/questdb/cutlass/text/CopyImportContext.java(2 hunks)core/src/main/java/io/questdb/cutlass/text/CopyImportRequestJob.java(7 hunks)core/src/main/java/io/questdb/griffin/CompiledQuery.java(1 hunks)core/src/main/java/io/questdb/griffin/CompiledQueryImpl.java(1 hunks)core/src/main/java/io/questdb/griffin/CopyDataProgressReporter.java(1 hunks)core/src/main/java/io/questdb/griffin/SqlCodeGenerator.java(11 hunks)core/src/main/java/io/questdb/griffin/SqlCompilerImpl.java(28 hunks)core/src/main/java/io/questdb/griffin/SqlExecutionContextImpl.java(3 hunks)core/src/main/java/io/questdb/griffin/SqlKeywords.java(4 hunks)core/src/main/java/io/questdb/griffin/SqlParser.java(16 hunks)core/src/main/java/io/questdb/griffin/engine/functions/activity/ExportActivityFunctionFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ExportFilesFunctionFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/functions/catalogue/FilesRecordCursor.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ImportFilesFunctionFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByNotKeyedVectorRecordCursorFactory.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/ops/CopyExportFactory.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/ops/CopyImportFactory.java(6 hunks)core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperation.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperationBuilderImpl.java(7 hunks)core/src/main/java/io/questdb/griffin/engine/ops/CreateTableOperationImpl.java(17 hunks)core/src/main/java/io/questdb/griffin/engine/ops/InsertAsSelectOperationImpl.java(1 hunks)core/src/main/java/io/questdb/griffin/engine/table/AsyncFilteredRecordCursorFactory.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByNotKeyedRecordCursorFactory.java(4 hunks)core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByRecordCursor.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/table/AsyncGroupByRecordCursorFactory.java(4 hunks)core/src/main/java/io/questdb/griffin/engine/table/AsyncJitFilteredRecordCursorFactory.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/table/AsyncTopKRecordCursorFactory.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursor.java(3 hunks)core/src/main/java/io/questdb/griffin/engine/table/LatestByAllIndexedRecordCursorFactory.java(2 hunks)core/src/main/java/io/questdb/griffin/engine/table/parquet/ParquetCompression.java(1 hunks)core/src/main/java/io/questdb/griffin/model/ExportModel.java(1 hunks)core/src/main/java/io/questdb/log/AbstractLogRecord.java(1 hunks)core/src/main/java/io/questdb/mp/WorkerPoolUtils.java(1 hunks)core/src/main/resources/io/questdb/site/conf/server.conf(4 hunks)core/src/test/java/io/questdb/test/AbstractCairoTest.java(1 hunks)core/src/test/java/io/questdb/test/PropServerConfigurationTest.java(7 hunks)core/src/test/java/io/questdb/test/ServerMainTest.java(7 hunks)core/src/test/java/io/questdb/test/cairo/RssMemoryLimitTest.java(2 hunks)core/src/test/java/io/questdb/test/cairo/SecurityContextTest.java(4 hunks)core/src/test/java/io/questdb/test/cairo/fuzz/ParallelGroupByFuzzTest.java(1 hunks)core/src/test/java/io/questdb/test/cairo/mv/MatViewTest.java(1 hunks)core/src/test/java/io/questdb/test/cutlass/http/ExpParquetExportTest.java(1 hunks)core/src/test/java/io/questdb/test/cutlass/http/HttpConnectionCountTest.java(9 hunks)core/src/test/java/io/questdb/test/cutlass/http/HttpErrorHandlingTest.java(2 hunks)core/src/test/java/io/questdb/test/cutlass/http/HttpQueryTestBuilder.java(5 hunks)core/src/test/java/io/questdb/test/cutlass/http/TestHttpClient.java(11 hunks)core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpsSenderTest.java(1 hunks)core/src/test/java/io/questdb/test/cutlass/pgwire/PGErrorHandlingTest.java(2 hunks)core/src/test/java/io/questdb/test/cutlass/pgwire/PGJobContextTest.java(3 hunks)core/src/test/java/io/questdb/test/griffin/AlterTableConvertPartitionTest.java(1 hunks)core/src/test/java/io/questdb/test/griffin/CheckpointTest.java(1 hunks)core/src/test/java/io/questdb/test/griffin/CopyExportTest.java(1 hunks)core/src/test/java/io/questdb/test/griffin/CopyImportTest.java(10 hunks)
⛔ Files not processed due to max files limit (2)
- core/src/test/java/io/questdb/test/griffin/OrderByTimeoutTest.java
- core/src/test/java/io/questdb/test/griffin/ParallelFilterTest.java
✅ Files skipped from review due to trivial changes (5)
- core/rust/qdbr/parquet2/Cargo.toml
- core/src/main/java/io/questdb/log/AbstractLogRecord.java
- core/src/test/java/io/questdb/test/cairo/mv/MatViewTest.java
- core/rust/qdbr/src/parquet_read/decode.rs
- core/rust/qdbr/src/parquet_read/meta.rs
🚧 Files skipped from review as they are similar to previous changes (11)
- core/src/main/java/io/questdb/DynamicPropServerConfiguration.java
- core/src/test/java/io/questdb/test/griffin/CopyImportTest.java
- core/src/test/java/io/questdb/test/PropServerConfigurationTest.java
- core/src/main/java/io/questdb/griffin/CompiledQueryImpl.java
- core/src/main/java/io/questdb/griffin/SqlExecutionContextImpl.java
- core/src/main/java/io/questdb/griffin/engine/functions/catalogue/ExportFilesFunctionFactory.java
- core/src/test/java/io/questdb/test/griffin/AlterTableConvertPartitionTest.java
- core/src/test/java/io/questdb/test/cutlass/pgwire/PGJobContextTest.java
- core/src/test/java/io/questdb/test/AbstractCairoTest.java
- core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java
- core/src/main/java/io/questdb/PropertyKey.java
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (27)
- GitHub Check: New pull request (Coverage Report Coverage Report)
- GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests on linux-arm64)
- GitHub Check: New pull request (Hosted Running tests on windows-other)
- GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin)
- GitHub Check: New pull request (Hosted Running tests on mac-other)
- GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo)
- GitHub Check: New pull request (Hosted Running tests on mac-griffin)
- GitHub Check: New pull request (SelfHosted Other tests Start X64Zfs Agent)
- GitHub Check: New pull request (SelfHosted Other tests Start ARM Agent)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start X64Zfs Agent)
- GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-other)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-pgwire)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-cairo)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-fuzz)
- GitHub Check: New pull request (SelfHosted Griffin And Fuzz tests Start ARM Agent)
- GitHub Check: New pull request (Hosted Running tests with cover on linux-griffin)
- GitHub Check: New pull request (Check Changes Check changes)
core/src/main/java/io/questdb/cairo/sql/AtomicBooleanCircuitBreaker.java
Show resolved
Hide resolved
core/src/main/java/io/questdb/cutlass/http/HttpRequestProcessor.java
Outdated
Show resolved
Hide resolved
core/src/main/java/io/questdb/cutlass/http/processors/ExportQueryProcessor.java
Show resolved
Hide resolved
core/src/test/java/io/questdb/test/cutlass/http/HttpQueryTestBuilder.java
Show resolved
Hide resolved
[PR Coverage check]😍 pass : 2282 / 2585 (88.28%) file detail
|
Co-authored-by: victor <[email protected]> Co-authored-by: Vlad Ilyushchenko <[email protected]> Co-authored-by: ideoma <[email protected]>
|
@nwoolmer excited to see this merge! Are there new docs associated with this? I assume this will be the preferred way to remove data partitions for long term storage and backup. |
|
Yes indeed, for OSS users. There is a draft docs PR that I will get merged. |
Related to questdb/roadmap#97
Closes #6278
Currently, to export
PARQUETfiles, you must convert partitions in-place fromNATIVEtoPARQUETformat. This is inflexible, and blocks writes whilst it occurs.This PR adds support for exporting parquet files via
COPY, improving ergonomics and making it easier for users to extract data as parquet files.Syntax
Changelist
TOoption toCOPYcopy_export_logtableparquetpartition writer.COPY 'table_name' TO 'folder_name' WITH FORMAT PARQUETCOPY (SELECT * FROM x) TO 'folder_name' WITH FORMAT PARQUETCOPYoptions:FORMAT PARQUETPARTITION_BY DAYCOMPRESSION_CODEC UNCOMPRESSED/ZSTD/etc.COMPRESSION_LEVEL NROW_GROUP_SIZE NDATA_PAGE_SIZE NSTATISTICS_ENABLED truePARQUET_VERSION NRAW_ARRAY_ENCODING true/exp/exp?query=<query>&fmt=parquetprovides a parquet file in response instead of csvnot supported yet&partition_by=&compression_codec=LZ4_RAW&compression_level=3&row_group_size=100000&data_page_size=1048576&statistics_enabled=true&parquet_version=1&raw_array_encoding=true@coderabbitai ignore