Repository navigation
fix: bound metric snapshot memory and reduce export overhead - #9505
Conversation
|
/ci |
|
Requested rerun of checks.yml, rust.yml, integration.yml, docs.yml for |
|
@codex review When you finish, append exactly this marker to the review summary: |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <[email protected]>
Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <[email protected]>
Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <[email protected]>
c7b6a74 to
07d8d91
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 07d8d9192d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: jeremyhi <[email protected]>
Signed-off-by: jeremyhi <[email protected]>
Signed-off-by: jeremyhi <[email protected]>
|
@codex review When you finish, append exactly this marker to the review summary: |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <[email protected]>
Signed-off-by: jeremyhi <[email protected]>
|
@codex review When you finish, append exactly this marker to the review summary: |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
WenyXu
left a comment
There was a problem hiding this comment.
src/operator/src/statement/export_logical_tables.rs:609
nit: The inline and blocking paths repeat the slice conversion, backing-size check, and Payload construction. Could we share that final step through a small private synchronous helper? Budget acquisition and the decision to use spawn_blocking_global should remain at the call sites so the fast path still avoids worker dispatch.
src/mito2/src/read/scan_util.rs:1210 — follow-up
nit: For a follow-up cleanup, could we consider renaming this to estimate_split_batch_rows? The should_* prefix suggests a boolean, but this function also returns the estimated row count after splitting. Documenting None as “splitting is not recommended” would make that combined contract clearer. This naming predates the PR, so there is no need to expand this change just for the rename.
src/operator/src/statement/export_logical_tables.rs:885 — follow-up
nit: These batch-conversion helpers are now shared by logical-table export and ordinary COPY. In a follow-up, could we move them into a focused operator-local module such as export_batch? That would reflect their shared responsibility without introducing a generic utils module or moving operator-specific error and budget semantics into a lower-level crate.
PR description — resource-limit wording
nit: Could we make the resource-limit wording explicit? The 8 MiB limit bounds write submissions/parts, while metadata serialization still retains the complete Vec<u8>. Likewise, the Parquet byte threshold triggers a flush after encoding; it is not a strict peak-memory cap. Keeping these distinctions visible would help readers interpret the memory guarantees correctly.
PR description — performance evidence
nit: The new tests demonstrate fewer batches/row groups and reduced retained dictionary backing, but they do not isolate the overall CPU or peak-memory effect of these changes. Arrow dictionary GC also allocates a remapping buffer proportional to the original dictionary length. Could we retain a reproducible reference to the controlled before/after measurements and distinguish end-to-end packed-export results from evidence for the individual optimizations?
Signed-off-by: jeremyhi <[email protected]>
Signed-off-by: jeremyhi <[email protected]>
|
Addressed in 9a56b8f. Extracted 56 focused tests passed on macOS; formatting passed. Windows and the scale matrix were not rerun. |
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports. Signed-off-by: WenyXu <[email protected]>
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports. Signed-off-by: WenyXu <[email protected]>
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports. Signed-off-by: WenyXu <[email protected]>
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports. Signed-off-by: WenyXu <[email protected]>
* fix(hdfs): clean up temporary file when an atomic write fails (#9483) * fix(hdfs): clean up temporary file when an atomic write fails Signed-off-by: fengyue-xve <[email protected]> * style: apply rustfmt to hdfs layer test Signed-off-by: fengyue-xve <[email protected]> * fix(hdfs): report earlier write failures and log cleanup failures - Later `write`/`close` calls on a writer whose failed write was cleaned up now mention the original failure instead of the bare "unavailable after close or abort" message. - `abort_and_delete` logs abort and deletion failures with the temporary path instead of discarding them; the original write error is still returned to the caller. - The temporary-file cleanup test now lists the directory holding the temporary file, so it fails if the cleanup is reverted. Signed-off-by: 风月 <[email protected]> Signed-off-by: fengyue-xve <[email protected]> --------- Signed-off-by: fengyue-xve <[email protected]> Signed-off-by: 风月 <[email protected]> (cherry picked from commit a35661d) Signed-off-by: WenyXu <[email protected]> * fix(jaeger): return the most recent traces when a search hits its limit (#9595) * fix(jaeger): return the most recent traces when a search hits its limit Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search matching more traces than the limit returned an arbitrary subset that changed between identical requests. Order traces by their latest matching span before applying the limit. Signed-off-by: Dennis Zhuang <[email protected]> * refactor(jaeger): name the trace dataframe builder with a verb and document execution Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit cf16c73) Signed-off-by: WenyXu <[email protected]> * fix(pprof): initialize framehop before signal sampling (#9612) fix(pprof): pin upstream framehop initialization fix Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes. Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit 0dc8177) Signed-off-by: WenyXu <[email protected]> * chore: bump rustls to 0.23.46 (#9614) rustls 0.23.46 fixes rustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout. (cherry picked from commit 33879b0) Signed-off-by: WenyXu <[email protected]> * fix(prom): return empty remote read results for missing metrics (#9613) * fix: return empty remote read results for missing metrics Signed-off-by: fys <[email protected]> * test(prom): cover empty reads for missing databases Signed-off-by: fys <[email protected]> * test(prom): simplify remote read regression cases Signed-off-by: fys <[email protected]> * fix(prom): log missing tables during remote read Signed-off-by: fys <[email protected]> --------- Signed-off-by: fys <[email protected]> (cherry picked from commit cf0f6c3) Signed-off-by: WenyXu <[email protected]> * fix(ci): compare release validation against the previous stable version (#9636) Ancestry-based tag selection skips stable tags that live only on older release branches. Signed-off-by: paomian <[email protected]> (cherry picked from commit 3de8610) Signed-off-by: WenyXu <[email protected]> * fix(mito2): keep processing bulk requests after one fails to encode (#9638) * fix(mito2): keep processing bulk requests after one fails to encode When `push_bulk` fails to build the WAL entry of a bulk request, it answers that request with the error. `process_bulk_requests` then returned from the loop, so the remaining requests of the drained batch were dropped without an answer and their callers only saw a receive error. Skip the failed request and continue with the rest of the batch instead. Signed-off-by: jeremyhi <[email protected]> * test(mito2): keep bulk install barriers independent per target The bulk install test barrier was a single process-wide slot, so tests that arm it while running in the same process replaced each other's barrier. Keep one barrier per armed target instead. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 88bdab0) Signed-off-by: WenyXu <[email protected]> * fix(promql): honor @ modifier on subqueries (#9446) * fix(promql): honor @ modifier on subqueries `prom_subquery_expr_to_plan` never read the `at` field of a subquery, so `sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and followed the outer evaluation grid. Handle `@` the way Prometheus does: fold the anchor into the subquery offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier` does), evaluate the subquery once at the start of the evaluation, and replay the folded window over the outer grid, reusing the anchored range selector machinery. A call whose range argument is an anchored subquery is promoted as step-invariant, like one over an anchored range selector. `@ start()`/`@ end()` and `offset` combine as for selectors, and an unrepresentable anchor is rejected with the existing error. Closes #9411 Signed-off-by: Aarav <[email protected]> * test(promql): add sqlness cases for @ on subqueries Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`, `@` combined with `offset` in either order, an instant query away from the anchor, `*_over_time` functions and `rate` over an anchored subquery, a computed inner expression, and an aggregation above the anchored call. Two series per table so a dropped or merged series shows up as a wrong row. Refs #9411 Signed-off-by: Aarav <[email protected]> --------- Signed-off-by: Aarav <[email protected]> (cherry picked from commit 095c288) Signed-off-by: WenyXu <[email protected]> * fix(mito2): validate bulk insert batches against the region schema (#9698) * fix(mito2): validate bulk insert batches against the region schema Bulk inserts were only checked for the time index column, unlike row writes, which WriteRequest::check_schema checks against the region schema. A batch with columns of another type, nulls in the time index or in non-nullable columns, duplicate or unknown columns reached the region worker: a time index that is not a timestamp or holds only nulls made the worker fail, and other mismatches failed after the WAL append or were stored as written, depending on the SST format. Move the column checks of check_schema into a core shared by row writes and bulk inserts, and check bulk insert batches at the same points as rows: when the engine builds the worker request, and again in the worker if the region schema changed since. check_schema keeps its behavior. Bulk insert columns must have the column's Arrow type, except that tags may use any string encoding, as the memtables already convert them. Sparse batches must carry one encoded primary key column of binary without nulls. A batch that misses columns gets them filled by the worker even if it claims to match the current schema. The metric engine checks logical bulk inserts against the logical region before rewriting them into the physical region, as the rewrite copies columns without checking them and would encode a duplicated tag into the primary key twice. Signed-off-by: jeremyhi <[email protected]> * fix(mito2): report an error when a write is interrupted by a panic WriteNotify answers its waiter when it is dropped and reported success unless an error had been set, also when it was dropped while unwinding from a panic in the middle of the write. Report an error in that case. Signed-off-by: jeremyhi <[email protected]> * fix(mito2): tolerate a poisoned series lock when dropping a series map SeriesMap::drop reads every series to update the memtable metrics and unwrapped the lock, so dropping the map while unwinding from a panic that poisoned a series lock panicked again. Read through the poisoned lock instead. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 791f429) Signed-off-by: WenyXu <[email protected]> * fix(mito2): fence writes before a graceful downgrade (#9716) * test(mito2): cover graceful role changes racing with writes A graceful role change flips the region state in a background task while the worker keeps handling writes. Add bulk and row write tests that check a write handled after the role change request is either rejected or covered by the returned last_entry_id. Add tests for DowngradingLeader while an ALTER keeps the region in Altering, alone and in the downgrade sequence the datanode runs (DowngradingLeader, Flush, Follower) with a migration candidate on its own local WAL. Add a test that the downgrade sequence covers a write in flight when a lease renewal sets Downgrading. Signed-off-by: jeremyhi <[email protected]> * fix(mito2): fence writes before a graceful downgrade set_region_role_state_gracefully hands the role change to a background task that waits for the manifest lock. The worker keeps handling writes until the task flips the state, so a write could be acknowledged with an entry id above the returned last_entry_id. Set Downgrading in the worker before spawning the task for Follower and DowngradingLeader requests, as lease renewal already does, so every write the worker handles afterwards is rejected. DowngradingLeader also did nothing and returned success when the region was in another leader state such as Altering. The region became writable again when the operation finished, and the datanode then flushed and turned it into a follower in one step. Return a retryable error instead, so metasrv retries the downgrade. Signed-off-by: jeremyhi <[email protected]> * test(mito2): assert the retryable downgrade error during ALTER Check that DowngradingLeader during an ALTER returns a retryable error, and release the paused ALTER on that path. The local WAL test now retries the downgrade after the ALTER, as metasrv does. Signed-off-by: jeremyhi <[email protected]> * test(mito2): write before the retried downgrade in the local WAL test Rows written before an ALTER were already flushed, so the candidate check did not depend on the downgrade's final flush. Write rows after the ALTER and before the downgrade is retried, which only that flush can carry to the candidate. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 660c4ef) Signed-off-by: WenyXu <[email protected]> * fix(meta): validate persisted partition bounds before altering regions (#9487) * fix(meta): validate persisted partition bounds before altering regions A partition bound is persisted with the type of the column at the time it was written. When an ALTER TABLE changes the type of a column referenced by a partition expression (widening the time index unit is the common case), each region rewrites the bound literal to the new type while building its scan predicate. A millisecond bound like year 3000 fits milliseconds but overflows nanoseconds, so every scan of the region fails once the table schema is updated -- and some regions may already be altered when the failure surfaces. Validate every region's persisted expression on prepare, before any region is modified: parse the persisted JSON (mirrored types, since common-meta cannot depend on the partition crate which depends on it), replay the same value -> ScalarValue conversion and checked cast the region performs, and reject the ALTER naming the offending region when a bound cannot be represented with the new column type. Incompatible changes are rejected atomically; compatible widening proceeds unchanged. Adds meta-level tests (multi-region rejection naming the second region, compatible widening completing) and an end-to-end sqlness case. Fixes #9454 Signed-off-by: fengyue-xve <[email protected]> * fix: access ColumnSchema fields directly and use plain format args ColumnSchema exposes name and data_type as public fields, not methods, and the telemetry macros only accept a format string after the error. Signed-off-by: fengyue-xve <[email protected]> * test: access ColumnSchema data_type field in alter table tests Same fix as the previous commit: data_type is a public field on ColumnSchema, not a method. Signed-off-by: fengyue-xve <[email protected]> * test: drop borrows now that data_type is read as a field Signed-off-by: fengyue-xve <[email protected]> * test: drop useless format! in nested expression test Signed-off-by: fengyue-xve <[email protected]> * test: cover the partition tail in change_col_type sqlness case Signed-off-by: fengyue-xve <[email protected]> * test: drop spurious blank line in change_col_type result Signed-off-by: fengyue-xve <[email protected]> * fix(meta): compare partition bounds with the scan comparison coercion The preflight validator cast the persisted bound directly to the altered column type, which rejected ALTERs the scan itself accepts: the region coerces both the column and the bound to the common comparison type (ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field column with an out-of-range bound still scans. Compare through comparison_coercion(target_type, bound_type) instead and only require the bound to be castable to the common type. Also add a round-trip test in the partition crate that serializes real PartitionExprs and runs them through the validator, so drift between the Persisted* mirror and the partition types fails CI, and fix the error message to match the comparison semantics. Signed-off-by: fengyue-xve <[email protected]> * test: update the sqlness fixture for the new incompatibility message The error message now reflects the comparison semantics: the persisted bound can no longer be compared with the new column type. Signed-off-by: fengyue-xve <[email protected]> * build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock file was not refreshed, so cargo check --locked refused to run on CI. Signed-off-by: fengyue-xve <[email protected]> * fix(meta): reject every non-widening type change of a partitioned column The region routes rows on write through `MultiDimPartitionRule::find_region`, which compares the persisted bound against row values directly: `Value::cmp` panics on mismatched variants and equality misroutes silently. After a column partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT compares `Value::Int32` rows against the persisted `Value::Int64` bound and panics in the frontend, even though the scan-side coercion this check mirrors still works. Reject any type change of a column referenced by a persisted partition expression unless it is a timestamp unit widening, which keeps the bound and the rows comparable because `Timestamp` comparisons convert units first; the widening keeps the scan cast check unchanged. Signed-off-by: fengyue-xve <[email protected]> --------- Signed-off-by: fengyue-xve <[email protected]> (cherry picked from commit 1e29e8d) Signed-off-by: WenyXu <[email protected]> * fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697) * fix(partition): route NULLs like row inserts and accept signed-minimum bounds Signed-off-by: Dennis Zhuang <[email protected]> * fix(partition): keep equality two-valued for NULL rows and accept unsigned -0 Signed-off-by: Dennis Zhuang <[email protected]> * fix(partition): treat NULL masks as false in batch routing and keep comparisons bare Signed-off-by: Dennis Zhuang <[email protected]> * test(compat): update partition filter in analyze verbose expected output Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit da88ff8) Signed-off-by: WenyXu <[email protected]> * fix(mito2): re-check a filled write as sent after a schema change (#9701) * fix(mito2): re-check a filled write as sent after a schema change The engine fills the columns a write leaves out with their defaults when it accepts the write, appending them to the rows without adding them to `name_to_index` and `has_null`. If the region schema changes before the worker handles the write, the worker checks the request again and `check_schema` looks up a filled column in `name_to_index`, which panics and stops the region worker. If the filled column was dropped, the write fails with an unknown column that the client never sent. Remove the filled columns before checking, so the request is checked as sent and filled again for the current schema. Nothing is filled yet when the engine checks the request, so that check does no extra work. Signed-off-by: jeremyhi <[email protected]> * test(mito2): rename put_request helper to build_put_request Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 93c3b11) Signed-off-by: WenyXu <[email protected]> * perf(mito2): gather last rows with one take per input batch (#9737) * perf(mito2): gather last rows with one take per input batch FlatLastTimestampSelector buffered a single-row slice per series and concatenated up to 8192 of them. Concatenating slices with dictionary columns merges the dictionaries once per slice, which made high-cardinality instant PromQL queries using the LastRow scan path (#9034) slower. Select the rows of series that end inside a batch with one take instead. Signed-off-by: Dennis Zhuang <[email protected]> * test(mito2): cover duplicate last timestamps in the take path Signed-off-by: Dennis Zhuang <[email protected]> * fix(mito2): group last rows by primary key bytes Flat batches may assign different dictionary keys to the same primary key, for example when binary keys are wrapped with identity keys. Merge adjacent ranges with equal key bytes before selecting last rows. Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit 4645ab9) Signed-off-by: WenyXu <[email protected]> * fix(mito2): apply SST internal column encodings to bulk-encoded parts (#9735) * fix(mito2): apply SST internal column encodings to bulk-encoded parts Bulk memtable parts are encoded to parquet and flushed as SSTs as-is, but the bulk encoder did not apply the time index / __sequence DELTA_BINARY_PACKED encoding (with dictionary disabled) and the uncompressed op_type setting that the regular SST writer uses. Share these settings through one helper. Signed-off-by: Dennis Zhuang <[email protected]> * chore(mito2): drop the answered bulk encoder compression TODO Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit 977e76d) Signed-off-by: WenyXu <[email protected]> * fix: avoid exact scan row counts for SSTs with unknown counts (#9759) fix: avoid exact scan row counts for legacy SSTs Treat selected SSTs with unknown row counts as non-exact before checking region partition visibility. Preserve exact statistics for known counts and cover legacy metadata mixed with known-count files. Fixes #9718 Signed-off-by: Aumnertic <[email protected]> (cherry picked from commit 6f53d57) Signed-off-by: WenyXu <[email protected]> * fix: avoid duplicate keys in update flow metadata txn (#9816) CREATE OR REPLACE FLOW deleted and then put the same flow route, flownode flow and table flow keys in a single txn when the replacement kept the partitions, flownodes or source tables, which etcd rejects with "duplicate key given in txn request". Only delete the old keys that are not put again. Closes #9810. Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit a1b2362) Signed-off-by: WenyXu <[email protected]> * fix(ci): increase release integration test stack size (#9817) Signed-off-by: evenyag <[email protected]> (cherry picked from commit e6fa1a0) Signed-off-by: WenyXu <[email protected]> * fix(datanode): release region map guard before awaiting role changes (#9824) Signed-off-by: WenyXu <[email protected]> (cherry picked from commit 0dc649e) Signed-off-by: WenyXu <[email protected]> * fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860) * fix(ci): increase Windows nightly test stack size Signed-off-by: evenyag <[email protected]> * fix(ci): align optional macOS artifact test stack size Signed-off-by: evenyag <[email protected]> --------- Signed-off-by: evenyag <[email protected]> (cherry picked from commit 8b1d683) Signed-off-by: WenyXu <[email protected]> * fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898) * fix(mito2): keep SSTs when a manifest save ends with RateLimited The object store's retry layer retries temporary errors and returns the last attempt's error. A manifest delta PUT can be stored by the server while the client sees a timeout, reset or 5xx, and the retries that follow can be throttled, so the save ends with RateLimited even though the delta exists. may_have_persisted_manifest_update treated RateLimited as proof that nothing was written, so flush and compaction deleted the SSTs the stored delta references. A later open or follower sync replays the delta and reads fail on the missing files. Treat RateLimited like other ambiguous errors: the SSTs are kept and become orphans when the save really failed. Fixes #9897 Signed-off-by: jeremyhi <[email protected]> * fix(mito2): keep SSTs when a manifest save ends with PermissionDenied The retry layer returns the last attempt's error, so a 403 from a revoked key, a changed bucket policy or clock skew can follow a delta PUT that the server already stored. Like RateLimited, PermissionDenied does not prove that nothing was written, so it no longer lets flush or compaction delete the SSTs; they become orphans when the save really failed. The compaction test that rejects a manifest write before persistence now injects Unsupported, which is raised before any request is sent. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> Co-authored-by: Yingwen <[email protected]> (cherry picked from commit 1cb01d0) Signed-off-by: WenyXu <[email protected]> * fix: upload staging manifests in chunks (#9919) Signed-off-by: WenyXu <[email protected]> (cherry picked from commit f2be11a) Signed-off-by: WenyXu <[email protected]> * perf: stream and prune SST manifest scans (#9931) * refactor: add reusable metadata pruning and projected SST batches Signed-off-by: WenyXu <[email protected]> * perf(mito2): stream and prune manifest SST metadata Signed-off-by: WenyXu <[email protected]> * perf(datanode): stream inspection queries through reusable providers Signed-off-by: WenyXu <[email protected]> * perf(catalog): push manifest filters through inspection queries Signed-off-by: WenyXu <[email protected]> * refactor: align manifest column constants with module conventions Signed-off-by: WenyXu <[email protected]> * perf: reduce manifest inspection allocations Signed-off-by: WenyXu <[email protected]> * refactor: narrow manifest inspection API visibility Signed-off-by: WenyXu <[email protected]> * refactor: retain residual filters for manifest inspection Signed-off-by: WenyXu <[email protected]> --------- Signed-off-by: WenyXu <[email protected]> (cherry picked from commit c96cdd0) Signed-off-by: WenyXu <[email protected]> * feat: improve SQL analyze scan diagnostics (#9565) * feat: improve SQL analyze scan diagnostics Signed-off-by: evenyag <[email protected]> * test: redact dynamic filters in sqlness results Signed-off-by: evenyag <[email protected]> * test: stabilize analyze snapshots for merge queue Signed-off-by: evenyag <[email protected]> * test: remove obsolete scan_details assertion Signed-off-by: evenyag <[email protected]> * test: refresh explain snapshots after partition filter changes Signed-off-by: evenyag <[email protected]> --------- Signed-off-by: evenyag <[email protected]> (cherry picked from commit 3e2751d) Signed-off-by: WenyXu <[email protected]> * fix(query): avoid lossy casts in partition pruning (#9600) * fix(query): keep lossy casts out of partition pruning Signed-off-by: discord9 <[email protected]> * test(query): align cast pruning expectations with conservative fallback Signed-off-by: discord9 <[email protected]> * test(query): record lossy cast partition pruning regression Signed-off-by: discord9 <[email protected]> * fix(query): retain safe pruning conjuncts beside unsupported casts Signed-off-by: discord9 <[email protected]> * test(query): refresh cast pruning plans after rebase Signed-off-by: discord9 <[email protected]> --------- Signed-off-by: discord9 <[email protected]> (cherry picked from commit df8f684) Signed-off-by: WenyXu <[email protected]> * fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601) * fix(query): skip LastRow pushdown for IGNORE NULLS Signed-off-by: discord9 <[email protected]> * fix(query): keep IGNORE NULLS aggregates above remote scans Signed-off-by: discord9 <[email protected]> * test(query): record IGNORE NULLS results before and after flush Signed-off-by: discord9 <[email protected]> * test(query): share IGNORE NULLS coverage across deployments Signed-off-by: discord9 <[email protected]> * test(query): refresh last-value plans after rebase Signed-off-by: discord9 <[email protected]> --------- Signed-off-by: discord9 <[email protected]> (cherry picked from commit 0dab3ae) Signed-off-by: WenyXu <[email protected]> * fix(meta): share peer discovery with plugin hooks (#9977) Signed-off-by: WenyXu <[email protected]> (cherry picked from commit 9f0e58e) Signed-off-by: WenyXu <[email protected]> * fix: adapt release backport test dependencies Copy the writer observation test utility from #9505 (18726c7) for the #9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports. Signed-off-by: WenyXu <[email protected]> * chore: bump version to 1.3.0 Signed-off-by: WenyXu <[email protected]> * fix(query): pin DataFusion cast constraint propagation fix Update the pinned GreptimeTeam/datafusion revision from e3221b8 to cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger regression test). CastExpr::propagate_constraints used to cast a filter's output interval straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so FilterExec advertised k as constant, the equivalence classes dropped the ordering required by ORDER BY k, and rows returned in scan order. The sqlness case keeps every #9600 assertion and the explicit ORDER BY, and adds ascending and descending checks for the partitioned table, the control table and a table whose matching rows are stored in neither ascending nor descending k order. Cargo.lock only rewrites the 35 datafusion source lines. Signed-off-by: discord9 <[email protected]> (cherry picked from commit fe637b0) Signed-off-by: WenyXu <[email protected]> --------- Signed-off-by: fengyue-xve <[email protected]> Signed-off-by: 风月 <[email protected]> Signed-off-by: WenyXu <[email protected]> Signed-off-by: Dennis Zhuang <[email protected]> Signed-off-by: Lei, HUANG <[email protected]> Signed-off-by: fys <[email protected]> Signed-off-by: paomian <[email protected]> Signed-off-by: jeremyhi <[email protected]> Signed-off-by: Aarav <[email protected]> Signed-off-by: Aumnertic <[email protected]> Signed-off-by: evenyag <[email protected]> Signed-off-by: discord9 <[email protected]> Signed-off-by: discord9 <[email protected]> Co-authored-by: fengyue-xve <[email protected]> Co-authored-by: dennis zhuang <[email protected]> Co-authored-by: Lei, HUANG <[email protected]> Co-authored-by: fys <[email protected]> Co-authored-by: localhost <[email protected]> Co-authored-by: jeremyhi <[email protected]> Co-authored-by: DeviousCardi <[email protected]> Co-authored-by: Aum <[email protected]> Co-authored-by: Yingwen <[email protected]> Co-authored-by: discord9 <[email protected]>
* fix: bound metric snapshot memory and reduce export overhead (#9505) * fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <[email protected]> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <[email protected]> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <[email protected]> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <[email protected]> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <[email protected]> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <[email protected]> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <[email protected]> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <[email protected]> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <[email protected]> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 18726c760257047af4029692ce4ce147fdc3d35c) Signed-off-by: WenyXu <[email protected]> * fix(prom): align remote read label matcher semantics (#9633) * fix(prom): align remote read label matcher semantics Signed-off-by: fys <[email protected]> * chore: optimize comment of validate_matchers * fix(prom): preserve existing binary label regex handling Signed-off-by: fys <[email protected]> * refactor: condition * test(prom): simplify remote read matcher tests and clarify comments * fix(prom): return binary remote read labels as UTF-8 * fix(prom): reject binary labels in remote read * refactor(prom): align remote read null label handling with PromQL Signed-off-by: fys <[email protected]> * refactor(prom): simplify null handling in label matcher branches Signed-off-by: fys <[email protected]> * fix(prom): restrict remote read labels to string types Signed-off-by: fys <[email protected]> * test(prom): use string label in default value column test Signed-off-by: fys <[email protected]> * fix: code review --------- Signed-off-by: fys <[email protected]> (cherry picked from commit 4e8bc6df86bbb1aeb6636bec0bebfcf5d5b49cad) Signed-off-by: WenyXu <[email protected]> * fix(mito2): restore source splitting for cross-range series merges (#9818) * fix: restore batched series merges across ranges Signed-off-by: WenyXu <[email protected]> * fix(mito2): split range sources before last-row selection Signed-off-by: WenyXu <[email protected]> * fix(mito2): avoid tiny splits for single-source cross-range merges Signed-off-by: WenyXu <[email protected]> --------- Signed-off-by: WenyXu <[email protected]> (cherry picked from commit 27c531336a3b7dece7712c682c53e277e4626d28) Signed-off-by: WenyXu <[email protected]> * fix(mito2): pick TWCS L1 merges by closed units so compaction terminates (#9741) * refactor(mito2): pick TWCS L1 merges by closed units TWCS could keep rewriting L1 files without new writes. Overlap-only picks could split their inputs into one more file than they read, and L1 file reduction then rewrote up to 16 files to remove that single file. The outputs, split by key, created new overlaps and the cycle repeated. Replace the three global pick phases with one pick per window, scheduled by class across windows. Every pick now decreases a termination measure when the window receives no writes. - Active windows only take merges that remove files. L1 files are grouped by conflicts and merged in key order, so size-split tails are merged together. A reduction rewrites at most 4 output files' worth of bytes per removed file. - Inactive windows absorb L0 files, then take closed merges: no other file intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each merge resolves conflicts or removes files, decreasing (conflicts, files). Files that do not take part in an overlap are not rewritten. - Overlaps that only share a boundary key no longer count as conflicts, so a size-split sorted run does not look overlapping. - Closed merges may take 4x the input limit to resolve a whole window once. - Cache primary key ranges per pick instead of resolving them per comparison. - Log the pick branch of each compaction output. Signed-off-by: Lei, HUANG <[email protected]> * docs(mito2): state the compactor assumptions behind TWCS termination Signed-off-by: Lei, HUANG <[email protected]> * feat(mito2): resolve active window overlaps on manual compaction A window that stops receiving writes keeps being the active window, whose picks only remove files, so its overlaps were never resolved. Manual compactions now pick every window as inactive, so a closed merge can resolve them. Automatic compactions are unchanged. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): defer overlap resolution in the window before the active one Late data keeps landing in the window right before the active window. A closed merge that resolves its overlaps early rewrites the whole window, then the late data brings the overlaps back, and the balanced run check keeps them there. Overlap-only merges now skip that window until it is older. File-reducing merges and manual compactions are unchanged. On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead of 2.08 at the same write amplification (3.01 vs 3.02). Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito2): only check earlier-starting files when closing units Closing units only checks files that start before a unit ends, and the overlap checks compare key bounds without cloning them. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): keep settling windows from rewriting L1 files The window before the active one still receives late data. Skipping only its overlap-only merges was not enough: file-reducing merges of conflicting files and mixed L0/L1 merges still rewrote its L1 files, and a mixed merge could rewrite a nearly full output for each small late file. A settling window now only merges L0 files among themselves and conflict-free files; the window resolves its overlaps once it is older. On the out-of-order bench, write amplification drops from 3.88 to 3.30 at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with overlap depth 1.0 in all three. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): resolve late overlaps once a window is quiet Late data that lands in a window older than the settling one is merged into a small file covering the whole key range. Resolving it rewrites the window's dominant sorted run, which the balanced run check refuses, so the overlap stayed. Removing the check instead rewrites the window for every late flush. A window is quiet once the region has written at least as many rows as the window holds since its newest file. Overlap merges in a quiet window may rewrite a dominant run, so each quiet period pays for at most one such rewrite. Sequences are per row and kept by compaction, so this needs no state. On the bench with late data up to 2 hours, mean overlap depth drops from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of 2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The remaining depth is in the final active window. Signed-off-by: Lei, HUANG <[email protected]> * test(mito2): check closed merges of quiet and settling windows terminate Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): split TWCS closures over the input limit A file spanning the keys of a large sorted run joins every run file into one closure. Once a closure held more files than one compaction accepts, closed merges skipped it, so the window never merged those files again and late files kept piling up. Split such closures into single files, like active-window picks do, and replace the closed check with a conflict bound: outputs only conflict with files intruding on the inputs' box, so a merge is taken if `outputs · I` does not exceed the conflicts its inputs had with other files. Intervals of whole closures have no such conflicts and still need to be closed, so other windows pick as before, and `(conflicts, files)` still strictly decreases. Small run files and late files over a large run can be merged again. A late file over a run that cannot shrink stays: resolving it would rewrite the whole run. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): address TWCS picker review findings - Manual compactions may rewrite a dominant sorted run to resolve overlaps. The active window is never quiet, so manual compactions could not resolve its late-data overlaps before. - Single-key intervals predict one output: the writer never splits a series, so large files of one series were never merged. - The active window falls back to single files when no group of conflicting files qualifies, so a large file no longer keeps its small peers unmerged. - Compute the settling window distance with checked_sub, which overflowed for windows at the timestamp limits. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): count inner keys as TWCS conflicts and check balance per group - A single key strictly inside another file's key range now conflicts, so a late file of one series inside a wide file is resolved like other overlaps. Files sharing only a boundary key still do not conflict. Intrusion covers the new pairs, which a randomized test checks. - Check the dominant-run guard per group of conflicting files. Runs of separate groups never overlap, so pairing one group's run with another group's peer made an interval of two dominant runs look balanced. - A window with unknown row counts, written before they were recorded, is not quiet: its threshold would drop to a single row. Signed-off-by: Lei, HUANG <[email protected]> * perf(mito2): index TWCS files by key to avoid quadratic grouping Grouping a window's files compared every file with all files sharing its time range, and closing groups and checking intrusions scanned the whole window. A sorted run split by key shares one time range, so a window of N such files took O(N²) checks without any conflict. The 1024-file fallback only covers the sorted-run path, not these groups. Index the files by lower primary key in a segment tree that keeps each subtree's largest upper key and its time bounds. Grouping, closing, and the conflict bound only check the files whose key and time ranges meet, so a file of a key-sliced run only meets its neighbors. Conflicts among split files are precomputed per file and counted per interval with a binary search instead of pairwise. On a run of 20,000 key-sliced files with 5 late files spanning all keys, a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick ~19 ms instead of ~3.5 s. Signed-off-by: Lei, HUANG <[email protected]> * test(object-store): retry transient S3 transport failures Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): bound TWCS closed-merge conflict graph work Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph. The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): charge TWCS index nodes to the conflict-graph budget The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted. Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): retry smaller TWCS dense compaction candidates If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): compact shared primary-key boundaries Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint. Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): compact small L1 files in inactive windows Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks. Signed-off-by: Lei, HUANG <[email protected]> * docs(mito2): clarify TWCS unit grouping guarantees Signed-off-by: Lei, HUANG <[email protected]> --------- Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit bb3f07b4825e074c207ae29db28de1e3354c11ae) Signed-off-by: WenyXu <[email protected]> * fix: reclaim HDFS temporary files left behind by a datanode crash (#9616) * fix: reclaim HDFS temporary files left behind by a datanode crash Signed-off-by: lczllx <[email protected]> * fix(mito2): stop the temporary file sweep from starving GC jobs Signed-off-by: lczllx <[email protected]> * feat(mito2): add metrics for the temporary file sweep Signed-off-by: lczllx <[email protected]> * refactor(object-store): share one definition of the temporary file name Signed-off-by: lczllx <[email protected]> * fix(mito2): skip the temporary file sweep when no GC job is allowed Signed-off-by: lczllx <[email protected]> * refactor: gate the temporary file sweep behind the hdfs-object-store feature Signed-off-by: lczllx <[email protected]> * fix(mito2): sweep temporary files when a create opens an existing region Signed-off-by: lczllx <[email protected]> * docs(config): document the temporary file sweep in the GC settings Signed-off-by: lczllx <[email protected]> * refactor: align the temporary file sweep code with the surrounding style Signed-off-by: lczllx <[email protected]> --------- Signed-off-by: lczllx <[email protected]> (cherry picked from commit 88cb2a986ef0776a1eb84b0df9581b3e7ffb189d) Signed-off-by: WenyXu <[email protected]> * fix(meta): fence datanode processes replaced by a newer process (#9878) * fix(meta): fence datanode processes replaced by a newer process Two datanode processes can run with the same node id, and metasrv grants leases and delivers instructions by node id only. Every datanode heartbeat already carries node_epoch, the time the process started. The metasrv leader records the highest accepted epoch per node id under __datanode_epoch/{node_id}, a prefix of the leader cache, and raises it with compare-and-put. The heartbeat session checks each datanode request with a non-zero epoch before the handler chain: a lower epoch gets a reply without running the chain, so no lease, keep-alive, stats or mailbox reply. An epoch more than 5 minutes ahead of the leader's clock is not recorded and is fenced too. Datanode pushers carry the epoch of their process: a lower epoch cannot replace a higher one, and a stream removes only its own pusher. Signed-off-by: jeremyhi <[email protected]> * fix(meta): keep the leader cache load and pusher updates coherent A write made while the leader cache is disabled, as during the load at leader start, went to the store without the cache noticing, so the load could install a value read before the write. Such writes now create new cache versions around the store call, and the load drops what it loaded for a prefix if a write happened meanwhile. This also covers node addresses, which heartbeats write during the load. A stream's pusher that takes its first epoch is updated in place instead of being replaced, so messages waiting on it are not cancelled, and the connection gauge counts only a pusher that fills an empty slot. The epoch check moves into the heartbeat service. Signed-off-by: jeremyhi <[email protected]> * fix(meta): keep the leader cache coherent with writes while it is disabled A write that started while the cache was disabled could commit after the load had read the old value and the cache had been enabled, leaving the old value cached. Writes now maintain the cache whether or not it is enabled, as enabled writes already did: new versions around the store call, then the key is updated or invalidated. Signed-off-by: jeremyhi <[email protected]> * test(meta): pin the epoch 0 exemption and pusher order, and update docs The datanode test without an epoch now runs after another process has recorded one, and the pusher test checks that a stream's first accepted request receives what its handlers broadcast. The leader cache load no longer claims that writes must not run while it loads, and the meta-srv guide states that epoch 0 is not checked. Signed-off-by: jeremyhi <[email protected]> * fix(meta): register a datanode pusher only after its first request passes A datanode stream registered an epoch-0 pusher on its handshake, so an older process's reconnecting stream could take an empty slot, or a newer process's pusher before its first heartbeat, for about a round trip. A datanode stream now registers its pusher on its first request that passes the epoch check (epoch-0 requests after the handshake included), still before the handlers run. Pusher registration is logged only when the insert happens. Tests cover a fenced stream removing its own pusher, a handshake not taking an empty slot, and a KV error failing the request; two steps that could not fail are removed. Signed-off-by: jeremyhi <[email protected]> * fix(meta): check a datanode epoch again before replying A request could pass the epoch check, wait in the handler chain while a newer process of the node was recorded, and still return a region lease. After the handlers run, the session checks the epoch again against the cached record and, if a newer process has been recorded, replies as to a fenced request and removes the stream's own pusher. This costs one more read of the cached record per checked datanode request. Signed-off-by: jeremyhi <[email protected]> * fix(meta): handle a datanode's requests one at a time A request of an older process could pass the epoch check, wait in the handlers while a newer process was admitted, and then keep the node alive with its own address. Checking the epoch again after the handlers only dropped the reply. The heartbeat session now takes an in-memory lock for the node id before checking a request that carries an epoch, and releases it after the handlers, before the reply is sent. A newer process is therefore admitted only after the handlers of an older request finish. The check after the handlers is removed. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 6f5c064f543eedfa65d312bebe64f33870c3ca60) Signed-off-by: WenyXu <[email protected]> * fix: acknowledge writes only within the region's leader lease (#9904) * fix: acknowledge writes only within the region's leader lease Signed-off-by: jeremyhi <[email protected]> * fix: drop cached logical columns after a failed logical drop Signed-off-by: jeremyhi <[email protected]> * fix: keep the prepared region for the lease check after the WAL append Signed-off-by: jeremyhi <[email protected]> * fix: serialize a logical drop with fills of the logical column cache Signed-off-by: jeremyhi <[email protected]> * test: check the drop lock and grant long leases deterministically Signed-off-by: jeremyhi <[email protected]> * test: cover a bulk insert that loses its lease during the WAL append Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 91a59b82fbff9d1443968024dc5df1c96e661d4e) Signed-off-by: WenyXu <[email protected]> * fix(mito): preserve SST lifetime across index replacement (#9950) * fix(mito): preserve SST lifetime across index replacement Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity. Fixes #9944 Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): list-based SST purge and single FilePurger::remove_file The final purge of a retired SST no longer depends on FileMeta: index files of every version are discovered by listing ({region_dir}/index/{file_id}. prefix), so even when the last live view is a stale one, the purge still removes all newer index files. - SstLifetime is slimmed down to { deleted, file_purger }; the purge now runs in the last FileHandleInner::drop (Arc::strong_count == 1, which is race-free here since only inners of the same file hold the lifetime). - FilePurger is merged back into a single remove_file(file_meta, is_delete) method; is_delete means the file was retired AND this is the last live view. - LocalFilePurger deletes the SST and all its index files via the new delete_file_and_all_indexes helper (list before delete, so a listing failure leaves the file untouched). - Remove the dead index_outdated flag and its accessors, which had no callers in the entire repo. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito): elect final SST view release atomically Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface. Add a deterministic concurrent-drop regression covering retired and live files. Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): restore Arc-owned SST cleanup and clarify reference release Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing. Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test. Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): track SST references by file lifetime Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling. Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers. Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes. Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): release SST references directly on lifetime drop Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge. Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior. Signed-off-by: Lei, HUANG <[email protected]> * test(mito): verify SST cleanup with real storage and isolate GC policy cases Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings. Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged. Signed-off-by: Lei, HUANG <[email protected]> * style(mito): format SST lifetime changes Signed-off-by: Lei, HUANG <[email protected]> * fix(mito): reclaim obsolete indexes by reader version Track exact index-version references per metadata view and restore version-specific Puffin liveness in both GC paths. Keep file-level Parquet protection and shared SST lifetime cleanup, without expanding snapshots into unreferenced index generations. Include old readers of imported files even when the source region is not open locally. Add real-storage GC coverage for reader release and lingering while retaining Parquet and the current index. Validated workspace checks with default and all features, all-target all-feature Clippy with warnings denied, and all 1853 mito2 tests. Signed-off-by: Lei, HUANG <[email protected]> --------- Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit 17ed988f49804af40dc79ff81f6c751b18be1dd1) Signed-off-by: WenyXu <[email protected]> * fix(mito2): coordinate range cache invalidation and singleflight reads (#9935) * fix(mito2): invalidate retired SST range results and coalesce loads Coordinate decoded range loads by their full scan key, bound shared work, and prevent late fills after SST retirement. Invalidate range results for local purge and object-store GC while preserving old snapshot reads. Fixes #9934 Signed-off-by: Lei, HUANG <[email protected]> * fix(mito): preserve SST lifetime across index replacement Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity. Fixes #9944 Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): list-based SST purge and single FilePurger::remove_file The final purge of a retired SST no longer depends on FileMeta: index files of every version are discovered by listing ({region_dir}/index/{file_id}. prefix), so even when the last live view is a stale one, the purge still removes all newer index files. - SstLifetime is slimmed down to { deleted, file_purger }; the purge now runs in the last FileHandleInner::drop (Arc::strong_count == 1, which is race-free here since only inners of the same file hold the lifetime). - FilePurger is merged back into a single remove_file(file_meta, is_delete) method; is_delete means the file was retired AND this is the last live view. - LocalFilePurger deletes the SST and all its index files via the new delete_file_and_all_indexes helper (list before delete, so a listing failure leaves the file untouched). - Remove the dead index_outdated flag and its accessors, which had no callers in the entire repo. Signed-off-by: Lei, HUANG <[email protected]> * fix(mito): elect final SST view release atomically Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface. Add a deterministic concurrent-drop regression covering retired and live files. Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): restore Arc-owned SST cleanup and clarify reference release Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing. Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test. Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): track SST references by file lifetime Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling. Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers. Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes. Signed-off-by: Lei, HUANG <[email protected]> * refactor(mito): release SST references directly on lifetime drop Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge. Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior. Signed-off-by: Lei, HUANG <[email protected]> * test(mito): verify SST cleanup with real storage and isolate GC policy cases Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings. Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged. Signed-off-by: Lei, HUANG <[email protected]> * style(mito): format SST lifetime changes Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): use crate-rooted range cache imports Signed-off-by: Lei, HUANG <[email protected]> * fix(mito2): account for range cache delivery lifetime Keep completed range results charged while watch subscribers or replay streams retain them. Cache entries keep weak delivery references, allowing cache-only ownership to release the delivery reservation. Cache hits reuse or nonblockingly acquire the reservation, falling back when the budget is exhausted. Resize input reservations to completed-result estimates and cover successive slow consumers, cache-only handoff, hit sharing, invalidation, admission rejection, and cancellation with regression tests. Signed-off-by: Lei, HUANG <[email protected]> --------- Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit 44df6e5dedfa6bddfa6c03ad4dd362b3564b38ce) Signed-off-by: WenyXu <[email protected]> * fix(meta): exclude route followers from failover targets (#10001) * fix(meta): exclude route followers from failover targets Failover target selection did not exclude the region's route followers. The selector could keep picking a follower, the submit rejected it as a peer conflict, and the region never failed over. Group the failed regions by their followers and exclude them per group. A group without a candidate is skipped instead of failing every region of the datanode. Signed-off-by: jeremyhi <[email protected]> * test: check failover continues after a group without candidates Signed-off-by: jeremyhi <[email protected]> * fix(meta): keep failover going when follower routes can't be read Fall back to selecting without follower exclusion when the routes can't be read, log skipped groups with their followers at warn level, and test a region without a route. Signed-off-by: jeremyhi <[email protected]> * refactor(meta): select failover targets per exclusion set in select_peers generate_failover_tasks now only reads the route followers of the regions. select_peers resolves the affinity and the unschedulable datanodes once, then selects per group of regions that exclude the same datanodes. Signed-off-by: jeremyhi <[email protected]> * refactor(meta): drop a comment that repeats the code Signed-off-by: jeremyhi <[email protected]> * refactor(meta): tighten failover target selection Name the route followers after the parameter they fill, read the routes with unwrap_or_else, and log a group without a candidate as an error, as the caller did before. Signed-off-by: jeremyhi <[email protected]> * fix(meta): let the region-stat-aware selector balance the whole batch The region-stat-aware selector now receives each region's excluded peers along with the whole batch, so it can still balance load across the regions while keeping a region off its own followers. A region without a candidate is left out of its output instead of failing the others. The selectors driven by SelectorOptions keep selecting per group of regions that exclude the same datanodes. Signed-off-by: jeremyhi <[email protected]> * test: name the batch type of the stat-aware recorder Signed-off-by: jeremyhi <[email protected]> * refactor(meta): treat a selector without candidates the same in both arms Both selector arms now place what they can and treat NoEnoughAvailableNode as nothing placed; the stat-aware selector's contract says so. Rename its parameter to exclude_peer_ids_by_region, check for unplaced regions in linear time, and describe what the code does in the comments. Signed-off-by: jeremyhi <[email protected]> * test: keep the stat-aware recorder's placements off excluded peers Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 62ebf71f90a6167966c1f3e64ed6d01dfc57b14e) Signed-off-by: WenyXu <[email protected]> * fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715) * fix(log-store): make raft-engine periodic WAL sync actually fsync On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and write() returns before fsync (tikv/raft-engine#395). The periodic sync task therefore never durable-flushed acknowledged WAL data. Work around by writing a minimal non-empty marker with sync=true so the fsync path is reached until a crates.io release includes tikv/raft-engine#400. Fixes #9712 Signed-off-by: Yvan Wang <[email protected]> * perf(log-store): reuse LogBatch across periodic WAL syncs Keep the periodic sync task's LogBatch across ticks instead of allocating a fresh one (4 KiB encode buffer plus item vec) every period. A successful Engine::write drains the batch but retains its buffers; after a failed write or a panicked blocking task the batch is discarded and recreated on the next tick, since it may be left partially encoded. Keep the marker put rather than deleting an empty key: the put overwrites a single key, whereas every delete would become a recovery tombstone that raft-engine keeps in memory while replaying append files. Also document that the marker value must not decode as a NamespaceImpl (the marker key lies inside the range scanned by list_namespaces), add a test covering batch reuse and namespace listing, and apply rustfmt. Refs #9712 Signed-off-by: Yvan Wang <[email protected]> --------- Signed-off-by: Yvan Wang <[email protected]> (cherry picked from commit 180e2716fc5fb3f192a6f6cd57eac308892ccdcc) Signed-off-by: WenyXu <[email protected]> * fix(catalog): propagate frontend process-list failures (#9999) * fix(catalog): propagate frontend process-list failures Return the existing invocation error when a selected frontend cannot list processes, so a successful response cannot omit that peer. Signed-off-by: Byte-Naut <[email protected]> * test(frontend): cover remote process-list errors and aggregation Exercise real metadata discovery and gRPC listeners for a missing management service and successful catalog-scoped aggregation. Signed-off-by: Byte-Naut <[email protected]> --------- Signed-off-by: Byte-Naut <[email protected]> (cherry picked from commit 3d2b73762a6275a4bb44400e750fea4aeda87478) Signed-off-by: WenyXu <[email protected]> * fix(log-store): skip records with unparsable keys instead of aborting the read (#9749) * fix(log-store): skip records with unparsable keys instead of aborting the read Signed-off-by: lczllx <[email protected]> * fix(log-store): keep undecodable RecordMeta keys fatal Signed-off-by: lczllx <[email protected]> --------- Signed-off-by: lczllx <[email protected]> (cherry picked from commit 5f0e22171ed54c4bb1245492648e2b5d04704b68) Signed-off-by: WenyXu <[email protected]> * fix(meta): register failure detectors for regions switched before a rollback (#9989) * fix(meta): register failure detectors for regions switched before a rollback A region migration switches the routes of its tables one table at a time. If it aborts after some tables were switched, those regions are already served by the target, but the rollback deregistered the target's failure detectors for every region and registered only the source's. The target's next heartbeat registers them again, so if the target stopped before sending one, nothing detected its failure and the switched regions stayed unavailable until the metasrv leader changed or the target came back. The rollback now collects, from the routes it already reads, the regions whose leader is the target and resets the target's failure detectors for them at the end. It also skips a table whose route no longer exists, so a table dropped meanwhile no longer fails the rollback before the detectors are handled. Signed-off-by: jeremyhi <[email protected]> * test(meta): roll back a downgrading leader in the rollback detector test The table that was not switched yet keeps a downgrading leader in a real abort, so the test now builds it that way and checks that the rollback clears it. Signed-off-by: jeremyhi <[email protected]> * fix(meta): monitor regions switched before a rollback only on the target The rollback registered the source's failure detectors for every region of the migration, including the regions already switched to the target. Those regions are served by the target after the rollback, so their source detectors watched the wrong peer: when one fired, the failover found the leader had changed and only removed it. The rollback now registers the source's detectors only for the regions that were not switched, and keeps resetting the target's detectors for the others. Signed-off-by: jeremyhi <[email protected]> * refactor(meta): rename reset_failure_detectors_for_candidate_regions to reset_failure_detectors Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 05536cd7f52a3b5fbe96f3d3f672b6e9c3855b3e) Signed-off-by: WenyXu <[email protected]> * fix: ignore stale leader grants after a region is downgraded (#9987) * fix: ignore stale leader grants after a region is downgraded Signed-off-by: jeremyhi <[email protected]> * fix: compare stale leader grants with the longest lease seen Signed-off-by: jeremyhi <[email protected]> * fix: treat a lease too long to store as unknown A lease longer than a u64 of nanoseconds saturated to a shorter one, so the derived send time could be later than the real one. Now such a lease makes the send time unknown, which counts as earlier than any gate. Signed-off-by: jeremyhi <[email protected]> * fix: address review comments on stale leader grants Scope the early send time claim to after a lease has been seen, drop the unused role parameter of demote, clarify a test comment and check that a fresh grant still makes a reopened region writable. Signed-off-by: jeremyhi <[email protected]> * refactor: compare stale leader grants with the heartbeat send time Pass the send time of the heartbeat behind a lease to set_region_role_with_lease, and return whether the role was applied. Mito compares the send time with the grant gate directly, so it no longer derives it from the longest lease seen. The datanode finalizes leadership only when the grant was applied. Signed-off-by: jeremyhi <[email protected]> * test: cover stale grants through the datanode and repeated downgrades Signed-off-by: jeremyhi <[email protected]> * refactor: keep leader grant rules in one region method Apply a role grant in MitoRegion::apply_role_grant, which renews the lease and decides whether the role changes; the engine only looks up the region. Rename the gate to downgrade_started_at and demote to begin_downgrade, keep the field private, and copy it on reopen through inherit_downgrade_state_from. Signed-off-by: jeremyhi <[email protected]> * refactor: drop a needless borrow Signed-off-by: jeremyhi <[email protected]> * refactor: keep lease concepts in the engine and role ordering in the region ManifestContext, which owns the role, records when the role was last demoted and promotes a region only for a promotion issued after that. The time starts when the context is created, so a reopened region ignores grants for heartbeats sent before the reopen, without copying state. The engine translates leases: it renews the deadline and promotes with the heartbeat send time. MitoRegion keeps no grant state. Signed-off-by: jeremyhi <[email protected]> * fix: keep the data region of a metric region off a grant its metadata region ignores The metric engine applies a grant to the metadata region first and stops if it is ignored, so the data region, whose role the engine reports, never becomes a leader that skips finalization. Also describe what promoting returns. Signed-off-by: jeremyhi <[email protected]> * docs: describe when a lease grant is ignored and flatten the finalize check Signed-off-by: jeremyhi <[email protected]> * refactor: move lease types out of the region and the Leader deadline into ManifestContext Signed-off-by: jeremyhi <[email protected]> * refactor: name the promotion fence after the downgrade it records Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 89a5df31418add4a168c7318d4cae55b90ddce00) Signed-off-by: WenyXu <[email protected]> * fix(mito2): preserve live index files across remote compactions (#10046) fix(mito2): reuse executor-owned compaction index resources Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit c7ea80558f9cf53a8feaa4fb6fdcdd07060f4ddf) Signed-off-by: WenyXu <[email protected]> * fix(datanode): pick a node epoch above the recorded one (#9984) * fix(datanode): pick a node epoch above the recorded one The metasrv fences a datanode whose node epoch is lower than the highest one it has recorded for the node id. A datanode used its start time as its epoch, so after a restart on a host whose clock is behind the previous process's, or after the clock stepped back, the new process picked a lower epoch and was fenced for good: no region lease, no keep-alive. Before it creates the heartbeat task, the datanode now reads the recorded epoch through its metadata backend and uses one more than it if its start time is not higher. The node info still reports the local start time. A failed read fails the startup, like the other metadata reads. Signed-off-by: jeremyhi <[email protected]> * test(datanode): tighten the node epoch tests Use a lower record that leaves room below the start time, so the test fails if the start time is ignored, and a nonzero node id, so it fails if the node id is ignored. Clarify the doc comment of the epoch choice. This assumes the previous process of a node id has stopped before a new one starts. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit c40e86c6533eac94b28652f0255f7cd07ea86c15) Signed-off-by: WenyXu <[email protected]> * fix(auth): preserve physical metric table targets (#10084) * fix(auth): preserve physical metric table targets Signed-off-by: shuiyisong <[email protected]> * fix: nit Signed-off-by: shuiyisong <[email protected]> --------- Signed-off-by: shuiyisong <[email protected]> (cherry picked from commit c17a5e09f74069a7e62b9a512d6d95229344c219) Signed-off-by: WenyXu <[email protected]> * fix: include peer context in distributed query errors (#10118) Signed-off-by: WenyXu <[email protected]> (cherry picked from commit 8deb3872b3283b01ebc629a5d1642b7053b65fbb) Signed-off-by: WenyXu <[email protected]> * fix(auth): redact inline user-provider credentials (#10115) * fix(auth): redact inline user-provider credentials Signed-off-by: shuiyisong <[email protected]> * fix(auth): close user-provider redaction gaps Signed-off-by: shuiyisong <[email protected]> --------- Signed-off-by: shuiyisong <[email protected]> (cherry picked from commit adf29740b28147cfb9035f53e1c41515b5b519b3) Signed-off-by: WenyXu <[email protected]> * fix: register trigger procedure loaders through DdlManager (#10123) Add a loader registration hook to TriggerDdlManager and invoke it before procedure recovery. Test that registration reaches the configured trigger DDL manager. Signed-off-by: fys <[email protected]> (cherry picked from commit b6cf3384dca74f7871c0746e60918e5b2852e7c8) Signed-off-by: WenyXu <[email protected]> * fix(time): parse zero-hour timezone offsets correctly (#10091) fix(time): preserve zero-hour timezone offset signs Signed-off-by: Minh Vu <[email protected]> (cherry picked from commit 25003cea16453526f37b77fa4189c68b52a49a3b) Signed-off-by: WenyXu <[email protected]> * fix(meta): invalidate all datanode tables on address change (#9986) When a datanode restarts with a new address, metasrv invalidated cached table routes only for the tables in that heartbeat's region stats. The first heartbeat after a restart often reports few or no regions (e.g. with init_regions_in_background), so frontends kept dialing the old address until metadata_cache_ttl expired. Read the previous address via prev_kv when saving it, and when it changed (or a failed save of the same epoch is retried) invalidate every table registered for the datanode in __dn_table. Signed-off-by: Floris Smit <[email protected]> Co-authored-by: Weny Xu <[email protected]> (cherry picked from commit 351f4207b4446e48befc02b03779849013fc2886) Signed-off-by: WenyXu <[email protected]> * fix(function): require ADMIN statement privileges to execute admin functions (#10120) * fix(function): require ADMIN statement privileges to execute admin functions Read-only accounts could execute the admin UDFs (flush_table, compact_table, flush_flow, gc_regions, migrate_region, reconcile_*, build_index, ...) through ordinary SELECT statements: SELECT is a read-only statement so it passes the statement-level privilege check, and the functions were only guarded by the greptime-catalog scope check inside the UDF. That let read-only users trigger WAL flushes, manifest version bumps, compactions, flow-sink writes and, in distributed mode, region GC/migration and SST deletion. Wrap the privilege check into the admin_fn macro itself, next to the existing greptime-catalog check: before executing, every #[admin_fn] function now requires the same privileges as its ADMIN statement. The check synthesizes the PermissionReq::SqlStatement request an ADMIN <fn>(...) statement produces and evaluates it with the same PermissionCheckerRef plugin the frontend installs, so query invocation (SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path are gated identically — in particular read-only accounts are denied with PermissionDenied (7006), while write-privileged users keep both invocation styles. With auth disabled (no permission checker), the functions stay invocable by everyone, exactly like ADMIN statements. The denial is wrapped as External(BoxedError) so the 7006 status code survives both the query execution path and the admin statement executor. Regression tests: unit tests for the check (no-checker allows, read-only denied 7006, read-write allowed), an end-to-end test over the macro-generated flush_table UDF, and an integration test where a read-only account is denied 7006 on all 13 functions from SELECT while a read-write account keeps SELECT and ADMIN FLUSH_TABLE working. Signed-off-by: Ning Sun <[email protected]> * fix(function): expose admin privilege check to downstream macros Signed-off-by: Ning Sun <[email protected]> --------- Signed-off-by: Ning Sun <[email protected]> (cherry picked from commit b21b1cdb693a0017b82ca667f347688e6fd6cbba) Signed-off-by: WenyXu <[email protected]> * fix(catalog): use advertised peer address as process identity (#10153) * fix(catalog): use advertised peer address as process identity The process manager identified the local frontend by its public gRPC address, while metasrv discovers peers by the address advertised in the heartbeat (the internal gRPC address when configured). With separate public and internal listeners, list_all_processes failed to exclude the local node, queried its own management service as a remote frontend, and appended the local registry again, so every local query was listed twice. The same mismatch broke routing of KILL to other frontends. Identify a frontend by the address it advertises to metasrv: - add FrontendOptions::peer_addr, shared by the heartbeat and the process manager, and FrontendOptions::create_process_manager to wire it once; - use the advertised address as the process identity, for self-exclusion in list_all_processes, and for local/remote routing in kill_process. Process ids now carry the address other frontends use to reach the node. When no internal gRPC listener is configured this is the public gRPC address, so the previous behavior is unchanged. The internal listener is the only one serving the frontend management service, so process ids stay routable. The new distributed regression tests exercise separate public and internal addresses: the local query must be listed exactly once while remote frontends are still listed, and KILL must route to a remote frontend. Signed-off-by: Lei, HUANG <[email protected]> * test: keep local peer address reserved in kill-routing test The test reserved the local frontend address with a temporary listener that was dropped immediately, then reserved the remote address separately. The kernel could assign the same port to the remote frontend, so the remote kill request would take the local branch and fail to find the remote process. Bind the local address once and keep the listener alive for the whole test so the local and remote frontend identities cannot collide. Signed-off-by: Lei, HUANG <[email protected]> --------- Signed-off-by: Lei, HUANG <[email protected]> (cherry picked from commit 557843dd680bf3ac94befea55377583442732d58) Signed-off-by: WenyXu <[email protected]> * fix(mito): use stored recovery-set bytes for manifest size (#10142) * fix(mito): track stored bytes for manifest recovery Signed-off-by: WenyXu <[email protected]> * fix(mito): retain manifest sizes on checkpoint and sync Signed-off-by: WenyXu <[email protected]> * test(mito): cover logical manifest size accounting Signed-off-by: WenyXu <[email protected]> * docs(mito): clarify manifest size retention invariants Signed-off-by: WenyXu <[email protected]> --------- Signed-off-by: WenyXu <[email protected]> (cherry picked from commit d7036df73c99d496510eec22bbf70cd731e93d5f) Signed-off-by: WenyXu <[email protected]> * fix(grpc): propagate cancellation for read query execution (#10166) * x * fix(grpc): restrict cascading cancellation to read query variants Signed-off-by: luofucong <[email protected]> --------- Signed-off-by: luofucong <[email protected]> (cherry picked from commit 2cb473607d8328489978040be67b8f44366fd7e9) Signed-off-by: WenyXu <[email protected]> * fix(datatypes): escape object keys when rendering jsonb as text (#10112) * fix(datatypes): escape object keys when rendering jsonb as text Signed-off-by: Yvan Wang <[email protected]> * fix(datatypes): skip unicode normalization for escaped jsonb keys Signed-off-by: Yvan Wang <[email protected]> * refactor(datatypes): serialize jsonb to text directly with serde_json Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that both object keys and string values are escaped by serde_json. This drops fix_unicode_point, its regex and the hand-written key escaping. Errors are propagated instead of falling back to lossy raw bytes. Characters such as U+2028 are now emitted as-is, which is still valid JSON, so the affected sqlness result lines are regenerated. Signed-off-by: Yvan Wang <[email protected]> * fix(datatypes): accept JSON text bytes in jsonb_to_string Some write paths (COPY FROM csv/json/parquet, JSON bound as text via MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode such bytes as jsonb before serializing with serde_json, so they still render (properly escaped) instead of failing with InvalidJsonb. Signed-off-by: Yvan Wang <[email protected]> --------- Signed-off-by: Yvan Wang <[email protected]> (cherry picked from commit ec1ad1f15b79b6e24e86ce52cd0578a3b7d979bc) Signed-off-by: WenyXu <[email protected]> * fix(mito2): share bulk memtable compaction dispatcher across workers (#10022) Each region worker built its own MemtableBuilderProvider, and every provider created its own CompactDispatcher. The limit on background bulk memtable compactions was therefore num_workers times max_background_compactions, while SST compaction uses one engine-wide pool of max_background_compactions. Create one dispatcher in WorkerGroup::start (and start_for_test), pass it to every worker, and make MemtableBuilderProvider::new take it instead of creating its own. SST and memtable compaction keep separate budgets of the same size. Signed-off-by: luckys <[email protected]> Co-authored-by: luckys <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: dennis zhuang <[email protected]> (cherry picked from commit 9f7cedf58f92b89c79e5f32f41ed78c1ce311472) Signed-off-by: WenyXu <[email protected]> * fix(mito): decode cached manifest payloads (#10179) * test(mito2): cover compressed manifest cache recovery Signed-off-by: WenyXu <[email protected]> * fix(mito2): decode cached manifest object payloads Signed-off-by: WenyXu <[email protected]> * fix(mito2): address manifest cache review feedback Signed-off-by: WenyXu <[email protected]> --------- Signed-off-by: WenyXu <[email protected]> (cherry picked from commit 4263f5fc016b65a1f56fe99a6b2a4aac8bdcf6cf) Signed-off-by: WenyXu <[email protected]> * fix: prioritize region ALTER over queued edits (#10193) * fix: prioritize region ALTER over queued edits Signed-off-by: evenyag <[email protected]> * test: use valid edits in ALTER priority regression Signed-off-by: evenyag <[email protected]> * refactor: make region edit dispatch outcomes explicit Signed-off-by: evenyag <[email protected]> * fix: reject queued alters and edits when dropping regions Signed-off-by: evenyag <[email protected]> * test: cover queued ALTER handoff before region edits Signed-off-by: evenyag <[email protected]> --------- Signed-off-by: evenyag <[email protected]> (cherry picked from commit ae422ef46532836a7b3203888773055b589b9bbb) Signed-off-by: WenyXu <[email protected]> * fix(meta-srv): invalidate leader cache on transaction errors (#10219) Signed-off-by: WenyXu <[email protected]> (cherry picked from commit 4688fa4be83dde7e513c393093c2a170a1e540d3) Signed-off-by: WenyXu <[email protected]> * fix(mito2): avoid reporting reader filter metrics twice (#10222) * perf(mito2): cache fixed-label reader metric counters Signed-off-by: Dennis Zhuang <[email protected]> * fix(mito2): report reader filter metrics only once Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit 5d88d283e5551d7ad14f1f1ac3ee451a1c9c4445) Signed-off-by: WenyXu <[email protected]> * perf(mito2): reuse static manifest checks across row groups (#10220) * perf(mito2): reuse static manifest checks for cached readers Signed-off-by: Dennis Zhuang <[email protected]> * test(mito2): drop redundant pruner test and release entry lock early Signed-off-by: Dennis Zhuang <[email protected]> * perf(mito2): cache static manifest keep verdicts directly Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit c844d447218ac341004d7154f48925b0a8c6747a) Signed-off-by: WenyXu <[email protected]> * fix: adapt v1.3.1 backports to release interfaces Signed-off-by: WenyXu <[email protected]> * fix!: enforce sparse primary key schema (#10020) * fix: preserve required sorting for ordinary sparse scans Signed-off-by: shuiyisong <[email protected]> * fix: enforce sparse primary key schema Signed-off-by: shuiyisong <[email protected]> * test: fix Signed-off-by: shuiyisong <[email protected]> --------- Signed-off-by: shuiyisong <[email protected]> (cherry picked from commit efed847ca6dadd37a76fbbaca2a454621c505a92) Signed-off-by: WenyXu <[email protected]> * fix: test (#10149) fix: fixture Signed-off-by: shuiyisong <[email protected]> (cherry picked from commit 4633c37dbbe9ae369732a8023ca5c0a23f17e9cf) Signed-off-by: WenyXu <[email protected]> * fix: tolerate invalid UTF-8 in JSON log ingestion (#10245) * fix: tolerate invalid UTF-8 in JSON ingestion Signed-off-by: shuiyisong <[email protected]> * fix: tolerate invalid UTF-8 in NDJSON ingestion Signed-off-by: shuiyisong <[email protected]> * chore: upgrade simd json Signed-off-by: shuiyisong <[email protected]> * feat: add server default for ignore_errors Signed-off-by: shuiyisong <[email protected]> --------- Signed-off-by: shuiyisong <[email protected]> (cherry picked from commit 0cb70d4f84b2fe1b4d6988fe75c970f862c2794a) Signed-off-by: WenyXu <[email protected]> * test: simplify backported JSON recovery cases Signed-off-by: WenyXu <[email protected]> * feat(otlp): add v1 schema for auto-created OTLP log tables (#10207) * feat(otlp): add v1 schema for auto-created OTLP log tables New tables created by direct OTLP logs ingestion get a v1 built-in schema: service_name tag from resource service.name ahead of scope_name in the primary key, a skipping index on trace_id, and event_name and observed_timestamp fields. The table is marked with table_data_model=greptime_log_v1. Existing tables without the marker keep the legacy schema. Signed-off-by: Dennis Zhuang <[email protected]> * refactor(otlp): share resource service.name parsing between logs and traces Log tables now derive service_name with the same rules as trace spans, so both signals agree on the value used to correlate them. The integration test writes twice to cover the existing v1 table path. Signed-off-by: Dennis Zhuang <[email protected]> --------- Signed-off-by: Dennis Zhuang <[email protected]> (cherry picked from commit 2714687ab701a2bf046b579a2aa2e255552b9df8) Signed-off-by: WenyXu <[email protected]> * fix(promql): bound scatter time filter expression depth (#10272) Signed-off-by: evenyag <[email protected]> (cherry picked from commit 361f3595db03f6297d8d45fba3fec924e6e7a776) Signed-off-by: WenyXu <[email protected]> * fix(query): use current partition columns in prepared plans (#10244) A prepared statement keeps its logical plan, and that plan holds the table as it was at prepare time. The analyzer reads the partition columns from that table while the regions to scan come from the current route. After a table's partition columns change, the analyzer can consider a query safe to push down, so each region runs the whole query and the frontend concatenates their results. The same happens to the source of a prepared INSERT ... SELECT, which then stores wrong values. Before executing a kept plan, compare each scanned table's partition columns with the current table found by id. Scans of a table whose columns changed get the current table when the schema is unchanged. Otherwise they keep their schema, because a scan's projection holds indices into it, and report the current columns as absent from it, so the query keeps merging on the frontend. Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 297a343fd45f07ea267eb88d79eb592f5a912240) Signed-off-by: WenyXu <[email protected]> * fix(time): avoid i32 overflow when converting an out-of-range date (#10299) Signed-off-by: Arpit Jain <[email protected]> Co-authored-by: Claude <[email protected]> (cherry picked from commit 402e4a97fc0b7c082cc419b380c6014388dd1a10) Signed-off-by: WenyXu <[email protected]> * feat: update dashboard to v0.14.5 (#10270) Co-authored-by: dennis zhuang <[email protected]> (cherry picked from commit 4c4c8a58fd85ef38feef90f5a076e4e1b863e2d7) Signed-off-by: WenyXu <[email protected]> * fix(mito2): keep rows written during staging in the WAL (#10253) * fix(mito2): keep rows written during staging in the WAL A flush during staging committed its SSTs to the staging manifest, then dropped the memtable and released the WAL. If staging did not end successfully those rows were gone. A flush that committed after staging had ended lost its rows even though staging succeeded. Staging regions no longer flush. They are skipped by memory-pressure, periodic and region-limit flushes, and their writes are rejected instead of queued at twice the region limit or when the node's staging memtables reach half of the engine's flush limit. A manual or repartition flush on a staging region with data gets a retryable error; the migration pre-flush and the WAL prune flush reply success without flushing. Closing a staging region leaves staging first, so skip-WAL rows are flushed to the normal manifest. Signed-off-by: jeremyhi <[email protected]> * fix(mito2): keep accounting and closing staging regions during staged edits Signed-off-by: jeremyhi <[email protected]> * fix(mito2): track staged edits without relying on the partition info Signed-off-by: jeremyhi <[email protected]> * test(mito2): synchronize staging close tests with worker admission Signed-off-by: jeremyhi <[email protected]> * test(mito2): observe the manifest lock wait when closing staging regions Signed-off-by: jeremyhi <[email protected]> * fix(mito2): detect staged edits through the staging partition info Signed-off-by: jeremyhi <[email protected]> * docs(mito2): record that staging regions must not flush Signed-off-by: jeremyhi <[email protected]> * fix(mito2): do not restore staging after a close abandons the edit Signed-off-by: jeremyhi <[email protected]> * fix(mito2): reject region edits during a close-time flush Signed-off-by: jeremyhi <[email protected]> * style(mito2): format the staging tests Signed-off-by: jeremyhi <[email protected]> * fix(mito2): reject staging edits once staging is abandoned Signed-off-by: jeremyhi <[email protected]> * fix(mito2): bind apply-staging edits to their partition expression Signed-off-by: jeremyhi <[email protected]> * docs(mito2): note that staging regions pin their topic's prune point Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> (cherry picked from commit 8da533c35ffc901db22827c62ed2877f2e33e6c5) Signed-off-by: WenyXu <[email protected]> * perf(query): push column projections through MergeScan (#10221) * perf(query): push column projections through MergeScan Signed-off-by: Dennis Zhuang <[email protected]> * fix(query): retain empty projections and refresh sqlness plans Signed-off-by: Dennis Zhuang <[email protected]> * test(query): update TSID join plans after projection pushdown Signed-off-by: Dennis Zhuang…
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
Part of #9120 (PR13).
What's changed and what's your intention?
Avoid splitting a single-source scan into tiny batches when it bypasses the merge reader. Bound snapshot metadata write submissions/parts to 8 MiB and invoke writer abort on failure.
Compact unused dictionary values before retained slices reach the merge. Apply the existing series-density heuristic to L0 files with known statistics, avoiding excessive tiny batches that exhausted the query pool on the representative wide/multi-SST fixture; preserve the fallback for older files without statistics. Use larger Parquet row groups, preserve compatible dictionaries, bound batch conversion, and omit redundant routing work in the filtered single-table plan. Multi-table routing, snapshot formats and APIs remain unchanged; the experimental gate remains enabled.
Validation: 78 targeted release tests passed, including real SST scans with multiple single-source ranges, compaction, writer-budget release on completion/cancellation, and 16 supported Metric type/layout round trips. Splitting now follows the current reader’s actual source count, including sources remaining after pruning, while preserving compaction’s cross-range merge. The wide/multi-SST case passes with a 2 GiB query pool and query parallelism 4. Native fault tests also pass process-crash and HTTP 504 recovery, plus two-region in-flight cancellation/retry at query parallelism 1/2/4, with exact values/DDL, fresh IDs and import-state cleanup.
The L0 fix additionally passed 111 targeted release scan/merge/dedup tests, including overlapping L0 files with ordered latest-value checks; formatting passed. The review cleanup passed 56 focused release nextest tests across snapshot storage, logical-table export, packed writer and Parquet writer on macOS; formatting passed. Windows was not rerun locally.
Resource guarantees: the 8 MiB limit bounds write submissions/parts; metadata serialization still retains the complete
Vec<u8>. The Parquet byte threshold triggers flushing after encoding and is not a strict peak-memory cap. The metadata failure test verifies thatabort()is invoked after write/close failures; it does not prove removal of partial output on every backend.Evidence scope: the A/D tables below compare ordinary export/restore against packed export plus batch DDL using the same binary. They measure the combined end-to-end paths, not the isolated CPU or peak-memory benefit of each optimization. Focused regression tests establish batch/row-group behavior, shared or reduced dictionary backing, and budget/cancellation behavior. They are not controlled before/after CPU or peak-RSS measurements. Arrow dictionary GC itself allocates a remapping buffer proportional to the original dictionary length.
The acceptance ledger retains the comparison method, fixtures, resource limits, request counts and validity qualifications. The supplementary wide-fixture measurements use the L0 fix represented by
24377f9008(only an explanatory comment differs from the tested source); the earlier accepted narrow-fixture pair remains separately recorded in #9120. This review cleanup is validated with targeted regression tests; the scale matrix is not rerun. Raw measurements and replay inputs remain in the local acceptance archive.Consolidate useful PoC conclusions and limitations into the main RFC and remove the separate experiment report.
For 100,000 logical tables sharing one physical table, each with a timestamp, two string tags and one DOUBLE field (32 rows per table; 3.2 million rows), one valid A/D pair at +100 ms per S3 request passed full data, schema, remapped-ID, import-state and procedure checks:
This pair had zero host swap-out, no guest paging/OOM or compilation, and normal sampled memory pressure throughout. Whole-case swap-in, including untimed checks, was 33,296 KiB. Both paths used the same native macOS release binary, 2 GiB query pool/q4, MinIO 2 CPU/1 GiB, Docker VM 2 GiB/swap disabled, and identical allocator decay settings.
Performance acceptance is complete under the single-pair scope agreed on 2026-10-12: D improved full export by 120.3x and full restore by 230.7x, exceeding the 3x target for each. These are single-pair ratios; repeatability, timing spread and three-round medians were not established. The two remaining planned pairs are canceled. Earlier invalid observations remain excluded. See #9120 for the accepted scope and remaining documentation work. The feature remains opt-in behind the experimental gate; removing the gate is outside this acceptance.
The supplementary representative-shape comparison is complete: 10 physical tables × 10,000 logical tables, 300 user columns per physical table (298 possible string tags, timestamp and DOUBLE), and 10 overlapping tags plus timestamp and DOUBLE per logical table. Each logical table has 32 rows (3.2 million total); S3 adds +100 ms per HTTP request. One D-then-A pair used the same fixed release binary, frozen source, equal limits and fresh destinations.
Both paths passed full typed-value/DDL, membership, remapped-ID, import-state and procedure checks for all 100,000 tables. Logical-create procedures fell from 100,000 to 790. The L0 scan fix resolved the 2 GiB query-pool exhaustion: D completed with a sampled export query-pool peak of 229.2 MiB and no observed rejection. Writes/parts and D range reads stayed within 8 MiB.
This supplementary pair is not a strictly resource-valid performance sample: A restore had 17.875 MiB host swap-out. The observed export/restore ratios are 89.5x/180.0x; the restore ratio is qualified by that interference. D had zero swap-out, including full correctness validation, and both export phases had zero swap-out. All 1,313 whole-case pressure samples were normal; there was no guest paging, MinIO swap/OOM or concurrent compilation. Whole-case swap-in was 83,888 KiB. This does not establish repeatability or replace the earlier accepted narrow-fixture pair.
The database used a 2 GiB query pool/q4 and a 256 MiB page cache; MinIO remained at 2 CPU/1 GiB and the Docker VM at 2 GiB with swap disabled. Limits were unchanged between controls. Raw evidence is retained outside the repository. The experimental switch remains opt-in.
PR Checklist
Please convert it to a draft if some of the following conditions are not met.
backport-<target>labels (e.g.backport-v1.3targetsrelease/v1.3).