Skip to content

fix: bound metric snapshot memory and reduce export overhead - #9505

Merged
fengjiachun merged 10 commits into
GreptimeTeam:mainfrom
fengjiachun:test/metric-snapshot-acceptance
Oct 12, 2026
Merged

fengjiachun merged 10 commits into
GreptimeTeam:mainfrom
fengjiachun:test/metric-snapshot-acceptance

Conversation

@fengjiachun

@fengjiachun fengjiachun commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

Part of #9120 (PR13).

What's changed and what's your intention?

Avoid splitting a single-source scan into tiny batches when it bypasses the merge reader. Bound snapshot metadata write submissions/parts to 8 MiB and invoke writer abort on failure.

Compact unused dictionary values before retained slices reach the merge. Apply the existing series-density heuristic to L0 files with known statistics, avoiding excessive tiny batches that exhausted the query pool on the representative wide/multi-SST fixture; preserve the fallback for older files without statistics. Use larger Parquet row groups, preserve compatible dictionaries, bound batch conversion, and omit redundant routing work in the filtered single-table plan. Multi-table routing, snapshot formats and APIs remain unchanged; the experimental gate remains enabled.

Validation: 78 targeted release tests passed, including real SST scans with multiple single-source ranges, compaction, writer-budget release on completion/cancellation, and 16 supported Metric type/layout round trips. Splitting now follows the current reader’s actual source count, including sources remaining after pruning, while preserving compaction’s cross-range merge. The wide/multi-SST case passes with a 2 GiB query pool and query parallelism 4. Native fault tests also pass process-crash and HTTP 504 recovery, plus two-region in-flight cancellation/retry at query parallelism 1/2/4, with exact values/DDL, fresh IDs and import-state cleanup.

The L0 fix additionally passed 111 targeted release scan/merge/dedup tests, including overlapping L0 files with ordered latest-value checks; formatting passed. The review cleanup passed 56 focused release nextest tests across snapshot storage, logical-table export, packed writer and Parquet writer on macOS; formatting passed. Windows was not rerun locally.

Resource guarantees: the 8 MiB limit bounds write submissions/parts; metadata serialization still retains the complete Vec<u8>. The Parquet byte threshold triggers flushing after encoding and is not a strict peak-memory cap. The metadata failure test verifies that abort() is invoked after write/close failures; it does not prove removal of partial output on every backend.

Evidence scope: the A/D tables below compare ordinary export/restore against packed export plus batch DDL using the same binary. They measure the combined end-to-end paths, not the isolated CPU or peak-memory benefit of each optimization. Focused regression tests establish batch/row-group behavior, shared or reduced dictionary backing, and budget/cancellation behavior. They are not controlled before/after CPU or peak-RSS measurements. Arrow dictionary GC itself allocates a remapping buffer proportional to the original dictionary length.

The acceptance ledger retains the comparison method, fixtures, resource limits, request counts and validity qualifications. The supplementary wide-fixture measurements use the L0 fix represented by 24377f9008 (only an explanatory comment differs from the tested source); the earlier accepted narrow-fixture pair remains separately recorded in #9120. This review cleanup is validated with targeted regression tests; the scale matrix is not rerun. Raw measurements and replay inputs remain in the local acceptance archive.

Consolidate useful PoC conclusions and limitations into the main RFC and remove the separate experiment report.

For 100,000 logical tables sharing one physical table, each with a timestamp, two string tags and one DOUBLE field (32 rows per table; 3.2 million rows), one valid A/D pair at +100 ms per S3 request passed full data, schema, remapped-ID, import-state and procedure checks:

Metric (single valid pair) Ordinary (A) Packed + batch DDL (D)
Full export 3055.646 s 25.400 s
Full restore 6640.371 s 28.777 s
Export S3 requests 100,111 42
Restore S3 requests 600,204 32
Export host swap-in 6,320 KiB 64 KiB
Restore host swap-in 24,464 KiB 0 KiB

This pair had zero host swap-out, no guest paging/OOM or compilation, and normal sampled memory pressure throughout. Whole-case swap-in, including untimed checks, was 33,296 KiB. Both paths used the same native macOS release binary, 2 GiB query pool/q4, MinIO 2 CPU/1 GiB, Docker VM 2 GiB/swap disabled, and identical allocator decay settings.

Performance acceptance is complete under the single-pair scope agreed on 2026-10-12: D improved full export by 120.3x and full restore by 230.7x, exceeding the 3x target for each. These are single-pair ratios; repeatability, timing spread and three-round medians were not established. The two remaining planned pairs are canceled. Earlier invalid observations remain excluded. See #9120 for the accepted scope and remaining documentation work. The feature remains opt-in behind the experimental gate; removing the gate is outside this acceptance.

The supplementary representative-shape comparison is complete: 10 physical tables × 10,000 logical tables, 300 user columns per physical table (298 possible string tags, timestamp and DOUBLE), and 10 overlapping tags plus timestamp and DOUBLE per logical table. Each logical table has 32 rows (3.2 million total); S3 adds +100 ms per HTTP request. One D-then-A pair used the same fixed release binary, frozen source, equal limits and fresh destinations.

Supplementary measurement Ordinary (A) Packed + batch DDL (D)
Full export 3416.710 s 38.164 s
Full restore 6745.171 s 37.474 s
Export S3 requests 100,115 77
Restore S3 requests 600,204 63
Export host swap-in 15,872 KiB 1,216 KiB
Restore host swap-in 59,824 KiB 128 KiB
Restore host swap-out 18,304 KiB 0 KiB

Both paths passed full typed-value/DDL, membership, remapped-ID, import-state and procedure checks for all 100,000 tables. Logical-create procedures fell from 100,000 to 790. The L0 scan fix resolved the 2 GiB query-pool exhaustion: D completed with a sampled export query-pool peak of 229.2 MiB and no observed rejection. Writes/parts and D range reads stayed within 8 MiB.

This supplementary pair is not a strictly resource-valid performance sample: A restore had 17.875 MiB host swap-out. The observed export/restore ratios are 89.5x/180.0x; the restore ratio is qualified by that interference. D had zero swap-out, including full correctness validation, and both export phases had zero swap-out. All 1,313 whole-case pressure samples were normal; there was no guest paging, MinIO swap/OOM or concurrent compilation. Whole-case swap-in was 83,888 KiB. This does not establish repeatability or replace the earlier accepted narrow-fixture pair.

The database used a 2 GiB query pool/q4 and a 256 MiB page cache; MinIO remained at 2 CPU/1 GiB and the Docker VM at 2 GiB with swap disabled. Limits were unchanged between controls. Raw evidence is retained outside the repository. The experimental switch remains opt-in.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.
  • This PR needs to be backported to release branches, and I have added the backport-<target> labels (e.g. backport-v1.3 targets release/v1.3).

@github-actions github-actions Bot added size/M documentation docs-not-required This change does not impact docs. labels Oct 10, 2026
@fengjiachun

Copy link
Copy Markdown
Collaborator Author

/ci

@github-actions

Copy link
Copy Markdown
Contributor

Requested rerun of checks.yml, rust.yml, integration.yml, docs.yml for 57ae8a534e8224667c7841a760bbaff52faa49f0.

@fengjiachun

Copy link
Copy Markdown
Collaborator Author

@codex review

When you finish, append exactly this marker to the review summary:

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T02:35:26.834974Z 24377f9 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 57ae8a534e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@fengjiachun fengjiachun changed the title fix: avoid redundant metric scan splitting and bound snapshot writes fix: bound metric snapshot memory and reduce export overhead Oct 10, 2026
Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.

Signed-off-by: jeremyhi <[email protected]>
Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.

Signed-off-by: jeremyhi <[email protected]>
Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.

Signed-off-by: jeremyhi <[email protected]>
@fengjiachun
fengjiachun force-pushed the test/metric-snapshot-acceptance branch from c7b6a74 to 07d8d91 Compare October 11, 2026 07:43
@fengjiachun

Copy link
Copy Markdown
Collaborator Author

@codex review

@fengjiachun
fengjiachun marked this pull request as ready for review October 11, 2026 07:46
@fengjiachun
fengjiachun requested review from a team, evenyag and v0y4g3r as code owners October 11, 2026 07:46

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07d8d9192d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/mito2/src/read/scan_util.rs Resolved
@fengjiachun

Copy link
Copy Markdown
Collaborator Author

@codex review

When you finish, append exactly this marker to the review summary:

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 88069669aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.

Cover short-series overlapping L0 files with an ordered latest-value regression.

Signed-off-by: jeremyhi <[email protected]>
@fengjiachun

Copy link
Copy Markdown
Collaborator Author

@codex review

When you finish, append exactly this marker to the review summary:

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 24377f9008

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WenyXu WenyXu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src/operator/src/statement/export_logical_tables.rs:609

nit: The inline and blocking paths repeat the slice conversion, backing-size check, and Payload construction. Could we share that final step through a small private synchronous helper? Budget acquisition and the decision to use spawn_blocking_global should remain at the call sites so the fast path still avoids worker dispatch.

src/mito2/src/read/scan_util.rs:1210 — follow-up

nit: For a follow-up cleanup, could we consider renaming this to estimate_split_batch_rows? The should_* prefix suggests a boolean, but this function also returns the estimated row count after splitting. Documenting None as “splitting is not recommended” would make that combined contract clearer. This naming predates the PR, so there is no need to expand this change just for the rename.

src/operator/src/statement/export_logical_tables.rs:885 — follow-up

nit: These batch-conversion helpers are now shared by logical-table export and ordinary COPY. In a follow-up, could we move them into a focused operator-local module such as export_batch? That would reflect their shared responsibility without introducing a generic utils module or moving operator-specific error and budget semantics into a lower-level crate.

PR description — resource-limit wording

nit: Could we make the resource-limit wording explicit? The 8 MiB limit bounds write submissions/parts, while metadata serialization still retains the complete Vec<u8>. Likewise, the Parquet byte threshold triggers a flush after encoding; it is not a strict peak-memory cap. Keeping these distinctions visible would help readers interpret the memory guarantees correctly.

PR description — performance evidence

nit: The new tests demonstrate fewer batches/row groups and reduced retained dictionary backing, but they do not isolate the overall CPU or peak-memory effect of these changes. Arrow dictionary GC also allocates a remapping buffer proportional to the original dictionary length. Could we retain a reproducible reference to the controlled before/after measurements and distinguish end-to-end packed-export results from evidence for the individual optimizations?

Comment thread src/cli/src/data/snapshot_storage.rs Resolved
Comment thread src/cli/src/data/snapshot_storage.rs Resolved
Comment thread src/cli/src/data/snapshot_storage.rs Resolved
Comment thread src/cli/src/data/snapshot_storage.rs Resolved
@fengjiachun

fengjiachun commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Addressed in 9a56b8f. Extracted convert_slice_payload and clarified the memory bounds and benchmark limitations in the PR description. The scan-helper rename and module move remain follow-ups.

56 focused tests passed on macOS; formatting passed. Windows and the scale matrix were not rerun.

@github-actions github-actions Bot added size/L and removed size/M labels Oct 12, 2026
@fengjiachun
fengjiachun enabled auto-merge October 12, 2026 05:46
@fengjiachun
fengjiachun added this pull request to the merge queue Oct 12, 2026
Merged via the queue into GreptimeTeam:main with commit 18726c7 Oct 12, 2026
57 of 58 checks passed
@fengjiachun
fengjiachun deleted the test/metric-snapshot-acceptance branch October 12, 2026 06:50
WenyXu added a commit to WenyXu/greptimedb that referenced this pull request Oct 27, 2026
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.

Signed-off-by: WenyXu <[email protected]>
WenyXu added a commit to WenyXu/greptimedb that referenced this pull request Oct 27, 2026
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.

Signed-off-by: WenyXu <[email protected]>
WenyXu added a commit to WenyXu/greptimedb that referenced this pull request Oct 27, 2026
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.

Signed-off-by: WenyXu <[email protected]>
WenyXu added a commit to WenyXu/greptimedb that referenced this pull request Oct 28, 2026
Copy the writer observation test utility from GreptimeTeam#9505 (18726c7) for the GreptimeTeam#9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.

Signed-off-by: WenyXu <[email protected]>
WenyXu added a commit that referenced this pull request Oct 28, 2026
* fix(hdfs): clean up temporary file when an atomic write fails (#9483)

* fix(hdfs): clean up temporary file when an atomic write fails

Signed-off-by: fengyue-xve <[email protected]>

* style: apply rustfmt to hdfs layer test

Signed-off-by: fengyue-xve <[email protected]>

* fix(hdfs): report earlier write failures and log cleanup failures

- Later `write`/`close` calls on a writer whose failed write was
  cleaned up now mention the original failure instead of the bare
  "unavailable after close or abort" message.
- `abort_and_delete` logs abort and deletion failures with the
  temporary path instead of discarding them; the original write error
  is still returned to the caller.
- The temporary-file cleanup test now lists the directory holding the
  temporary file, so it fails if the cleanup is reverted.

Signed-off-by: 风月 <[email protected]>
Signed-off-by: fengyue-xve <[email protected]>

---------

Signed-off-by: fengyue-xve <[email protected]>
Signed-off-by: 风月 <[email protected]>
(cherry picked from commit a35661d)
Signed-off-by: WenyXu <[email protected]>

* fix(jaeger): return the most recent traces when a search hits its limit (#9595)

* fix(jaeger): return the most recent traces when a search hits its limit

Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search
matching more traces than the limit returned an arbitrary subset that
changed between identical requests. Order traces by their latest matching
span before applying the limit.

Signed-off-by: Dennis Zhuang <[email protected]>

* refactor(jaeger): name the trace dataframe builder with a verb and document execution

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit cf16c73)
Signed-off-by: WenyXu <[email protected]>

* fix(pprof): initialize framehop before signal sampling (#9612)

fix(pprof): pin upstream framehop initialization fix

Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes.

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit 0dc8177)
Signed-off-by: WenyXu <[email protected]>

* chore: bump rustls to 0.23.46 (#9614)

rustls 0.23.46 fixes rustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout.

(cherry picked from commit 33879b0)
Signed-off-by: WenyXu <[email protected]>

* fix(prom): return empty remote read results for missing metrics (#9613)

* fix: return empty remote read results for missing metrics

Signed-off-by: fys <[email protected]>

* test(prom): cover empty reads for missing databases

Signed-off-by: fys <[email protected]>

* test(prom): simplify remote read regression cases

Signed-off-by: fys <[email protected]>

* fix(prom): log missing tables during remote read

Signed-off-by: fys <[email protected]>

---------

Signed-off-by: fys <[email protected]>
(cherry picked from commit cf0f6c3)
Signed-off-by: WenyXu <[email protected]>

* fix(ci): compare release validation against the previous stable version (#9636)

Ancestry-based tag selection skips stable tags that live only on older release branches.

Signed-off-by: paomian <[email protected]>
(cherry picked from commit 3de8610)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): keep processing bulk requests after one fails to encode (#9638)

* fix(mito2): keep processing bulk requests after one fails to encode

When `push_bulk` fails to build the WAL entry of a bulk request, it answers
that request with the error. `process_bulk_requests` then returned from the
loop, so the remaining requests of the drained batch were dropped without an
answer and their callers only saw a receive error. Skip the failed request
and continue with the rest of the batch instead.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): keep bulk install barriers independent per target

The bulk install test barrier was a single process-wide slot, so tests
that arm it while running in the same process replaced each other's
barrier. Keep one barrier per armed target instead.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 88bdab0)
Signed-off-by: WenyXu <[email protected]>

* fix(promql): honor @ modifier on subqueries (#9446)

* fix(promql): honor @ modifier on subqueries

`prom_subquery_expr_to_plan` never read the `at` field of a subquery, so
`sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and
followed the outer evaluation grid.

Handle `@` the way Prometheus does: fold the anchor into the subquery
offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier`
does), evaluate the subquery once at the start of the evaluation, and
replay the folded window over the outer grid, reusing the anchored
range selector machinery. A call whose range argument is an anchored
subquery is promoted as step-invariant, like one over an anchored range
selector. `@ start()`/`@ end()` and `offset` combine as for selectors,
and an unrepresentable anchor is rejected with the existing error.

Closes #9411

Signed-off-by: Aarav <[email protected]>

* test(promql): add sqlness cases for @ on subqueries

Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`,
`@` combined with `offset` in either order, an instant query away from
the anchor, `*_over_time` functions and `rate` over an anchored
subquery, a computed inner expression, and an aggregation above the
anchored call. Two series per table so a dropped or merged series
shows up as a wrong row.

Refs #9411

Signed-off-by: Aarav <[email protected]>

---------

Signed-off-by: Aarav <[email protected]>
(cherry picked from commit 095c288)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): validate bulk insert batches against the region schema (#9698)

* fix(mito2): validate bulk insert batches against the region schema

Bulk inserts were only checked for the time index column, unlike row
writes, which WriteRequest::check_schema checks against the region
schema. A batch with columns of another type, nulls in the time index or
in non-nullable columns, duplicate or unknown columns reached the region
worker: a time index that is not a timestamp or holds only nulls made
the worker fail, and other mismatches failed after the WAL append or
were stored as written, depending on the SST format.

Move the column checks of check_schema into a core shared by row writes
and bulk inserts, and check bulk insert batches at the same points as
rows: when the engine builds the worker request, and again in the worker
if the region schema changed since. check_schema keeps its behavior.
Bulk insert columns must have the column's Arrow type, except that tags
may use any string encoding, as the memtables already convert them.
Sparse batches must carry one encoded primary key column of binary
without nulls. A batch that misses columns gets them filled by the
worker even if it claims to match the current schema.

The metric engine checks logical bulk inserts against the logical region
before rewriting them into the physical region, as the rewrite copies
columns without checking them and would encode a duplicated tag into the
primary key twice.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): report an error when a write is interrupted by a panic

WriteNotify answers its waiter when it is dropped and reported success
unless an error had been set, also when it was dropped while unwinding
from a panic in the middle of the write. Report an error in that case.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): tolerate a poisoned series lock when dropping a series map

SeriesMap::drop reads every series to update the memtable metrics and
unwrapped the lock, so dropping the map while unwinding from a panic
that poisoned a series lock panicked again. Read through the poisoned
lock instead.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 791f429)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): fence writes before a graceful downgrade (#9716)

* test(mito2): cover graceful role changes racing with writes

A graceful role change flips the region state in a background task while
the worker keeps handling writes. Add bulk and row write tests that check a
write handled after the role change request is either rejected or covered
by the returned last_entry_id.

Add tests for DowngradingLeader while an ALTER keeps the region in
Altering, alone and in the downgrade sequence the datanode runs
(DowngradingLeader, Flush, Follower) with a migration candidate on its own
local WAL. Add a test that the downgrade sequence covers a write in flight
when a lease renewal sets Downgrading.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): fence writes before a graceful downgrade

set_region_role_state_gracefully hands the role change to a background task
that waits for the manifest lock. The worker keeps handling writes until the
task flips the state, so a write could be acknowledged with an entry id above
the returned last_entry_id. Set Downgrading in the worker before spawning the
task for Follower and DowngradingLeader requests, as lease renewal already
does, so every write the worker handles afterwards is rejected.

DowngradingLeader also did nothing and returned success when the region was
in another leader state such as Altering. The region became writable again
when the operation finished, and the datanode then flushed and turned it
into a follower in one step. Return a retryable error instead, so metasrv
retries the downgrade.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): assert the retryable downgrade error during ALTER

Check that DowngradingLeader during an ALTER returns a retryable error, and
release the paused ALTER on that path. The local WAL test now retries the
downgrade after the ALTER, as metasrv does.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): write before the retried downgrade in the local WAL test

Rows written before an ALTER were already flushed, so the candidate check
did not depend on the downgrade's final flush. Write rows after the ALTER
and before the downgrade is retried, which only that flush can carry to the
candidate.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 660c4ef)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): validate persisted partition bounds before altering regions (#9487)

* fix(meta): validate persisted partition bounds before altering regions

A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.

Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.

Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.

Fixes #9454

Signed-off-by: fengyue-xve <[email protected]>

* fix: access ColumnSchema fields directly and use plain format args

ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.

Signed-off-by: fengyue-xve <[email protected]>

* test: access ColumnSchema data_type field in alter table tests

Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.

Signed-off-by: fengyue-xve <[email protected]>

* test: drop borrows now that data_type is read as a field

Signed-off-by: fengyue-xve <[email protected]>

* test: drop useless format! in nested expression test

Signed-off-by: fengyue-xve <[email protected]>

* test: cover the partition tail in change_col_type sqlness case

Signed-off-by: fengyue-xve <[email protected]>

* test: drop spurious blank line in change_col_type result

Signed-off-by: fengyue-xve <[email protected]>

* fix(meta): compare partition bounds with the scan comparison coercion

The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.

Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.

Signed-off-by: fengyue-xve <[email protected]>

* test: update the sqlness fixture for the new incompatibility message

The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.

Signed-off-by: fengyue-xve <[email protected]>

* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency

common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.

Signed-off-by: fengyue-xve <[email protected]>

* fix(meta): reject every non-widening type change of a partitioned column

The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.

Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.

Signed-off-by: fengyue-xve <[email protected]>

---------

Signed-off-by: fengyue-xve <[email protected]>
(cherry picked from commit 1e29e8d)
Signed-off-by: WenyXu <[email protected]>

* fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)

* fix(partition): route NULLs like row inserts and accept signed-minimum bounds

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(partition): keep equality two-valued for NULL rows and accept unsigned -0

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(partition): treat NULL masks as false in batch routing and keep comparisons bare

Signed-off-by: Dennis Zhuang <[email protected]>

* test(compat): update partition filter in analyze verbose expected output

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit da88ff8)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): re-check a filled write as sent after a schema change (#9701)

* fix(mito2): re-check a filled write as sent after a schema change

The engine fills the columns a write leaves out with their defaults when it
accepts the write, appending them to the rows without adding them to
`name_to_index` and `has_null`. If the region schema changes before the
worker handles the write, the worker checks the request again and
`check_schema` looks up a filled column in `name_to_index`, which panics and
stops the region worker. If the filled column was dropped, the write fails
with an unknown column that the client never sent.

Remove the filled columns before checking, so the request is checked as sent
and filled again for the current schema. Nothing is filled yet when the
engine checks the request, so that check does no extra work.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): rename put_request helper to build_put_request

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 93c3b11)
Signed-off-by: WenyXu <[email protected]>

* perf(mito2): gather last rows with one take per input batch (#9737)

* perf(mito2): gather last rows with one take per input batch

FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (#9034) slower. Select the rows of
series that end inside a batch with one take instead.

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): cover duplicate last timestamps in the take path

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): group last rows by primary key bytes

Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 4645ab9)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): apply SST internal column encodings to bulk-encoded parts (#9735)

* fix(mito2): apply SST internal column encodings to bulk-encoded parts

Bulk memtable parts are encoded to parquet and flushed as SSTs as-is, but the
bulk encoder did not apply the time index / __sequence DELTA_BINARY_PACKED
encoding (with dictionary disabled) and the uncompressed op_type setting that
the regular SST writer uses. Share these settings through one helper.

Signed-off-by: Dennis Zhuang <[email protected]>

* chore(mito2): drop the answered bulk encoder compression TODO

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 977e76d)
Signed-off-by: WenyXu <[email protected]>

* fix: avoid exact scan row counts for SSTs with unknown counts (#9759)

fix: avoid exact scan row counts for legacy SSTs

Treat selected SSTs with unknown row counts as non-exact before checking region partition visibility. Preserve exact statistics for known counts and cover legacy metadata mixed with known-count files.

Fixes #9718

Signed-off-by: Aumnertic <[email protected]>
(cherry picked from commit 6f53d57)
Signed-off-by: WenyXu <[email protected]>

* fix: avoid duplicate keys in update flow metadata txn (#9816)

CREATE OR REPLACE FLOW deleted and then put the same flow route,
flownode flow and table flow keys in a single txn when the replacement
kept the partitions, flownodes or source tables, which etcd rejects
with "duplicate key given in txn request".

Only delete the old keys that are not put again. Closes #9810.

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit a1b2362)
Signed-off-by: WenyXu <[email protected]>

* fix(ci): increase release integration test stack size (#9817)

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit e6fa1a0)
Signed-off-by: WenyXu <[email protected]>

* fix(datanode): release region map guard before awaiting role changes (#9824)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 0dc649e)
Signed-off-by: WenyXu <[email protected]>

* fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860)

* fix(ci): increase Windows nightly test stack size

Signed-off-by: evenyag <[email protected]>

* fix(ci): align optional macOS artifact test stack size

Signed-off-by: evenyag <[email protected]>

---------

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit 8b1d683)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)

* fix(mito2): keep SSTs when a manifest save ends with RateLimited

The object store's retry layer retries temporary errors and returns the
last attempt's error. A manifest delta PUT can be stored by the server
while the client sees a timeout, reset or 5xx, and the retries that
follow can be throttled, so the save ends with RateLimited even though
the delta exists. may_have_persisted_manifest_update treated RateLimited
as proof that nothing was written, so flush and compaction deleted the
SSTs the stored delta references. A later open or follower sync replays
the delta and reads fail on the missing files.

Treat RateLimited like other ambiguous errors: the SSTs are kept and
become orphans when the save really failed.

Fixes #9897

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): keep SSTs when a manifest save ends with PermissionDenied

The retry layer returns the last attempt's error, so a 403 from a revoked
key, a changed bucket policy or clock skew can follow a delta PUT that
the server already stored. Like RateLimited, PermissionDenied does not
prove that nothing was written, so it no longer lets flush or compaction
delete the SSTs; they become orphans when the save really failed.

The compaction test that rejects a manifest write before persistence now
injects Unsupported, which is raised before any request is sent.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
Co-authored-by: Yingwen <[email protected]>
(cherry picked from commit 1cb01d0)
Signed-off-by: WenyXu <[email protected]>

* fix: upload staging manifests in chunks (#9919)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit f2be11a)
Signed-off-by: WenyXu <[email protected]>

* perf: stream and prune SST manifest scans (#9931)

* refactor: add reusable metadata pruning and projected SST batches

Signed-off-by: WenyXu <[email protected]>

* perf(mito2): stream and prune manifest SST metadata

Signed-off-by: WenyXu <[email protected]>

* perf(datanode): stream inspection queries through reusable providers

Signed-off-by: WenyXu <[email protected]>

* perf(catalog): push manifest filters through inspection queries

Signed-off-by: WenyXu <[email protected]>

* refactor: align manifest column constants with module conventions

Signed-off-by: WenyXu <[email protected]>

* perf: reduce manifest inspection allocations

Signed-off-by: WenyXu <[email protected]>

* refactor: narrow manifest inspection API visibility

Signed-off-by: WenyXu <[email protected]>

* refactor: retain residual filters for manifest inspection

Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit c96cdd0)
Signed-off-by: WenyXu <[email protected]>

* feat: improve SQL analyze scan diagnostics (#9565)

* feat: improve SQL analyze scan diagnostics

Signed-off-by: evenyag <[email protected]>

* test: redact dynamic filters in sqlness results

Signed-off-by: evenyag <[email protected]>

* test: stabilize analyze snapshots for merge queue

Signed-off-by: evenyag <[email protected]>

* test: remove obsolete scan_details assertion

Signed-off-by: evenyag <[email protected]>

* test: refresh explain snapshots after partition filter changes

Signed-off-by: evenyag <[email protected]>

---------

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit 3e2751d)
Signed-off-by: WenyXu <[email protected]>

* fix(query): avoid lossy casts in partition pruning (#9600)

* fix(query): keep lossy casts out of partition pruning

Signed-off-by: discord9 <[email protected]>

* test(query): align cast pruning expectations with conservative fallback

Signed-off-by: discord9 <[email protected]>

* test(query): record lossy cast partition pruning regression

Signed-off-by: discord9 <[email protected]>

* fix(query): retain safe pruning conjuncts beside unsupported casts

Signed-off-by: discord9 <[email protected]>

* test(query): refresh cast pruning plans after rebase

Signed-off-by: discord9 <[email protected]>

---------

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit df8f684)
Signed-off-by: WenyXu <[email protected]>

* fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)

* fix(query): skip LastRow pushdown for IGNORE NULLS

Signed-off-by: discord9 <[email protected]>

* fix(query): keep IGNORE NULLS aggregates above remote scans

Signed-off-by: discord9 <[email protected]>

* test(query): record IGNORE NULLS results before and after flush

Signed-off-by: discord9 <[email protected]>

* test(query): share IGNORE NULLS coverage across deployments

Signed-off-by: discord9 <[email protected]>

* test(query): refresh last-value plans after rebase

Signed-off-by: discord9 <[email protected]>

---------

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit 0dab3ae)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): share peer discovery with plugin hooks (#9977)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 9f0e58e)
Signed-off-by: WenyXu <[email protected]>

* fix: adapt release backport test dependencies

Copy the writer observation test utility from #9505 (18726c7) for the #9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.

Signed-off-by: WenyXu <[email protected]>

* chore: bump version to 1.3.0

Signed-off-by: WenyXu <[email protected]>

* fix(query): pin DataFusion cast constraint propagation fix

Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).

CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.

The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.

Cargo.lock only rewrites the 35 datafusion source lines.

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit fe637b0)
Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: fengyue-xve <[email protected]>
Signed-off-by: 风月 <[email protected]>
Signed-off-by: WenyXu <[email protected]>
Signed-off-by: Dennis Zhuang <[email protected]>
Signed-off-by: Lei, HUANG <[email protected]>
Signed-off-by: fys <[email protected]>
Signed-off-by: paomian <[email protected]>
Signed-off-by: jeremyhi <[email protected]>
Signed-off-by: Aarav <[email protected]>
Signed-off-by: Aumnertic <[email protected]>
Signed-off-by: evenyag <[email protected]>
Signed-off-by: discord9 <[email protected]>
Signed-off-by: discord9 <[email protected]>
Co-authored-by: fengyue-xve <[email protected]>
Co-authored-by: dennis zhuang <[email protected]>
Co-authored-by: Lei, HUANG <[email protected]>
Co-authored-by: fys <[email protected]>
Co-authored-by: localhost <[email protected]>
Co-authored-by: jeremyhi <[email protected]>
Co-authored-by: DeviousCardi <[email protected]>
Co-authored-by: Aum <[email protected]>
Co-authored-by: Yingwen <[email protected]>
Co-authored-by: discord9 <[email protected]>
WenyXu added a commit that referenced this pull request Nov 13, 2026
* fix: bound metric snapshot memory and reduce export overhead (#9505)

* fix(mito): bound retained scan memory

Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.

Signed-off-by: jeremyhi <[email protected]>

* perf(snapshot): bound writes and reduce logical export overhead

Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.

Signed-off-by: jeremyhi <[email protected]>

* docs: clarify metric snapshot acceptance criteria

Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito): split batches only for merged readers

Signed-off-by: jeremyhi <[email protected]>

* test(snapshot): cover supported types in both layouts

Signed-off-by: jeremyhi <[email protected]>

* docs: consolidate metric snapshot experiment conclusions

Signed-off-by: jeremyhi <[email protected]>

* fix(mito): honor series statistics when splitting L0 scans

Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.

Cover short-series overlapping L0 files with an ordered latest-value regression.

Signed-off-by: jeremyhi <[email protected]>

* docs(mito): explain memory risk of splitting short L0 series

Signed-off-by: jeremyhi <[email protected]>

* refactor: address snapshot export review feedback

Signed-off-by: jeremyhi <[email protected]>

* fix(cli): accept source errors in lazy snapshot contexts

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 18726c760257047af4029692ce4ce147fdc3d35c)
Signed-off-by: WenyXu <[email protected]>

* fix(prom): align remote read label matcher semantics (#9633)

* fix(prom): align remote read label matcher semantics

Signed-off-by: fys <[email protected]>

* chore: optimize comment of validate_matchers

* fix(prom): preserve existing binary label regex handling

Signed-off-by: fys <[email protected]>

* refactor: condition

* test(prom): simplify remote read matcher tests and clarify comments

* fix(prom): return binary remote read labels as UTF-8

* fix(prom): reject binary labels in remote read

* refactor(prom): align remote read null label handling with PromQL

Signed-off-by: fys <[email protected]>

* refactor(prom): simplify null handling in label matcher branches

Signed-off-by: fys <[email protected]>

* fix(prom): restrict remote read labels to string types

Signed-off-by: fys <[email protected]>

* test(prom): use string label in default value column test

Signed-off-by: fys <[email protected]>

* fix: code review

---------

Signed-off-by: fys <[email protected]>
(cherry picked from commit 4e8bc6df86bbb1aeb6636bec0bebfcf5d5b49cad)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): restore source splitting for cross-range series merges (#9818)

* fix: restore batched series merges across ranges

Signed-off-by: WenyXu <[email protected]>

* fix(mito2): split range sources before last-row selection

Signed-off-by: WenyXu <[email protected]>

* fix(mito2): avoid tiny splits for single-source cross-range merges

Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 27c531336a3b7dece7712c682c53e277e4626d28)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): pick TWCS L1 merges by closed units so compaction terminates (#9741)

* refactor(mito2): pick TWCS L1 merges by closed units

TWCS could keep rewriting L1 files without new writes. Overlap-only picks
could split their inputs into one more file than they read, and L1 file
reduction then rewrote up to 16 files to remove that single file. The
outputs, split by key, created new overlaps and the cycle repeated.

Replace the three global pick phases with one pick per window, scheduled by
class across windows. Every pick now decreases a termination measure when the
window receives no writes.

- Active windows only take merges that remove files. L1 files are grouped by
  conflicts and merged in key order, so size-split tails are merged together.
  A reduction rewrites at most 4 output files' worth of bytes per removed file.
- Inactive windows absorb L0 files, then take closed merges: no other file
  intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each
  merge resolves conflicts or removes files, decreasing (conflicts, files).
  Files that do not take part in an overlap are not rewritten.
- Overlaps that only share a boundary key no longer count as conflicts, so a
  size-split sorted run does not look overlapping.
- Closed merges may take 4x the input limit to resolve a whole window once.
- Cache primary key ranges per pick instead of resolving them per comparison.
- Log the pick branch of each compaction output.

Signed-off-by: Lei, HUANG <[email protected]>

* docs(mito2): state the compactor assumptions behind TWCS termination

Signed-off-by: Lei, HUANG <[email protected]>

* feat(mito2): resolve active window overlaps on manual compaction

A window that stops receiving writes keeps being the active window, whose
picks only remove files, so its overlaps were never resolved. Manual
compactions now pick every window as inactive, so a closed merge can resolve
them. Automatic compactions are unchanged.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): defer overlap resolution in the window before the active one

Late data keeps landing in the window right before the active window. A
closed merge that resolves its overlaps early rewrites the whole window,
then the late data brings the overlaps back, and the balanced run check
keeps them there. Overlap-only merges now skip that window until it is
older. File-reducing merges and manual compactions are unchanged.

On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead
of 2.08 at the same write amplification (3.01 vs 3.02).

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito2): only check earlier-starting files when closing units

Closing units only checks files that start before a unit ends, and the
overlap checks compare key bounds without cloning them.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): keep settling windows from rewriting L1 files

The window before the active one still receives late data. Skipping only
its overlap-only merges was not enough: file-reducing merges of
conflicting files and mixed L0/L1 merges still rewrote its L1 files,
and a mixed merge could rewrite a nearly full output for each small late
file. A settling window now only merges L0 files among themselves and
conflict-free files; the window resolves its overlaps once it is older.

On the out-of-order bench, write amplification drops from 3.88 to 3.30
at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with
overlap depth 1.0 in all three.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): resolve late overlaps once a window is quiet

Late data that lands in a window older than the settling one is merged
into a small file covering the whole key range. Resolving it rewrites
the window's dominant sorted run, which the balanced run check refuses,
so the overlap stayed. Removing the check instead rewrites the window
for every late flush.

A window is quiet once the region has written at least as many rows as
the window holds since its newest file. Overlap merges in a quiet window
may rewrite a dominant run, so each quiet period pays for at most one
such rewrite. Sequences are per row and kept by compaction, so this
needs no state.

On the bench with late data up to 2 hours, mean overlap depth drops
from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write
amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of
2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The
remaining depth is in the final active window.

Signed-off-by: Lei, HUANG <[email protected]>

* test(mito2): check closed merges of quiet and settling windows terminate

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): split TWCS closures over the input limit

A file spanning the keys of a large sorted run joins every run file into one
closure. Once a closure held more files than one compaction accepts, closed
merges skipped it, so the window never merged those files again and late
files kept piling up.

Split such closures into single files, like active-window picks do, and
replace the closed check with a conflict bound: outputs only conflict with
files intruding on the inputs' box, so a merge is taken if `outputs · I`
does not exceed the conflicts its inputs had with other files. Intervals of
whole closures have no such conflicts and still need to be closed, so other
windows pick as before, and `(conflicts, files)` still strictly decreases.

Small run files and late files over a large run can be merged again. A late
file over a run that cannot shrink stays: resolving it would rewrite the
whole run.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): address TWCS picker review findings

- Manual compactions may rewrite a dominant sorted run to resolve overlaps.
  The active window is never quiet, so manual compactions could not resolve
  its late-data overlaps before.
- Single-key intervals predict one output: the writer never splits a series,
  so large files of one series were never merged.
- The active window falls back to single files when no group of conflicting
  files qualifies, so a large file no longer keeps its small peers unmerged.
- Compute the settling window distance with checked_sub, which overflowed for
  windows at the timestamp limits.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): count inner keys as TWCS conflicts and check balance per group

- A single key strictly inside another file's key range now conflicts, so
  a late file of one series inside a wide file is resolved like other
  overlaps. Files sharing only a boundary key still do not conflict.
  Intrusion covers the new pairs, which a randomized test checks.
- Check the dominant-run guard per group of conflicting files. Runs of
  separate groups never overlap, so pairing one group's run with another
  group's peer made an interval of two dominant runs look balanced.
- A window with unknown row counts, written before they were recorded, is
  not quiet: its threshold would drop to a single row.

Signed-off-by: Lei, HUANG <[email protected]>

* perf(mito2): index TWCS files by key to avoid quadratic grouping

Grouping a window's files compared every file with all files sharing its
time range, and closing groups and checking intrusions scanned the whole
window. A sorted run split by key shares one time range, so a window of N
such files took O(N²) checks without any conflict. The 1024-file fallback
only covers the sorted-run path, not these groups.

Index the files by lower primary key in a segment tree that keeps each
subtree's largest upper key and its time bounds. Grouping, closing, and the
conflict bound only check the files whose key and time ranges meet, so a
file of a key-sliced run only meets its neighbors. Conflicts among split
files are precomputed per file and counted per interval with a binary
search instead of pairwise.

On a run of 20,000 key-sliced files with 5 late files spanning all keys,
a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick
~19 ms instead of ~3.5 s.

Signed-off-by: Lei, HUANG <[email protected]>

* test(object-store): retry transient S3 transport failures

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): bound TWCS closed-merge conflict graph work

Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph.

The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): charge TWCS index nodes to the conflict-graph budget

The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted.

Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): retry smaller TWCS dense compaction candidates

If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): compact shared primary-key boundaries

Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint.

Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): compact small L1 files in inactive windows

Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks.

Signed-off-by: Lei, HUANG <[email protected]>

* docs(mito2): clarify TWCS unit grouping guarantees

Signed-off-by: Lei, HUANG <[email protected]>

---------

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit bb3f07b4825e074c207ae29db28de1e3354c11ae)
Signed-off-by: WenyXu <[email protected]>

* fix: reclaim HDFS temporary files left behind by a datanode crash (#9616)

* fix: reclaim HDFS temporary files left behind by a datanode crash

Signed-off-by: lczllx <[email protected]>

* fix(mito2): stop the temporary file sweep from starving GC jobs

Signed-off-by: lczllx <[email protected]>

* feat(mito2): add metrics for the temporary file sweep

Signed-off-by: lczllx <[email protected]>

* refactor(object-store): share one definition of the temporary file name

Signed-off-by: lczllx <[email protected]>

* fix(mito2): skip the temporary file sweep when no GC job is allowed

Signed-off-by: lczllx <[email protected]>

* refactor: gate the temporary file sweep behind the hdfs-object-store feature

Signed-off-by: lczllx <[email protected]>

* fix(mito2): sweep temporary files when a create opens an existing region

Signed-off-by: lczllx <[email protected]>

* docs(config): document the temporary file sweep in the GC settings

Signed-off-by: lczllx <[email protected]>

* refactor: align the temporary file sweep code with the surrounding style

Signed-off-by: lczllx <[email protected]>

---------

Signed-off-by: lczllx <[email protected]>
(cherry picked from commit 88cb2a986ef0776a1eb84b0df9581b3e7ffb189d)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): fence datanode processes replaced by a newer process (#9878)

* fix(meta): fence datanode processes replaced by a newer process

Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.

The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.

Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): keep the leader cache load and pusher updates coherent

A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.

A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.

The epoch check moves into the heartbeat service.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): keep the leader cache coherent with writes while it is disabled

A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.

Signed-off-by: jeremyhi <[email protected]>

* test(meta): pin the epoch 0 exemption and pusher order, and update docs

The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): register a datanode pusher only after its first request passes

A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.

Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): check a datanode epoch again before replying

A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): handle a datanode's requests one at a time

A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.

The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 6f5c064f543eedfa65d312bebe64f33870c3ca60)
Signed-off-by: WenyXu <[email protected]>

* fix: acknowledge writes only within the region's leader lease (#9904)

* fix: acknowledge writes only within the region's leader lease

Signed-off-by: jeremyhi <[email protected]>

* fix: drop cached logical columns after a failed logical drop

Signed-off-by: jeremyhi <[email protected]>

* fix: keep the prepared region for the lease check after the WAL append

Signed-off-by: jeremyhi <[email protected]>

* fix: serialize a logical drop with fills of the logical column cache

Signed-off-by: jeremyhi <[email protected]>

* test: check the drop lock and grant long leases deterministically

Signed-off-by: jeremyhi <[email protected]>

* test: cover a bulk insert that loses its lease during the WAL append

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 91a59b82fbff9d1443968024dc5df1c96e661d4e)
Signed-off-by: WenyXu <[email protected]>

* fix(mito): preserve SST lifetime across index replacement (#9950)

* fix(mito): preserve SST lifetime across index replacement

Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.

Fixes #9944

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): list-based SST purge and single FilePurger::remove_file

The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.

- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
  runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
  is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
  is_delete) method; is_delete means the file was retired AND this is the
  last live view.
- LocalFilePurger deletes the SST and all its index files via the new
  delete_file_and_all_indexes helper (list before delete, so a listing
  failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
  callers in the entire repo.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito): elect final SST view release atomically

Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.

Add a deterministic concurrent-drop regression covering retired and live files.

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): restore Arc-owned SST cleanup and clarify reference release

Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.

Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): track SST references by file lifetime

Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.

Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.

Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): release SST references directly on lifetime drop

Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.

Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.

Signed-off-by: Lei, HUANG <[email protected]>

* test(mito): verify SST cleanup with real storage and isolate GC policy cases

Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.

Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.

Signed-off-by: Lei, HUANG <[email protected]>

* style(mito): format SST lifetime changes

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito): reclaim obsolete indexes by reader version

Track exact index-version references per metadata view and restore version-specific Puffin liveness in both GC paths. Keep file-level Parquet protection and shared SST lifetime cleanup, without expanding snapshots into unreferenced index generations.

Include old readers of imported files even when the source region is not open locally. Add real-storage GC coverage for reader release and lingering while retaining Parquet and the current index.

Validated workspace checks with default and all features, all-target all-feature Clippy with warnings denied, and all 1853 mito2 tests.

Signed-off-by: Lei, HUANG <[email protected]>

---------

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit 17ed988f49804af40dc79ff81f6c751b18be1dd1)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): coordinate range cache invalidation and singleflight reads (#9935)

* fix(mito2): invalidate retired SST range results and coalesce loads

Coordinate decoded range loads by their full scan key, bound shared work, and prevent late fills after SST retirement. Invalidate range results for local purge and object-store GC while preserving old snapshot reads.

Fixes #9934

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito): preserve SST lifetime across index replacement

Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.

Fixes #9944

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): list-based SST purge and single FilePurger::remove_file

The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.

- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
  runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
  is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
  is_delete) method; is_delete means the file was retired AND this is the
  last live view.
- LocalFilePurger deletes the SST and all its index files via the new
  delete_file_and_all_indexes helper (list before delete, so a listing
  failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
  callers in the entire repo.

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito): elect final SST view release atomically

Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.

Add a deterministic concurrent-drop regression covering retired and live files.

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): restore Arc-owned SST cleanup and clarify reference release

Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.

Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): track SST references by file lifetime

Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.

Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.

Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.

Signed-off-by: Lei, HUANG <[email protected]>

* refactor(mito): release SST references directly on lifetime drop

Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.

Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.

Signed-off-by: Lei, HUANG <[email protected]>

* test(mito): verify SST cleanup with real storage and isolate GC policy cases

Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.

Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.

Signed-off-by: Lei, HUANG <[email protected]>

* style(mito): format SST lifetime changes

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): use crate-rooted range cache imports

Signed-off-by: Lei, HUANG <[email protected]>

* fix(mito2): account for range cache delivery lifetime

Keep completed range results charged while watch subscribers or replay streams retain them. Cache entries keep weak delivery references, allowing cache-only ownership to release the delivery reservation. Cache hits reuse or nonblockingly acquire the reservation, falling back when the budget is exhausted.

Resize input reservations to completed-result estimates and cover successive slow consumers, cache-only handoff, hit sharing, invalidation, admission rejection, and cancellation with regression tests.

Signed-off-by: Lei, HUANG <[email protected]>

---------

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit 44df6e5dedfa6bddfa6c03ad4dd362b3564b38ce)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): exclude route followers from failover targets (#10001)

* fix(meta): exclude route followers from failover targets

Failover target selection did not exclude the region's route followers.
The selector could keep picking a follower, the submit rejected it as a
peer conflict, and the region never failed over.

Group the failed regions by their followers and exclude them per group.
A group without a candidate is skipped instead of failing every region of
the datanode.

Signed-off-by: jeremyhi <[email protected]>

* test: check failover continues after a group without candidates

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): keep failover going when follower routes can't be read

Fall back to selecting without follower exclusion when the routes can't
be read, log skipped groups with their followers at warn level, and test
a region without a route.

Signed-off-by: jeremyhi <[email protected]>

* refactor(meta): select failover targets per exclusion set in select_peers

generate_failover_tasks now only reads the route followers of the regions.
select_peers resolves the affinity and the unschedulable datanodes once,
then selects per group of regions that exclude the same datanodes.

Signed-off-by: jeremyhi <[email protected]>

* refactor(meta): drop a comment that repeats the code

Signed-off-by: jeremyhi <[email protected]>

* refactor(meta): tighten failover target selection

Name the route followers after the parameter they fill, read the routes
with unwrap_or_else, and log a group without a candidate as an error, as
the caller did before.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): let the region-stat-aware selector balance the whole batch

The region-stat-aware selector now receives each region's excluded peers
along with the whole batch, so it can still balance load across the
regions while keeping a region off its own followers. A region without a
candidate is left out of its output instead of failing the others. The
selectors driven by SelectorOptions keep selecting per group of regions
that exclude the same datanodes.

Signed-off-by: jeremyhi <[email protected]>

* test: name the batch type of the stat-aware recorder

Signed-off-by: jeremyhi <[email protected]>

* refactor(meta): treat a selector without candidates the same in both arms

Both selector arms now place what they can and treat NoEnoughAvailableNode
as nothing placed; the stat-aware selector's contract says so. Rename its
parameter to exclude_peer_ids_by_region, check for unplaced regions in
linear time, and describe what the code does in the comments.

Signed-off-by: jeremyhi <[email protected]>

* test: keep the stat-aware recorder's placements off excluded peers

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 62ebf71f90a6167966c1f3e64ed6d01dfc57b14e)
Signed-off-by: WenyXu <[email protected]>

* fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715)

* fix(log-store): make raft-engine periodic WAL sync actually fsync

On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and
write() returns before fsync (tikv/raft-engine#395). The periodic
sync task therefore never durable-flushed acknowledged WAL data.

Work around by writing a minimal non-empty marker with sync=true so
the fsync path is reached until a crates.io release includes
tikv/raft-engine#400.

Fixes #9712

Signed-off-by: Yvan Wang <[email protected]>

* perf(log-store): reuse LogBatch across periodic WAL syncs

Keep the periodic sync task's LogBatch across ticks instead of
allocating a fresh one (4 KiB encode buffer plus item vec) every
period. A successful Engine::write drains the batch but retains its
buffers; after a failed write or a panicked blocking task the batch is
discarded and recreated on the next tick, since it may be left
partially encoded.

Keep the marker put rather than deleting an empty key: the put
overwrites a single key, whereas every delete would become a recovery
tombstone that raft-engine keeps in memory while replaying append
files.

Also document that the marker value must not decode as a NamespaceImpl
(the marker key lies inside the range scanned by list_namespaces), add
a test covering batch reuse and namespace listing, and apply rustfmt.

Refs #9712

Signed-off-by: Yvan Wang <[email protected]>

---------

Signed-off-by: Yvan Wang <[email protected]>
(cherry picked from commit 180e2716fc5fb3f192a6f6cd57eac308892ccdcc)
Signed-off-by: WenyXu <[email protected]>

* fix(catalog): propagate frontend process-list failures (#9999)

* fix(catalog): propagate frontend process-list failures

Return the existing invocation error when a selected frontend cannot list processes, so a successful response cannot omit that peer.

Signed-off-by: Byte-Naut <[email protected]>

* test(frontend): cover remote process-list errors and aggregation

Exercise real metadata discovery and gRPC listeners for a missing management service and successful catalog-scoped aggregation.

Signed-off-by: Byte-Naut <[email protected]>

---------

Signed-off-by: Byte-Naut <[email protected]>
(cherry picked from commit 3d2b73762a6275a4bb44400e750fea4aeda87478)
Signed-off-by: WenyXu <[email protected]>

* fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)

* fix(log-store): skip records with unparsable keys instead of aborting the read

Signed-off-by: lczllx <[email protected]>

* fix(log-store): keep undecodable RecordMeta keys fatal

Signed-off-by: lczllx <[email protected]>

---------

Signed-off-by: lczllx <[email protected]>
(cherry picked from commit 5f0e22171ed54c4bb1245492648e2b5d04704b68)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): register failure detectors for regions switched before a rollback (#9989)

* fix(meta): register failure detectors for regions switched before a rollback

A region migration switches the routes of its tables one table at a
time. If it aborts after some tables were switched, those regions are
already served by the target, but the rollback deregistered the
target's failure detectors for every region and registered only the
source's. The target's next heartbeat registers them again, so if the
target stopped before sending one, nothing detected its failure and
the switched regions stayed unavailable until the metasrv leader
changed or the target came back.

The rollback now collects, from the routes it already reads, the
regions whose leader is the target and resets the target's failure
detectors for them at the end. It also skips a table whose route no
longer exists, so a table dropped meanwhile no longer fails the
rollback before the detectors are handled.

Signed-off-by: jeremyhi <[email protected]>

* test(meta): roll back a downgrading leader in the rollback detector test

The table that was not switched yet keeps a downgrading leader in a real
abort, so the test now builds it that way and checks that the rollback
clears it.

Signed-off-by: jeremyhi <[email protected]>

* fix(meta): monitor regions switched before a rollback only on the target

The rollback registered the source's failure detectors for every region
of the migration, including the regions already switched to the target.
Those regions are served by the target after the rollback, so their
source detectors watched the wrong peer: when one fired, the failover
found the leader had changed and only removed it. The rollback now
registers the source's detectors only for the regions that were not
switched, and keeps resetting the target's detectors for the others.

Signed-off-by: jeremyhi <[email protected]>

* refactor(meta): rename reset_failure_detectors_for_candidate_regions to reset_failure_detectors

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 05536cd7f52a3b5fbe96f3d3f672b6e9c3855b3e)
Signed-off-by: WenyXu <[email protected]>

* fix: ignore stale leader grants after a region is downgraded (#9987)

* fix: ignore stale leader grants after a region is downgraded

Signed-off-by: jeremyhi <[email protected]>

* fix: compare stale leader grants with the longest lease seen

Signed-off-by: jeremyhi <[email protected]>

* fix: treat a lease too long to store as unknown

A lease longer than a u64 of nanoseconds saturated to a shorter one, so the
derived send time could be later than the real one. Now such a lease makes
the send time unknown, which counts as earlier than any gate.

Signed-off-by: jeremyhi <[email protected]>

* fix: address review comments on stale leader grants

Scope the early send time claim to after a lease has been seen, drop the
unused role parameter of demote, clarify a test comment and check that a
fresh grant still makes a reopened region writable.

Signed-off-by: jeremyhi <[email protected]>

* refactor: compare stale leader grants with the heartbeat send time

Pass the send time of the heartbeat behind a lease to set_region_role_with_lease,
and return whether the role was applied. Mito compares the send time with the
grant gate directly, so it no longer derives it from the longest lease seen.
The datanode finalizes leadership only when the grant was applied.

Signed-off-by: jeremyhi <[email protected]>

* test: cover stale grants through the datanode and repeated downgrades

Signed-off-by: jeremyhi <[email protected]>

* refactor: keep leader grant rules in one region method

Apply a role grant in MitoRegion::apply_role_grant, which renews the lease
and decides whether the role changes; the engine only looks up the region.
Rename the gate to downgrade_started_at and demote to begin_downgrade, keep
the field private, and copy it on reopen through
inherit_downgrade_state_from.

Signed-off-by: jeremyhi <[email protected]>

* refactor: drop a needless borrow

Signed-off-by: jeremyhi <[email protected]>

* refactor: keep lease concepts in the engine and role ordering in the region

ManifestContext, which owns the role, records when the role was last
demoted and promotes a region only for a promotion issued after that. The
time starts when the context is created, so a reopened region ignores
grants for heartbeats sent before the reopen, without copying state. The
engine translates leases: it renews the deadline and promotes with the
heartbeat send time. MitoRegion keeps no grant state.

Signed-off-by: jeremyhi <[email protected]>

* fix: keep the data region of a metric region off a grant its metadata region ignores

The metric engine applies a grant to the metadata region first and stops
if it is ignored, so the data region, whose role the engine reports, never
becomes a leader that skips finalization. Also describe what promoting
returns.

Signed-off-by: jeremyhi <[email protected]>

* docs: describe when a lease grant is ignored and flatten the finalize check

Signed-off-by: jeremyhi <[email protected]>

* refactor: move lease types out of the region and the Leader deadline into ManifestContext

Signed-off-by: jeremyhi <[email protected]>

* refactor: name the promotion fence after the downgrade it records

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 89a5df31418add4a168c7318d4cae55b90ddce00)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): preserve live index files across remote compactions (#10046)

fix(mito2): reuse executor-owned compaction index resources

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit c7ea80558f9cf53a8feaa4fb6fdcdd07060f4ddf)
Signed-off-by: WenyXu <[email protected]>

* fix(datanode): pick a node epoch above the recorded one (#9984)

* fix(datanode): pick a node epoch above the recorded one

The metasrv fences a datanode whose node epoch is lower than the
highest one it has recorded for the node id. A datanode used its start
time as its epoch, so after a restart on a host whose clock is behind
the previous process's, or after the clock stepped back, the new
process picked a lower epoch and was fenced for good: no region lease,
no keep-alive.

Before it creates the heartbeat task, the datanode now reads the
recorded epoch through its metadata backend and uses one more than it
if its start time is not higher. The node info still reports the local
start time. A failed read fails the startup, like the other metadata
reads.

Signed-off-by: jeremyhi <[email protected]>

* test(datanode): tighten the node epoch tests

Use a lower record that leaves room below the start time, so the test
fails if the start time is ignored, and a nonzero node id, so it fails
if the node id is ignored. Clarify the doc comment of the epoch choice.

This assumes the previous process of a node id has stopped before a new
one starts.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit c40e86c6533eac94b28652f0255f7cd07ea86c15)
Signed-off-by: WenyXu <[email protected]>

* fix(auth): preserve physical metric table targets (#10084)

* fix(auth): preserve physical metric table targets

Signed-off-by: shuiyisong <[email protected]>

* fix: nit

Signed-off-by: shuiyisong <[email protected]>

---------

Signed-off-by: shuiyisong <[email protected]>
(cherry picked from commit c17a5e09f74069a7e62b9a512d6d95229344c219)
Signed-off-by: WenyXu <[email protected]>

* fix: include peer context in distributed query errors (#10118)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 8deb3872b3283b01ebc629a5d1642b7053b65fbb)
Signed-off-by: WenyXu <[email protected]>

* fix(auth): redact inline user-provider credentials (#10115)

* fix(auth): redact inline user-provider credentials

Signed-off-by: shuiyisong <[email protected]>

* fix(auth): close user-provider redaction gaps

Signed-off-by: shuiyisong <[email protected]>

---------

Signed-off-by: shuiyisong <[email protected]>
(cherry picked from commit adf29740b28147cfb9035f53e1c41515b5b519b3)
Signed-off-by: WenyXu <[email protected]>

* fix: register trigger procedure loaders through DdlManager (#10123)

Add a loader registration hook to TriggerDdlManager and invoke it
  before procedure recovery. Test that registration reaches the
  configured trigger DDL manager.

Signed-off-by: fys <[email protected]>
(cherry picked from commit b6cf3384dca74f7871c0746e60918e5b2852e7c8)
Signed-off-by: WenyXu <[email protected]>

* fix(time): parse zero-hour timezone offsets correctly (#10091)

fix(time): preserve zero-hour timezone offset signs

Signed-off-by: Minh Vu <[email protected]>
(cherry picked from commit 25003cea16453526f37b77fa4189c68b52a49a3b)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): invalidate all datanode tables on address change (#9986)

When a datanode restarts with a new address, metasrv invalidated cached
table routes only for the tables in that heartbeat's region stats. The
first heartbeat after a restart often reports few or no regions (e.g.
with init_regions_in_background), so frontends kept dialing the old
address until metadata_cache_ttl expired.

Read the previous address via prev_kv when saving it, and when it
changed (or a failed save of the same epoch is retried) invalidate
every table registered for the datanode in __dn_table.

Signed-off-by: Floris Smit <[email protected]>
Co-authored-by: Weny Xu <[email protected]>
(cherry picked from commit 351f4207b4446e48befc02b03779849013fc2886)
Signed-off-by: WenyXu <[email protected]>

* fix(function): require ADMIN statement privileges to execute admin functions (#10120)

* fix(function): require ADMIN statement privileges to execute admin functions

Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.

Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.

Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.

Signed-off-by: Ning Sun <[email protected]>

* fix(function): expose admin privilege check to downstream macros

Signed-off-by: Ning Sun <[email protected]>

---------

Signed-off-by: Ning Sun <[email protected]>
(cherry picked from commit b21b1cdb693a0017b82ca667f347688e6fd6cbba)
Signed-off-by: WenyXu <[email protected]>

* fix(catalog): use advertised peer address as process identity (#10153)

* fix(catalog): use advertised peer address as process identity

The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.

Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
  manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
  in list_all_processes, and for local/remote routing in kill_process.

Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.

The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.

Signed-off-by: Lei, HUANG <[email protected]>

* test: keep local peer address reserved in kill-routing test

The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.

Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.

Signed-off-by: Lei, HUANG <[email protected]>

---------

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit 557843dd680bf3ac94befea55377583442732d58)
Signed-off-by: WenyXu <[email protected]>

* fix(mito): use stored recovery-set bytes for manifest size (#10142)

* fix(mito): track stored bytes for manifest recovery

Signed-off-by: WenyXu <[email protected]>

* fix(mito): retain manifest sizes on checkpoint and sync

Signed-off-by: WenyXu <[email protected]>

* test(mito): cover logical manifest size accounting

Signed-off-by: WenyXu <[email protected]>

* docs(mito): clarify manifest size retention invariants

Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit d7036df73c99d496510eec22bbf70cd731e93d5f)
Signed-off-by: WenyXu <[email protected]>

* fix(grpc): propagate cancellation for read query execution (#10166)

* x

* fix(grpc): restrict cascading cancellation to read query variants

Signed-off-by: luofucong <[email protected]>

---------

Signed-off-by: luofucong <[email protected]>
(cherry picked from commit 2cb473607d8328489978040be67b8f44366fd7e9)
Signed-off-by: WenyXu <[email protected]>

* fix(datatypes): escape object keys when rendering jsonb as text (#10112)

* fix(datatypes): escape object keys when rendering jsonb as text

Signed-off-by: Yvan Wang <[email protected]>

* fix(datatypes): skip unicode normalization for escaped jsonb keys

Signed-off-by: Yvan Wang <[email protected]>

* refactor(datatypes): serialize jsonb to text directly with serde_json

Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.

Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.

Signed-off-by: Yvan Wang <[email protected]>

* fix(datatypes): accept JSON text bytes in jsonb_to_string

Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.

Signed-off-by: Yvan Wang <[email protected]>

---------

Signed-off-by: Yvan Wang <[email protected]>
(cherry picked from commit ec1ad1f15b79b6e24e86ce52cd0578a3b7d979bc)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): share bulk memtable compaction dispatcher across workers (#10022)

Each region worker built its own MemtableBuilderProvider, and every
provider created its own CompactDispatcher. The limit on background
bulk memtable compactions was therefore num_workers times
max_background_compactions, while SST compaction uses one
engine-wide pool of max_background_compactions.

Create one dispatcher in WorkerGroup::start (and start_for_test),
pass it to every worker, and make MemtableBuilderProvider::new take
it instead of creating its own. SST and memtable compaction keep
separate budgets of the same size.

Signed-off-by: luckys <[email protected]>
Co-authored-by: luckys <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: dennis zhuang <[email protected]>
(cherry picked from commit 9f7cedf58f92b89c79e5f32f41ed78c1ce311472)
Signed-off-by: WenyXu <[email protected]>

* fix(mito): decode cached manifest payloads (#10179)

* test(mito2): cover compressed manifest cache recovery

Signed-off-by: WenyXu <[email protected]>

* fix(mito2): decode cached manifest object payloads

Signed-off-by: WenyXu <[email protected]>

* fix(mito2): address manifest cache review feedback

Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 4263f5fc016b65a1f56fe99a6b2a4aac8bdcf6cf)
Signed-off-by: WenyXu <[email protected]>

* fix: prioritize region ALTER over queued edits (#10193)

* fix: prioritize region ALTER over queued edits

Signed-off-by: evenyag <[email protected]>

* test: use valid edits in ALTER priority regression

Signed-off-by: evenyag <[email protected]>

* refactor: make region edit dispatch outcomes explicit

Signed-off-by: evenyag <[email protected]>

* fix: reject queued alters and edits when dropping regions

Signed-off-by: evenyag <[email protected]>

* test: cover queued ALTER handoff before region edits

Signed-off-by: evenyag <[email protected]>

---------

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit ae422ef46532836a7b3203888773055b589b9bbb)
Signed-off-by: WenyXu <[email protected]>

* fix(meta-srv): invalidate leader cache on transaction errors (#10219)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 4688fa4be83dde7e513c393093c2a170a1e540d3)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): avoid reporting reader filter metrics twice (#10222)

* perf(mito2): cache fixed-label reader metric counters

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): report reader filter metrics only once

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 5d88d283e5551d7ad14f1f1ac3ee451a1c9c4445)
Signed-off-by: WenyXu <[email protected]>

* perf(mito2): reuse static manifest checks across row groups (#10220)

* perf(mito2): reuse static manifest checks for cached readers

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): drop redundant pruner test and release entry lock early

Signed-off-by: Dennis Zhuang <[email protected]>

* perf(mito2): cache static manifest keep verdicts directly

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit c844d447218ac341004d7154f48925b0a8c6747a)
Signed-off-by: WenyXu <[email protected]>

* fix: adapt v1.3.1 backports to release interfaces

Signed-off-by: WenyXu <[email protected]>

* fix!: enforce sparse primary key schema (#10020)

* fix: preserve required sorting for ordinary sparse scans

Signed-off-by: shuiyisong <[email protected]>

* fix: enforce sparse primary key schema

Signed-off-by: shuiyisong <[email protected]>

* test: fix

Signed-off-by: shuiyisong <[email protected]>

---------

Signed-off-by: shuiyisong <[email protected]>
(cherry picked from commit efed847ca6dadd37a76fbbaca2a454621c505a92)
Signed-off-by: WenyXu <[email protected]>

* fix: test (#10149)

fix: fixture

Signed-off-by: shuiyisong <[email protected]>
(cherry picked from commit 4633c37dbbe9ae369732a8023ca5c0a23f17e9cf)
Signed-off-by: WenyXu <[email protected]>

* fix: tolerate invalid UTF-8 in JSON log ingestion (#10245)

* fix: tolerate invalid UTF-8 in JSON ingestion

Signed-off-by: shuiyisong <[email protected]>

* fix: tolerate invalid UTF-8 in NDJSON ingestion

Signed-off-by: shuiyisong <[email protected]>

* chore: upgrade simd json

Signed-off-by: shuiyisong <[email protected]>

* feat: add server default for ignore_errors

Signed-off-by: shuiyisong <[email protected]>

---------

Signed-off-by: shuiyisong <[email protected]>
(cherry picked from commit 0cb70d4f84b2fe1b4d6988fe75c970f862c2794a)
Signed-off-by: WenyXu <[email protected]>

* test: simplify backported JSON recovery cases

Signed-off-by: WenyXu <[email protected]>

* feat(otlp): add v1 schema for auto-created OTLP log tables (#10207)

* feat(otlp): add v1 schema for auto-created OTLP log tables

New tables created by direct OTLP logs ingestion get a v1 built-in schema:
service_name tag from resource service.name ahead of scope_name in the
primary key, a skipping index on trace_id, and event_name and
observed_timestamp fields. The table is marked with
table_data_model=greptime_log_v1. Existing tables without the marker keep
the legacy schema.

Signed-off-by: Dennis Zhuang <[email protected]>

* refactor(otlp): share resource service.name parsing between logs and traces

Log tables now derive service_name with the same rules as trace spans, so
both signals agree on the value used to correlate them. The integration test
writes twice to cover the existing v1 table path.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 2714687ab701a2bf046b579a2aa2e255552b9df8)
Signed-off-by: WenyXu <[email protected]>

* fix(promql): bound scatter time filter expression depth (#10272)

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit 361f3595db03f6297d8d45fba3fec924e6e7a776)
Signed-off-by: WenyXu <[email protected]>

* fix(query): use current partition columns in prepared plans (#10244)

A prepared statement keeps its logical plan, and that plan holds the
table as it was at prepare time. The analyzer reads the partition
columns from that table while the regions to scan come from the current
route. After a table's partition columns change, the analyzer can
consider a query safe to push down, so each region runs the whole query
and the frontend concatenates their results. The same happens to the
source of a prepared INSERT ... SELECT, which then stores wrong values.

Before executing a kept plan, compare each scanned table's partition
columns with the current table found by id. Scans of a table whose
columns changed get the current table when the schema is unchanged.
Otherwise they keep their schema, because a scan's projection holds
indices into it, and report the current columns as absent from it, so
the query keeps merging on the frontend.

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 297a343fd45f07ea267eb88d79eb592f5a912240)
Signed-off-by: WenyXu <[email protected]>

* fix(time): avoid i32 overflow when converting an out-of-range date (#10299)

Signed-off-by: Arpit Jain <[email protected]>
Co-authored-by: Claude <[email protected]>
(cherry picked from commit 402e4a97fc0b7c082cc419b380c6014388dd1a10)
Signed-off-by: WenyXu <[email protected]>

* feat: update dashboard to v0.14.5 (#10270)

Co-authored-by: dennis zhuang <[email protected]>
(cherry picked from commit 4c4c8a58fd85ef38feef90f5a076e4e1b863e2d7)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): keep rows written during staging in the WAL (#10253)

* fix(mito2): keep rows written during staging in the WAL

A flush during staging committed its SSTs to the staging manifest, then
dropped the memtable and released the WAL. If staging did not end
successfully those rows were gone. A flush that committed after staging
had ended lost its rows even though staging succeeded.

Staging regions no longer flush. They are skipped by memory-pressure,
periodic and region-limit flushes, and their writes are rejected instead
of queued at twice the region limit or when the node's staging memtables
reach half of the engine's flush limit. A manual or repartition flush on
a staging region with data gets a retryable error; the migration
pre-flush and the WAL prune flush reply success without flushing. Closing
a staging region leaves staging first, so skip-WAL rows are flushed to
the normal manifest.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): keep accounting and closing staging regions during staged edits

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): track staged edits without relying on the partition info

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): synchronize staging close tests with worker admission

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): observe the manifest lock wait when closing staging regions

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): detect staged edits through the staging partition info

Signed-off-by: jeremyhi <[email protected]>

* docs(mito2): record that staging regions must not flush

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): do not restore staging after a close abandons the edit

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): reject region edits during a close-time flush

Signed-off-by: jeremyhi <[email protected]>

* style(mito2): format the staging tests

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): reject staging edits once staging is abandoned

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): bind apply-staging edits to their partition expression

Signed-off-by: jeremyhi <[email protected]>

* docs(mito2): note that staging regions pin their topic's prune point

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 8da533c35ffc901db22827c62ed2877f2e33e6c5)
Signed-off-by: WenyXu <[email protected]>

* perf(query): push column projections through MergeScan (#10221)

* perf(query): push column projections through MergeScan

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(query): retain empty projections and refresh sqlness plans

Signed-off-by: Dennis Zhuang <[email protected]>

* test(query): update TSID join plans after projection pushdown

Signed-off-by: Dennis Zhuang…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required This change does not impact docs. documentation size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants