Skip to content

perf(promql): push down last row for instant queries - #9034

Merged
discord9 merged 20 commits into
GreptimeTeam:mainfrom
discord9:perf/promql-instant-last-row-hint
Sep 9, 2026
Merged

discord9 merged 20 commits into
GreptimeTeam:mainfrom
discord9:perf/promql-instant-last-row-hint

Conversation

@discord9

@discord9 discord9 commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

Supersedes the query-side portion of #8759 with a smaller implementation based on the current scan path.

What's changed and what's your intention?

PromQL instant queries evaluate one timestamp but currently read every sample in the lookback window before InstantManipulate keeps the latest sample per series. This PR passes the existing TimeSeriesRowSelector::LastRow hint to each concrete table-scan use site when InstantManipulate has start == end.

The optimizer now performs one scoped logical-plan rewrite so the hint follows the actual parent path instead of relying on matching traversal order across two passes. It clones each hinted DummyTableProvider scan request, which keeps shared providers isolated when instant and range scans coexist in one plan.

Correctness boundaries:

  • The instant LastRow hint currently applies only to second/millisecond time indexes. Microsecond and nanosecond tables, including OTLP metrics stored as TIMESTAMP(9), remain on the unhinted path until native timestamp selection semantics are preserved.

  • Range evaluations (start != end) do not receive the hint.

  • Expression subqueries do not inherit an outer instant-query hint; an instant query inside a subquery can establish its own hint.

  • Limit/TopK blocks instant LastRow; existing vector hints remain supported on that path. If an eligible scan has competing LastRow/vector hints, they are not combined.

  • Existing ordering, per-series distribution, aggregate last_value, and vector-only hints remain supported.

Safe selector-path allowlist

The rule is deliberately scoped to plans emitted by the PromQL planner, not arbitrary logical plans. Between a single-evaluation InstantManipulate and a scan, it permits:

  • Sort only without fetch (not TopK), and SubqueryAlias.
  • Projection only for column references, same-name column aliases, or the planner's same-name seconds/milliseconds-to-milliseconds timestamp cast. The planner must preserve time-index and series identity.
  • SeriesDivide, and SeriesNormalize only when filter_stale_markers == false (the instant-offset path).
  • TableScan, subject to the separate attached-predicate and seconds/milliseconds precision checks. Only recognized tag/time scan predicates qualify; field predicates and unsupported expressions do not.

All other nodes—including residual Filter, Limit, fetching Sort, Union, Join, Aggregate, Window, and unknown extensions—block the instant-derived hint. A blocker remains effective down that path; a nested InstantManipulate establishes its own evaluation scope. Existing aggregate hints are handled separately.

Typical eligible chain after predicate pushdown:

InstantManipulate(start == end)
  → [SeriesNormalize(offset, filter_stale_markers=false)]
  → SeriesDivide → Sort(fetch=None)
  → [restricted Projection / SubqueryAlias] → TableScan

Examples:

Instant → Filter → Scan           blocked: filtering precedes sample selection
Filter → Instant → Scan           eligible: filtering follows sample selection
Instant → Limit(1) → Sort → Scan   blocked: LIMIT changes the candidate rows

For one series with (ts=100ms, value=10) and (ts=900ms, value=1), an ascending sort followed by LIMIT 1 must keep 10. Applying LastRow below that limit would incorrectly keep 1. This is a logical-plan composition regression test, not a claim that ordinary PromQL selectors generate that LIMIT chain.

The instant hint selects LastRow after the existing merge/deduplication path, so SST-local shortcuts cannot discard rows needed for deletion or timestamp correctness. The existing aggregate hint retains its behavior.

Verification of the allowlist (1283629c2cd):

  • cargo fmt --check; cargo check -p query -p promql --tests.
  • cargo nextest run -p query --features vector_index scan_hint --retries 0: 47 passed.
  • cargo nextest run -p query -p promql --retries 0: 927 passed, 2 skipped.
  • Built greptime; ran sqlness-runner bare --bins-dir <fresh-binary-dir> --case-dir tests/cases -t instant_last_row, with and without --enable-flat-format: standalone/distributed passed in both formats; expected results unchanged.
  • Actual TQL ANALYZE confirms LastRow remains active for ordinary, tag-filtered, offset, seconds, and metric-engine selectors, and absent for range and field-value matchers. Returned-value checks also pass.

Reviewer follow-up (4f0d92437f8):

  • The performance regression fixture now uses TIMESTAMP(3). Actual plan validation confirms instant LastRow on the candidate only, with range controls unhinted on both revisions.
  • Returned-value regressions cover post-selection comparison, a field-value matcher, and filtered SQL last_value. SQLness passes in standalone and distributed modes with both primary and flat SST formats.

Display review and rebase

Rebased onto main 0d5d97e7cb6b979bbd979ab38976664ab8e62bb4; current revision: 8ab31ab86e97a38149394b77f2887dff3fa75e98.

Removed the diagnostic display compatibility handling: LastRow { after_merge: true/false } is now shown directly, without a separate series_row_selector_after_merge field. Regenerated six EXPLAIN/ANALYZE result files (37 selector-string changes only).

Validation after rebase:

  • cargo fmt --check.
  • cargo nextest run -p query -p promql -p store-api --retries 0: 1,003 passed, 2 skipped.
  • Performance harness case-selection and slash-command Python tests: 3 + 31 passed.
  • Fresh debug greptime build; 13 selected SQLness cases in each of Basic and Flat formats, across standalone/distributed (26 executions), with no further expected-output changes.

Performance regression CI (2026-09-09)

Ordinary instant-query follow-up: CI run 34345776369 passed on separate branch perf/pr9034-eval-ci (06cac59017e271700ed24ecb0abd32d706ea9339). That commit only adds benchmark queries; production code is identical to this PR's 8ab31ab86e9. Base remains 0d5d97e7cb6.

Ordinary query (TQL EVAL, no ANALYZE output) Base median / p95 Candidate median / p95 Median change
Instant selector 94.89 / 98.89 ms 25.54 / 26.73 ms −73.1%
Range control 104.55 / 107.29 ms 104.68 / 121.36 ms +0.1%

Each query used 3 warmups and 9 measurements per target. Inspected all measured EVAL responses: instant returns 256 identical rows and range returns 2,816 identical rows across both targets and all repetitions. This measures ordinary query execution and normal result serialization through /v1/sql, not the Prometheus /api/v1/query endpoint. It confirms the gain in this workload does not depend on ANALYZE metrics/plan transfer. The same run's ANALYZE instant median decreased 66.3%; the earlier diagnostic-only measurement below remains separately labelled. No local timing figures are used.

Run 34337454470 passed, comparing base 0d5d97e7cb6b979bbd979ab38976664ab8e62bb4 with final candidate 8ab31ab86e97a38149394b77f2887dff3fa75e98 using the optimized nightly profile on the CI ECS runner.

The instant and range queries measure TQL ANALYZE VERBOSE end-to-end request latency, not TQL EVAL; the aggregate control is ordinary SQL. Each query has 3 warmups and 9 measured requests per target. Changes below compare the two medians, not paired samples.

Query Base median ms Base p95 ms Candidate median ms Candidate p95 ms Median latency change
instant_selector_last_row_candidate 100.24 133.01 34.26 35.30 -65.8%
range_selector_last_row_control 99.97 110.15 102.28 124.90 +2.3%
sql_aggregate_last_row_control 23.25 23.92 22.94 23.56 -1.4%
  • Instant-selector median latency fell 65.8% (100.24 → 34.26 ms). Both controls stayed within the configured +25% regression limit; all three thresholds passed.
  • Fixture: TIMESTAMP(3), one region, flat append-mode, 64 SSTs × 4,096 rows (262,144 rows), 256 series, non-overlapping SST time ranges and timestamp-major generation.
  • Inspected all measured instant/range plans in the CI JSON: candidate instant uses LastRow { after_merge: true }; base instant and both range controls have no LastRow selector. All 54 measured requests succeeded and both targets report zero validation errors. Returned-value correctness is covered separately by the tests above; ANALYZE output is not a row-by-row result oracle.
  • Evidence: this run's query-regression-report artifact contains query-regression-summary.md, the JSON report with samples/plans, fixture metadata and component logs. These are CI measurements for this workload, not a general latency guarantee. No local timing figures are used here, and existing CI runs were not cancelled.

PR Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.
  • This PR needs to be backported to release branches, and I have added the backport-<target> labels (e.g. backport-v1.3 targets release/v1.3).

@github-actions github-actions Bot added size/M docs-not-required This change does not impact docs. labels Sep 4, 2026
@discord9
discord9 force-pushed the perf/promql-instant-last-row-hint branch from d5f77c9 to 08164b7 Compare September 7, 2026 05:13
@github-actions github-actions Bot added size/L and removed size/M labels Sep 7, 2026
@evenyag

evenyag commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

I wonder the last row selector may not return the correct timestamp we want. The last timestamp may not always be within the time range we need.

@discord9
discord9 marked this pull request as ready for review September 8, 2026 03:18
@discord9
discord9 requested review from a team, WenyXu, evenyag and v0y4g3r as code owners September 8, 2026 03:18
@github-actions github-actions Bot added size/XXL and removed size/XL labels Sep 8, 2026
@discord9

discord9 commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor Author

I wonder the last row selector may not return the correct timestamp we want. The last timestamp may not always be within the time range we need.

@evenyag Added SQLness coverage for this: an in-window sample followed by a future sample in the same series, lookback boundaries, and positive/negative offsets. The query returns the in-window sample. All cases pass in standalone and distributed modes with both SST formats. Tests and fixes are pushed.

Comment thread src/query/src/optimizer/scan_hint.rs Resolved
Comment thread src/query/src/optimizer/scan_hint.rs Resolved
Comment thread src/mito2/src/read/series_scan.rs Resolved
Comment thread src/query/src/dummy_catalog.rs Resolved
Comment thread src/query/src/optimizer/scan_hint.rs Resolved
Comment thread src/query/src/optimizer/scan_hint.rs Resolved
Comment thread src/store-api/src/storage/requests.rs Resolved
@discord9
discord9 force-pushed the perf/promql-instant-last-row-hint branch from 1283629 to 8ab31ab Compare September 9, 2026 09:55
@discord9
discord9 added this pull request to the merge queue Sep 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 9, 2026
@discord9
discord9 added this pull request to the merge queue Sep 9, 2026
Merged via the queue into GreptimeTeam:main with commit da5cb1a Sep 9, 2026
59 checks passed
@discord9
discord9 deleted the perf/promql-instant-last-row-hint branch September 9, 2026 13:15
Aumnertic pushed a commit to Aumnertic/greptimedb that referenced this pull request Oct 20, 2026
…Team#9737)

* perf(mito2): gather last rows with one take per input batch

FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (GreptimeTeam#9034) slower. Select the rows of
series that end inside a batch with one take instead.

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): cover duplicate last timestamps in the take path

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): group last rows by primary key bytes

Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
WenyXu pushed a commit to WenyXu/greptimedb that referenced this pull request Oct 27, 2026
…Team#9737)

* perf(mito2): gather last rows with one take per input batch

FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (GreptimeTeam#9034) slower. Select the rows of
series that end inside a batch with one take instead.

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): cover duplicate last timestamps in the take path

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): group last rows by primary key bytes

Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 4645ab9)
Signed-off-by: WenyXu <[email protected]>
WenyXu pushed a commit to WenyXu/greptimedb that referenced this pull request Oct 27, 2026
…Team#9737)

* perf(mito2): gather last rows with one take per input batch

FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (GreptimeTeam#9034) slower. Select the rows of
series that end inside a batch with one take instead.

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): cover duplicate last timestamps in the take path

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): group last rows by primary key bytes

Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 4645ab9)
Signed-off-by: WenyXu <[email protected]>
WenyXu pushed a commit to WenyXu/greptimedb that referenced this pull request Oct 27, 2026
…Team#9737)

* perf(mito2): gather last rows with one take per input batch

FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (GreptimeTeam#9034) slower. Select the rows of
series that end inside a batch with one take instead.

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): cover duplicate last timestamps in the take path

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): group last rows by primary key bytes

Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 4645ab9)
Signed-off-by: WenyXu <[email protected]>
WenyXu added a commit that referenced this pull request Oct 28, 2026
* fix(hdfs): clean up temporary file when an atomic write fails (#9483)

* fix(hdfs): clean up temporary file when an atomic write fails

Signed-off-by: fengyue-xve <[email protected]>

* style: apply rustfmt to hdfs layer test

Signed-off-by: fengyue-xve <[email protected]>

* fix(hdfs): report earlier write failures and log cleanup failures

- Later `write`/`close` calls on a writer whose failed write was
  cleaned up now mention the original failure instead of the bare
  "unavailable after close or abort" message.
- `abort_and_delete` logs abort and deletion failures with the
  temporary path instead of discarding them; the original write error
  is still returned to the caller.
- The temporary-file cleanup test now lists the directory holding the
  temporary file, so it fails if the cleanup is reverted.

Signed-off-by: 风月 <[email protected]>
Signed-off-by: fengyue-xve <[email protected]>

---------

Signed-off-by: fengyue-xve <[email protected]>
Signed-off-by: 风月 <[email protected]>
(cherry picked from commit a35661d)
Signed-off-by: WenyXu <[email protected]>

* fix(jaeger): return the most recent traces when a search hits its limit (#9595)

* fix(jaeger): return the most recent traces when a search hits its limit

Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search
matching more traces than the limit returned an arbitrary subset that
changed between identical requests. Order traces by their latest matching
span before applying the limit.

Signed-off-by: Dennis Zhuang <[email protected]>

* refactor(jaeger): name the trace dataframe builder with a verb and document execution

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit cf16c73)
Signed-off-by: WenyXu <[email protected]>

* fix(pprof): initialize framehop before signal sampling (#9612)

fix(pprof): pin upstream framehop initialization fix

Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes.

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit 0dc8177)
Signed-off-by: WenyXu <[email protected]>

* chore: bump rustls to 0.23.46 (#9614)

rustls 0.23.46 fixes rustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout.

(cherry picked from commit 33879b0)
Signed-off-by: WenyXu <[email protected]>

* fix(prom): return empty remote read results for missing metrics (#9613)

* fix: return empty remote read results for missing metrics

Signed-off-by: fys <[email protected]>

* test(prom): cover empty reads for missing databases

Signed-off-by: fys <[email protected]>

* test(prom): simplify remote read regression cases

Signed-off-by: fys <[email protected]>

* fix(prom): log missing tables during remote read

Signed-off-by: fys <[email protected]>

---------

Signed-off-by: fys <[email protected]>
(cherry picked from commit cf0f6c3)
Signed-off-by: WenyXu <[email protected]>

* fix(ci): compare release validation against the previous stable version (#9636)

Ancestry-based tag selection skips stable tags that live only on older release branches.

Signed-off-by: paomian <[email protected]>
(cherry picked from commit 3de8610)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): keep processing bulk requests after one fails to encode (#9638)

* fix(mito2): keep processing bulk requests after one fails to encode

When `push_bulk` fails to build the WAL entry of a bulk request, it answers
that request with the error. `process_bulk_requests` then returned from the
loop, so the remaining requests of the drained batch were dropped without an
answer and their callers only saw a receive error. Skip the failed request
and continue with the rest of the batch instead.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): keep bulk install barriers independent per target

The bulk install test barrier was a single process-wide slot, so tests
that arm it while running in the same process replaced each other's
barrier. Keep one barrier per armed target instead.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 88bdab0)
Signed-off-by: WenyXu <[email protected]>

* fix(promql): honor @ modifier on subqueries (#9446)

* fix(promql): honor @ modifier on subqueries

`prom_subquery_expr_to_plan` never read the `at` field of a subquery, so
`sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and
followed the outer evaluation grid.

Handle `@` the way Prometheus does: fold the anchor into the subquery
offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier`
does), evaluate the subquery once at the start of the evaluation, and
replay the folded window over the outer grid, reusing the anchored
range selector machinery. A call whose range argument is an anchored
subquery is promoted as step-invariant, like one over an anchored range
selector. `@ start()`/`@ end()` and `offset` combine as for selectors,
and an unrepresentable anchor is rejected with the existing error.

Closes #9411

Signed-off-by: Aarav <[email protected]>

* test(promql): add sqlness cases for @ on subqueries

Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`,
`@` combined with `offset` in either order, an instant query away from
the anchor, `*_over_time` functions and `rate` over an anchored
subquery, a computed inner expression, and an aggregation above the
anchored call. Two series per table so a dropped or merged series
shows up as a wrong row.

Refs #9411

Signed-off-by: Aarav <[email protected]>

---------

Signed-off-by: Aarav <[email protected]>
(cherry picked from commit 095c288)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): validate bulk insert batches against the region schema (#9698)

* fix(mito2): validate bulk insert batches against the region schema

Bulk inserts were only checked for the time index column, unlike row
writes, which WriteRequest::check_schema checks against the region
schema. A batch with columns of another type, nulls in the time index or
in non-nullable columns, duplicate or unknown columns reached the region
worker: a time index that is not a timestamp or holds only nulls made
the worker fail, and other mismatches failed after the WAL append or
were stored as written, depending on the SST format.

Move the column checks of check_schema into a core shared by row writes
and bulk inserts, and check bulk insert batches at the same points as
rows: when the engine builds the worker request, and again in the worker
if the region schema changed since. check_schema keeps its behavior.
Bulk insert columns must have the column's Arrow type, except that tags
may use any string encoding, as the memtables already convert them.
Sparse batches must carry one encoded primary key column of binary
without nulls. A batch that misses columns gets them filled by the
worker even if it claims to match the current schema.

The metric engine checks logical bulk inserts against the logical region
before rewriting them into the physical region, as the rewrite copies
columns without checking them and would encode a duplicated tag into the
primary key twice.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): report an error when a write is interrupted by a panic

WriteNotify answers its waiter when it is dropped and reported success
unless an error had been set, also when it was dropped while unwinding
from a panic in the middle of the write. Report an error in that case.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): tolerate a poisoned series lock when dropping a series map

SeriesMap::drop reads every series to update the memtable metrics and
unwrapped the lock, so dropping the map while unwinding from a panic
that poisoned a series lock panicked again. Read through the poisoned
lock instead.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 791f429)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): fence writes before a graceful downgrade (#9716)

* test(mito2): cover graceful role changes racing with writes

A graceful role change flips the region state in a background task while
the worker keeps handling writes. Add bulk and row write tests that check a
write handled after the role change request is either rejected or covered
by the returned last_entry_id.

Add tests for DowngradingLeader while an ALTER keeps the region in
Altering, alone and in the downgrade sequence the datanode runs
(DowngradingLeader, Flush, Follower) with a migration candidate on its own
local WAL. Add a test that the downgrade sequence covers a write in flight
when a lease renewal sets Downgrading.

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): fence writes before a graceful downgrade

set_region_role_state_gracefully hands the role change to a background task
that waits for the manifest lock. The worker keeps handling writes until the
task flips the state, so a write could be acknowledged with an entry id above
the returned last_entry_id. Set Downgrading in the worker before spawning the
task for Follower and DowngradingLeader requests, as lease renewal already
does, so every write the worker handles afterwards is rejected.

DowngradingLeader also did nothing and returned success when the region was
in another leader state such as Altering. The region became writable again
when the operation finished, and the datanode then flushed and turned it
into a follower in one step. Return a retryable error instead, so metasrv
retries the downgrade.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): assert the retryable downgrade error during ALTER

Check that DowngradingLeader during an ALTER returns a retryable error, and
release the paused ALTER on that path. The local WAL test now retries the
downgrade after the ALTER, as metasrv does.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): write before the retried downgrade in the local WAL test

Rows written before an ALTER were already flushed, so the candidate check
did not depend on the downgrade's final flush. Write rows after the ALTER
and before the downgrade is retried, which only that flush can carry to the
candidate.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 660c4ef)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): validate persisted partition bounds before altering regions (#9487)

* fix(meta): validate persisted partition bounds before altering regions

A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.

Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.

Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.

Fixes #9454

Signed-off-by: fengyue-xve <[email protected]>

* fix: access ColumnSchema fields directly and use plain format args

ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.

Signed-off-by: fengyue-xve <[email protected]>

* test: access ColumnSchema data_type field in alter table tests

Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.

Signed-off-by: fengyue-xve <[email protected]>

* test: drop borrows now that data_type is read as a field

Signed-off-by: fengyue-xve <[email protected]>

* test: drop useless format! in nested expression test

Signed-off-by: fengyue-xve <[email protected]>

* test: cover the partition tail in change_col_type sqlness case

Signed-off-by: fengyue-xve <[email protected]>

* test: drop spurious blank line in change_col_type result

Signed-off-by: fengyue-xve <[email protected]>

* fix(meta): compare partition bounds with the scan comparison coercion

The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.

Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.

Signed-off-by: fengyue-xve <[email protected]>

* test: update the sqlness fixture for the new incompatibility message

The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.

Signed-off-by: fengyue-xve <[email protected]>

* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency

common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.

Signed-off-by: fengyue-xve <[email protected]>

* fix(meta): reject every non-widening type change of a partitioned column

The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.

Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.

Signed-off-by: fengyue-xve <[email protected]>

---------

Signed-off-by: fengyue-xve <[email protected]>
(cherry picked from commit 1e29e8d)
Signed-off-by: WenyXu <[email protected]>

* fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)

* fix(partition): route NULLs like row inserts and accept signed-minimum bounds

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(partition): keep equality two-valued for NULL rows and accept unsigned -0

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(partition): treat NULL masks as false in batch routing and keep comparisons bare

Signed-off-by: Dennis Zhuang <[email protected]>

* test(compat): update partition filter in analyze verbose expected output

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit da88ff8)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): re-check a filled write as sent after a schema change (#9701)

* fix(mito2): re-check a filled write as sent after a schema change

The engine fills the columns a write leaves out with their defaults when it
accepts the write, appending them to the rows without adding them to
`name_to_index` and `has_null`. If the region schema changes before the
worker handles the write, the worker checks the request again and
`check_schema` looks up a filled column in `name_to_index`, which panics and
stops the region worker. If the filled column was dropped, the write fails
with an unknown column that the client never sent.

Remove the filled columns before checking, so the request is checked as sent
and filled again for the current schema. Nothing is filled yet when the
engine checks the request, so that check does no extra work.

Signed-off-by: jeremyhi <[email protected]>

* test(mito2): rename put_request helper to build_put_request

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
(cherry picked from commit 93c3b11)
Signed-off-by: WenyXu <[email protected]>

* perf(mito2): gather last rows with one take per input batch (#9737)

* perf(mito2): gather last rows with one take per input batch

FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (#9034) slower. Select the rows of
series that end inside a batch with one take instead.

Signed-off-by: Dennis Zhuang <[email protected]>

* test(mito2): cover duplicate last timestamps in the take path

Signed-off-by: Dennis Zhuang <[email protected]>

* fix(mito2): group last rows by primary key bytes

Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 4645ab9)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): apply SST internal column encodings to bulk-encoded parts (#9735)

* fix(mito2): apply SST internal column encodings to bulk-encoded parts

Bulk memtable parts are encoded to parquet and flushed as SSTs as-is, but the
bulk encoder did not apply the time index / __sequence DELTA_BINARY_PACKED
encoding (with dictionary disabled) and the uncompressed op_type setting that
the regular SST writer uses. Share these settings through one helper.

Signed-off-by: Dennis Zhuang <[email protected]>

* chore(mito2): drop the answered bulk encoder compression TODO

Signed-off-by: Dennis Zhuang <[email protected]>

---------

Signed-off-by: Dennis Zhuang <[email protected]>
(cherry picked from commit 977e76d)
Signed-off-by: WenyXu <[email protected]>

* fix: avoid exact scan row counts for SSTs with unknown counts (#9759)

fix: avoid exact scan row counts for legacy SSTs

Treat selected SSTs with unknown row counts as non-exact before checking region partition visibility. Preserve exact statistics for known counts and cover legacy metadata mixed with known-count files.

Fixes #9718

Signed-off-by: Aumnertic <[email protected]>
(cherry picked from commit 6f53d57)
Signed-off-by: WenyXu <[email protected]>

* fix: avoid duplicate keys in update flow metadata txn (#9816)

CREATE OR REPLACE FLOW deleted and then put the same flow route,
flownode flow and table flow keys in a single txn when the replacement
kept the partitions, flownodes or source tables, which etcd rejects
with "duplicate key given in txn request".

Only delete the old keys that are not put again. Closes #9810.

Signed-off-by: Lei, HUANG <[email protected]>
(cherry picked from commit a1b2362)
Signed-off-by: WenyXu <[email protected]>

* fix(ci): increase release integration test stack size (#9817)

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit e6fa1a0)
Signed-off-by: WenyXu <[email protected]>

* fix(datanode): release region map guard before awaiting role changes (#9824)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 0dc649e)
Signed-off-by: WenyXu <[email protected]>

* fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860)

* fix(ci): increase Windows nightly test stack size

Signed-off-by: evenyag <[email protected]>

* fix(ci): align optional macOS artifact test stack size

Signed-off-by: evenyag <[email protected]>

---------

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit 8b1d683)
Signed-off-by: WenyXu <[email protected]>

* fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)

* fix(mito2): keep SSTs when a manifest save ends with RateLimited

The object store's retry layer retries temporary errors and returns the
last attempt's error. A manifest delta PUT can be stored by the server
while the client sees a timeout, reset or 5xx, and the retries that
follow can be throttled, so the save ends with RateLimited even though
the delta exists. may_have_persisted_manifest_update treated RateLimited
as proof that nothing was written, so flush and compaction deleted the
SSTs the stored delta references. A later open or follower sync replays
the delta and reads fail on the missing files.

Treat RateLimited like other ambiguous errors: the SSTs are kept and
become orphans when the save really failed.

Fixes #9897

Signed-off-by: jeremyhi <[email protected]>

* fix(mito2): keep SSTs when a manifest save ends with PermissionDenied

The retry layer returns the last attempt's error, so a 403 from a revoked
key, a changed bucket policy or clock skew can follow a delta PUT that
the server already stored. Like RateLimited, PermissionDenied does not
prove that nothing was written, so it no longer lets flush or compaction
delete the SSTs; they become orphans when the save really failed.

The compaction test that rejects a manifest write before persistence now
injects Unsupported, which is raised before any request is sent.

Signed-off-by: jeremyhi <[email protected]>

---------

Signed-off-by: jeremyhi <[email protected]>
Co-authored-by: Yingwen <[email protected]>
(cherry picked from commit 1cb01d0)
Signed-off-by: WenyXu <[email protected]>

* fix: upload staging manifests in chunks (#9919)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit f2be11a)
Signed-off-by: WenyXu <[email protected]>

* perf: stream and prune SST manifest scans (#9931)

* refactor: add reusable metadata pruning and projected SST batches

Signed-off-by: WenyXu <[email protected]>

* perf(mito2): stream and prune manifest SST metadata

Signed-off-by: WenyXu <[email protected]>

* perf(datanode): stream inspection queries through reusable providers

Signed-off-by: WenyXu <[email protected]>

* perf(catalog): push manifest filters through inspection queries

Signed-off-by: WenyXu <[email protected]>

* refactor: align manifest column constants with module conventions

Signed-off-by: WenyXu <[email protected]>

* perf: reduce manifest inspection allocations

Signed-off-by: WenyXu <[email protected]>

* refactor: narrow manifest inspection API visibility

Signed-off-by: WenyXu <[email protected]>

* refactor: retain residual filters for manifest inspection

Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit c96cdd0)
Signed-off-by: WenyXu <[email protected]>

* feat: improve SQL analyze scan diagnostics (#9565)

* feat: improve SQL analyze scan diagnostics

Signed-off-by: evenyag <[email protected]>

* test: redact dynamic filters in sqlness results

Signed-off-by: evenyag <[email protected]>

* test: stabilize analyze snapshots for merge queue

Signed-off-by: evenyag <[email protected]>

* test: remove obsolete scan_details assertion

Signed-off-by: evenyag <[email protected]>

* test: refresh explain snapshots after partition filter changes

Signed-off-by: evenyag <[email protected]>

---------

Signed-off-by: evenyag <[email protected]>
(cherry picked from commit 3e2751d)
Signed-off-by: WenyXu <[email protected]>

* fix(query): avoid lossy casts in partition pruning (#9600)

* fix(query): keep lossy casts out of partition pruning

Signed-off-by: discord9 <[email protected]>

* test(query): align cast pruning expectations with conservative fallback

Signed-off-by: discord9 <[email protected]>

* test(query): record lossy cast partition pruning regression

Signed-off-by: discord9 <[email protected]>

* fix(query): retain safe pruning conjuncts beside unsupported casts

Signed-off-by: discord9 <[email protected]>

* test(query): refresh cast pruning plans after rebase

Signed-off-by: discord9 <[email protected]>

---------

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit df8f684)
Signed-off-by: WenyXu <[email protected]>

* fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)

* fix(query): skip LastRow pushdown for IGNORE NULLS

Signed-off-by: discord9 <[email protected]>

* fix(query): keep IGNORE NULLS aggregates above remote scans

Signed-off-by: discord9 <[email protected]>

* test(query): record IGNORE NULLS results before and after flush

Signed-off-by: discord9 <[email protected]>

* test(query): share IGNORE NULLS coverage across deployments

Signed-off-by: discord9 <[email protected]>

* test(query): refresh last-value plans after rebase

Signed-off-by: discord9 <[email protected]>

---------

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit 0dab3ae)
Signed-off-by: WenyXu <[email protected]>

* fix(meta): share peer discovery with plugin hooks (#9977)

Signed-off-by: WenyXu <[email protected]>
(cherry picked from commit 9f0e58e)
Signed-off-by: WenyXu <[email protected]>

* fix: adapt release backport test dependencies

Copy the writer observation test utility from #9505 (18726c7) for the #9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.

Signed-off-by: WenyXu <[email protected]>

* chore: bump version to 1.3.0

Signed-off-by: WenyXu <[email protected]>

* fix(query): pin DataFusion cast constraint propagation fix

Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).

CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.

The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.

Cargo.lock only rewrites the 35 datafusion source lines.

Signed-off-by: discord9 <[email protected]>
(cherry picked from commit fe637b0)
Signed-off-by: WenyXu <[email protected]>

---------

Signed-off-by: fengyue-xve <[email protected]>
Signed-off-by: 风月 <[email protected]>
Signed-off-by: WenyXu <[email protected]>
Signed-off-by: Dennis Zhuang <[email protected]>
Signed-off-by: Lei, HUANG <[email protected]>
Signed-off-by: fys <[email protected]>
Signed-off-by: paomian <[email protected]>
Signed-off-by: jeremyhi <[email protected]>
Signed-off-by: Aarav <[email protected]>
Signed-off-by: Aumnertic <[email protected]>
Signed-off-by: evenyag <[email protected]>
Signed-off-by: discord9 <[email protected]>
Signed-off-by: discord9 <[email protected]>
Co-authored-by: fengyue-xve <[email protected]>
Co-authored-by: dennis zhuang <[email protected]>
Co-authored-by: Lei, HUANG <[email protected]>
Co-authored-by: fys <[email protected]>
Co-authored-by: localhost <[email protected]>
Co-authored-by: jeremyhi <[email protected]>
Co-authored-by: DeviousCardi <[email protected]>
Co-authored-by: Aum <[email protected]>
Co-authored-by: Yingwen <[email protected]>
Co-authored-by: discord9 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci docs-not-required This change does not impact docs. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants