Skip to content

Commit 3dced98

Browse files
csun5285claude
andcommitted
[fix](thrift) Align TQueryOptions field ids with branch-4.2
### What problem does this PR solve? Issue Number: None Related PR: #68314, #68562 Problem Summary: branch-4.1 and branch-4.2 assigned their own ids to TQueryOptions fields that were picked from master, so the same option has different ids on the two lines and the same id means different options: master branch-4.2 183 enable_use_hybrid_sort 183 enable_aggregate_function_null_v2 189 enable_aggregate_function_null_v2 - 210 max_scan_mem_ratio 210 enable_use_hybrid_sort 213 file_presigned_url_ttl_seconds 213 max_scan_mem_ratio 219 enable_segment_limit_pushdown 219 ann_index_candidate_rows_threshold 220 enable_ann_index_result_cache 220 ann_index_candidate_rows_percent_threshold 221 ann_index_candidate_rows_threshold - 222 ann_index_candidate_rows_percent_threshold - 226 enable_prune_nested_column 226 enable_local_exchange_before_streaming_agg 227 new_version_bitmap_op_count 227 file_presigned_url_ttl_seconds 228 enable_local_exchange_before_streaming_agg 228 enable_hyperscan_fallback 229 coordinator_thrift_max_message_size 229 enable_paimon_rust_reader 230 supports_external_file_report_ack 230 enable_prune_nested_column 231 enable_hyperscan_fallback - Thrift skips a field whose wire type differs from the receiver's definition, but a bool that lands on another bool is read as that other option. During a rolling upgrade from a 4.2 release to a release cut from master, a 4.2 FE talking to a new BE would turn enable_aggregate_function_null_v2 into enable_use_hybrid_sort and read enable_hyperscan_fallback as enable_local_exchange_before_streaming_agg, with no error anywhere. This renumbers master to the branch-4.2 ids for every shared field and moves the master-only fields that sat on 4.2 ids to 233-236. 229 gets a placeholder enable_paimon_rust_reader, the switch branch-4.1 and branch-4.2 carry at that id, so the id cannot be handed to another field here; FE never sets it. The fields are reordered so ids ascend again. FE and BE refer to these fields by name, so no code changes. master has no release that uses the old ids; a mixed-version master dev cluster must upgrade FE and BE together across this change. TQueryOptions also carries an older collision that no master change can fix: branch-3.1 and branch-4.0 both shipped ids 119-121 as keep_carriage_return, enable_match_without_inverted_index and enable_fallback_on_missing_inverted_index, in a different order on each branch, so a 3.1 FE talking to a 4.0 BE during that upgrade sends enable_fallback_on_missing_inverted_index on the id the BE reads as keep_carriage_return. To keep this from repeating, `gensrc/AGENTS.md` states the one rule behind the ids (every definition means the same thing on both sides of every upgrade step, 3.1 → 4.0 → 4.1 → 4.2 → master, and on disk) and how to review a change against it branch by branch; the root AGENTS.md points to it, and the code-review skill gets a matching checkpoint, a module-guide row and a protocol-compatibility item. ### Release note None ### Check List (For Author) - Test: Manual test - thrift --gen cpp on PaloInternalService.thrift succeeds; a scan of all gensrc thrift and proto definitions reports no TQueryOptions id or name mismatch between this change and branch-4.2. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 1cc0456 commit 3dced98

4 files changed

Lines changed: 57 additions & 21 deletions

File tree

‎.claude/skills/code-review/SKILL.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ The following checkpoints must be **individually confirmed with conclusions** du
8787
- Can affected processes detect the change promptly without restart?
8888
- Does it involve incompatible changes like function symbols or storage formats? If yes:
8989
- Is compatibility code added? Can it correctly handle requests during rolling upgrades?
90+
- Does it change `gensrc/thrift` or `gensrc/proto`? If yes: read every touched definition on HEAD and on each release branch and compare. An id or enum value that means something else on another branch, a reused id, or a new required field is a finding. (See `gensrc/AGENTS.md`)
9091
- Are there functionally parallel code paths to the modified one? If yes:
9192
- Should this modification be applied to other paths, and has it been?
9293
- Are there special conditional checks? If yes:
@@ -205,6 +206,12 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
205206
| `cloud/src/meta-service/AGENTS.md` | Cloud transaction commit paths, RPC retry contract, Cloud MoW contract |
206207
| `cloud/src/recycler/AGENTS.md` | Recycler safety, two-phase delete, packed-file ordering |
207208

209+
### Shared Definitions
210+
211+
| Directory | Coverage |
212+
|-----------|----------|
213+
| `gensrc/AGENTS.md` | Thrift / protobuf ids and enum values: same meaning on every branch, never reused, new fields optional |
214+
208215
---
209216

210217
## Part 3: Cross-Module Concerns
@@ -214,6 +221,7 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
214221
- [ ] Do new `TPlanNodeType` values have matching BE handling?
215222
- [ ] Are all FE-to-BE send paths updated when new transmitted variables are added?
216223
- [ ] Is mixed-version compatibility preserved for serialization or function metadata changes?
224+
- [ ] For `gensrc` changes: does every id and enum value mean the same thing on every release branch, with no reused id and no new required field?
217225

218226
### 3.2 Cloud Mode vs Shared-Nothing Mode
219227

‎AGENTS.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ Every consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc /
1818

1919
Separately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.
2020

21+
## Thrift and Protobuf Field Ids
22+
23+
Every change under `gensrc/thrift` and `gensrc/proto` must keep every upgrade compatible (branch-3.1 → 4.0 → 4.1 → 4.2 → master): a field id or enum value means the same thing on every branch, an id is never reused, and a new field is optional. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.
24+
2125
## When running in a WORKTREE directory
2226

2327
To ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.

‎gensrc/AGENTS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# AGENTS.md — gensrc (thrift / protobuf)
2+
3+
Everything under `gensrc/thrift` and `gensrc/proto` is shared by every release, on the wire and on disk. One rule: a change here must keep every upgrade compatible, 3.1 → 4.0 → 4.1 → 4.2 → master. A field id or enum value means the same thing on every branch that has it, an id is never reused, and a field that only one side has is optional.
4+
5+
## Review
6+
7+
Read every touched definition on HEAD and on each release branch that still takes picks (today branch-4.2, branch-4.1, branch-4.0 and branch-3.1; add a new branch when it is cut) with `git show <remote>/branch-4.2:<path>`. An id or enum value that means something else on another branch, a reused id, or a new required field is a finding. A collision that already exists between two shipped releases cannot be fixed by a PR; note it.

‎gensrc/thrift/PaloInternalService.thrift‎

Lines changed: 38 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -421,9 +421,8 @@ struct TQueryOptions {
421421
180: optional i32 max_file_scanners_concurrency = 0;
422422
181: optional i32 min_file_scanners_concurrency = 0;
423423
182: optional i32 ivf_nprobe = 32;
424-
// Enable hybrid sorting: dynamically selects between PdqSort and TimSort based on
425-
// runtime profiling to choose the most efficient algorithm for the data pattern
426-
183: optional bool enable_use_hybrid_sort = false;
424+
183: optional bool enable_aggregate_function_null_v2 = false;
425+
427426
184: optional i32 cte_max_recursion_depth;
428427

429428
185: optional bool enable_parquet_file_page_cache = true;
@@ -434,11 +433,10 @@ struct TQueryOptions {
434433

435434
188: optional bool enable_broadcast_join_force_passthrough;
436435

437-
189: optional bool enable_aggregate_function_null_v2 = false;
438-
439436
195: optional bool enable_left_semi_direct_return_opt;
440437

441438
200: optional bool enable_adjust_conjunct_order_by_cost;
439+
442440
// Deprecated: the paimon-cpp reader has been removed. Retained for wire compatibility.
443441
201: optional bool enable_paimon_cpp_reader = false;
444442

@@ -459,7 +457,9 @@ struct TQueryOptions {
459457
// memory exceeds the corresponding threshold, it proactively spills to disk.
460458
// Default is 64MB for all three.
461459
205: optional i64 spill_join_build_sink_mem_limit_bytes = 67108864
460+
462461
206: optional i64 spill_aggregation_sink_mem_limit_bytes = 67108864
462+
463463
207: optional i64 spill_sort_sink_mem_limit_bytes = 67108864
464464

465465
// Total memory budget for the sort merge phase after spill. Divided by
@@ -473,32 +473,35 @@ struct TQueryOptions {
473473
// session variable `spill_repartition_max_depth` in FE. Default is 8.
474474
209: optional i32 spill_repartition_max_depth = 8
475475

476-
210: optional double max_scan_mem_ratio = 0.3;
476+
// Enable hybrid sorting: dynamically selects between PdqSort and TimSort based on
477+
// runtime profiling to choose the most efficient algorithm for the data pattern
478+
210: optional bool enable_use_hybrid_sort = false;
479+
477480
211: optional bool enable_adaptive_scan = false;
478481

479482
212: optional bool enable_local_exchange_before_agg = true;
480483

481-
213: optional i64 file_presigned_url_ttl_seconds = 3600;
484+
213: optional double max_scan_mem_ratio = 0.3;
485+
482486
214: optional i32 embed_max_batch_size = 5;
487+
483488
215: optional i64 ai_context_window_size = 131072;
484489

485490
// Use Rust-based Lance reader for FORMAT_LANCE scan ranges
486491
216: optional bool enable_rust_lance_reader = false; // deprecated
492+
487493
217: optional bool new_version_percentile = false
488494

489495
// Adaptive batch size: target output block size in bytes. Valid range [1MB, 512MB].
490496
// Default 8MB. Sent by FE session variable preferred_block_size_bytes.
491497
218: optional i64 preferred_block_size_bytes = 8388608
492498

493-
// Push LIMIT into SegmentIterator when safe.
494-
219: optional bool enable_segment_limit_pushdown = true
495-
496-
220: optional bool enable_ann_index_result_cache = true
497499
// ANN search falls back to exact vector distance evaluation when candidate rows
498500
// before ANN search are less than this value. 0 disables the absolute threshold.
499-
221: optional i64 ann_index_candidate_rows_threshold = 0
501+
219: optional i64 ann_index_candidate_rows_threshold = 0
502+
500503
// Candidate row ratio threshold against segment rows. Existing default is 0.3.
501-
222: optional double ann_index_candidate_rows_percent_threshold = 0.3
504+
220: optional double ann_index_candidate_rows_percent_threshold = 0.3
502505

503506
// enable plan local exchange node in fe
504507
223: optional bool enable_local_shuffle_planner;
@@ -509,16 +512,30 @@ struct TQueryOptions {
509512

510513
225: optional i64 runtime_filter_tree_publish_max_send_bytes = 268435456
511514

512-
226: optional bool enable_prune_nested_column = false;
513-
227: optional bool new_version_bitmap_op_count = false;
514-
228: optional bool enable_local_exchange_before_streaming_agg = false;
515-
// FE is the receiver of fragment reports, so BE must also honor its message limit.
516-
229: optional i32 coordinator_thrift_max_message_size;
517-
// FE can explicitly and idempotently acknowledge external-file commit reports.
518-
230: optional bool supports_external_file_report_ack = false;
515+
226: optional bool enable_local_exchange_before_streaming_agg = false;
516+
517+
227: optional i64 file_presigned_url_ttl_seconds = 3600;
518+
519519
// Fall back to RE2 when Hyperscan cannot compile a regular expression.
520-
231: optional bool enable_hyperscan_fallback = true;
520+
228: optional bool enable_hyperscan_fallback = true;
521+
229: optional bool enable_paimon_rust_reader = false;
522+
523+
230: optional bool enable_prune_nested_column = false;
524+
525+
// Push LIMIT into SegmentIterator when safe.
526+
231: optional bool enable_segment_limit_pushdown = true
527+
521528
232: optional bool enable_runtime_filter_bucket_prune = true;
529+
530+
233: optional bool enable_ann_index_result_cache = true
531+
532+
234: optional bool new_version_bitmap_op_count = false;
533+
534+
// FE is the receiver of fragment reports, so BE must also honor its message limit.
535+
235: optional i32 coordinator_thrift_max_message_size;
536+
537+
// FE can explicitly and idempotently acknowledge external-file commit reports.
538+
236: optional bool supports_external_file_report_ack = false;
522539
// For cloud, to control if the content would be written into file cache
523540
// In write path, to control if the content would be written into file cache.
524541
// In read path, read from file cache or remote storage when execute query.

0 commit comments

Comments
 (0)