Skip to content

Commit 899c581

Browse files
csun5285claude
andcommitted
[fix](thrift) Align TQueryOptions field ids with branch-4.2
### What problem does this PR solve? Issue Number: None Related PR: #68314, #68562 Problem Summary: branch-4.1 and branch-4.2 assigned their own ids to TQueryOptions fields that were picked from master, so the same option has different ids on the two lines and the same id means different options: master branch-4.2 183 enable_use_hybrid_sort 183 enable_aggregate_function_null_v2 189 enable_aggregate_function_null_v2 - 210 max_scan_mem_ratio 210 enable_use_hybrid_sort 213 file_presigned_url_ttl_seconds 213 max_scan_mem_ratio 219 enable_segment_limit_pushdown 219 ann_index_candidate_rows_threshold 220 enable_ann_index_result_cache 220 ann_index_candidate_rows_percent_threshold 221 ann_index_candidate_rows_threshold - 222 ann_index_candidate_rows_percent_threshold - 226 enable_prune_nested_column 226 enable_local_exchange_before_streaming_agg 227 new_version_bitmap_op_count 227 file_presigned_url_ttl_seconds 228 enable_local_exchange_before_streaming_agg 228 enable_hyperscan_fallback 229 coordinator_thrift_max_message_size 229 enable_paimon_rust_reader 230 supports_external_file_report_ack 230 enable_prune_nested_column 231 enable_hyperscan_fallback - Thrift skips a field whose wire type differs from the receiver's definition, but a bool that lands on another bool is read as that other option. During a rolling upgrade from a 4.2 release to a release cut from master, a 4.2 FE talking to a new BE would turn enable_aggregate_function_null_v2 into enable_use_hybrid_sort and read enable_hyperscan_fallback as enable_local_exchange_before_streaming_agg, with no error anywhere. This renumbers master to the branch-4.2 ids for every shared field and moves the master-only fields that sat on 4.2 ids to 233-236. 229 gets a placeholder enable_paimon_rust_reader, the switch branch-4.1 and branch-4.2 carry at that id, so the id cannot be handed to another field here; FE never sets it. The declaration order is also changed to match branch-4.1/4.2, with the master-only fields last. FE persists TQueryOptions into its image through Gson together with the thrift isset bit vector, whose bit index is the position of each primitive field in the struct, not its id; with the old order a 4.2 image with a pending schema change job replayed on master with 34 shifted bits. Same ids and same order means the same bits. FE and BE refer to these fields by name, so no code changes. master has no release that uses the old ids; a mixed-version master dev cluster must upgrade FE and BE together across this change. To keep this from repeating, `gensrc/AGENTS.md` states the one rule behind the ids (every definition means the same thing on both sides of every upgrade step, 3.1 → 4.0 → 4.1 → 4.2 → master, and on disk) and how to review a change against it branch by branch; the root AGENTS.md points to it, and the code-review skill gets a matching checkpoint, a module-guide row and a protocol-compatibility item. ### Release note None ### Check List (For Author) - Test: Manual test - thrift --gen cpp on PaloInternalService.thrift succeeds; a scan of all gensrc thrift and proto definitions reports no TQueryOptions id or name mismatch between this change and branch-4.2. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 1cc0456 commit 899c581

4 files changed

Lines changed: 81 additions & 29 deletions

File tree

‎.claude/skills/code-review/SKILL.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ The following checkpoints must be **individually confirmed with conclusions** du
8787
- Can affected processes detect the change promptly without restart?
8888
- Does it involve incompatible changes like function symbols or storage formats? If yes:
8989
- Is compatibility code added? Can it correctly handle requests during rolling upgrades?
90+
- Does it change `gensrc/thrift` or `gensrc/proto`, or how FE/BE handle an unset thrift/proto field? If yes: read every touched definition, and every definition using a typedef, struct or enum it touches, on HEAD and on every release branch in the upgrade chain and compare id by id. A finding is the same id or enum value meaning something else on any branch, a reused id, a one-sided field that is not `optional`, a renamed or removed method an older side still calls, a stored definition whose names or primitive-field order differ from the latest release, or an unset handling that changes without keeping the older side's behaviour. (See `gensrc/AGENTS.md`)
9091
- Are there functionally parallel code paths to the modified one? If yes:
9192
- Should this modification be applied to other paths, and has it been?
9293
- Are there special conditional checks? If yes:
@@ -205,6 +206,12 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
205206
| `cloud/src/meta-service/AGENTS.md` | Cloud transaction commit paths, RPC retry contract, Cloud MoW contract |
206207
| `cloud/src/recycler/AGENTS.md` | Recycler safety, two-phase delete, packed-file ordering |
207208

209+
### Shared Definitions
210+
211+
| Directory | Coverage |
212+
|-----------|----------|
213+
| `gensrc/AGENTS.md` | Thrift / protobuf definitions: same meaning for every id, enum value and method across the upgrade chain, on stored data and for external IDL copies |
214+
208215
---
209216

210217
## Part 3: Cross-Module Concerns
@@ -214,6 +221,7 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
214221
- [ ] Do new `TPlanNodeType` values have matching BE handling?
215222
- [ ] Are all FE-to-BE send paths updated when new transmitted variables are added?
216223
- [ ] Is mixed-version compatibility preserved for serialization or function metadata changes?
224+
- [ ] For `gensrc` changes: was every touched definition compared against every release branch, with no id or enum value meaning something else, no reused id, no new `required` field, and no stored definition whose names or primitive-field order differ from the latest release?
217225

218226
### 3.2 Cloud Mode vs Shared-Nothing Mode
219227

‎AGENTS.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ Every consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc /
1818

1919
Separately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.
2020

21+
## Thrift and Protobuf Field Ids
22+
23+
Every definition under `gensrc/thrift` and `gensrc/proto` must mean the same thing on both sides of every upgrade step (branch-3.1 → 4.0 → 4.1 → 4.2 → master), on stored data for every release that can still read it, and forever for the IDL copies compiled into UDF servers, connectors, brokers and ccr-syncer. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.
24+
2125
## When running in a WORKTREE directory
2226

2327
To ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.

‎gensrc/AGENTS.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# AGENTS.md — gensrc (thrift / protobuf)
2+
3+
Everything under `gensrc/thrift` and `gensrc/proto` is a wire format between FE, BE and the cloud services, FE↔FE and BE↔BE included during a rolling upgrade. The wire carries field ids, enum values and service method names, not field or enum names. A field that arrives on an id of the same wire type is read as that field, so same wire type is not enough: thrift container element types and proto integer width and signedness (int32, int64, uint32, bool, enum and sint* are all varints) must match too. A missing or mistyped `required` field fails the whole read. An enum value the receiver does not know is a raw integer in thrift C++ but unset, so the default, in thrift Java and proto2.
4+
5+
Some definitions are also stored, or compiled into software outside this repo:
6+
7+
- Stored: `olap_file.proto` and its imports `olap_common.proto` / `types.proto`, `segment_v2.proto`, `snii.proto`, `file_cache.proto`, `cloud.proto` (meta-service FoundationDB), and the thrift structs and enums FE writes into its image (`TBinlog` in binary; `TQueryOptions`, `TStorageMedium` and others through Gson, which stores field names, enum names and the position of every primitive field).
8+
- External: `function_service.proto` and `types.proto` (remote UDF servers), `DorisExternalService.thrift` (connectors), `PaloBrokerService.thrift` (brokers), the ccr methods of `FrontendService.thrift` and `BackendService.thrift` (ccr-syncer). `parquet.thrift` is Apache Parquet's file; change it only by copying an upstream release.
9+
10+
## The rule
11+
12+
Every definition must mean the same thing on both sides of every upgrade step (branch-3.1 → 4.0 → 4.1 → 4.2 → master; 3.0 and 2.1 no longer take gensrc changes), for stored data on every release that can still read it, and for external consumers forever. That means:
13+
14+
- An id or enum value on both sides is the same field with the same declared type, requiredness and meaning, and the receiver does the same thing when the sender leaves it unset (the declared default or the `__isset` / `has_` fallback).
15+
- An id or value on one side only is `optional` and unused on every other branch in the chain, not just the adjacent one. A dropped field keeps its id (`// Deprecated`, a commented-out line, proto `reserved`); `deprecated` is not permission to reuse. A new enum value reads as the default on older peers: add one only where that is acceptable, or map it for them as `MetaServiceCode` does.
16+
- A method is renamed or removed only when no caller on the other side of any step still uses it.
17+
- A stored definition keeps the field names, enum names and primitive-field order that the latest release has; they are on disk. Master follows the release, never the reverse.
18+
- An id that no release carries may still be renumbered on master; say so in the PR, since mixed master builds break.
19+
20+
## Review
21+
22+
For every definition the PR touches, and every definition that uses a typedef, struct or enum it touches, read it on HEAD and on every branch in the chain (`git show <remote>/branch-4.2:<path>` with the `.thrift` or `.proto` path, after fetching the remote that points to apache/doris) and compare id by id and value by value, counting commented-out and `reserved` ids as taken. A finding is: the same id or value meaning something else on any branch; an id or value reused; a one-sided field that is not `optional`; a renamed or removed method an older side still calls; a stored definition whose names or primitive-field order differ from the latest release that has it; an unset handling that changes, unless it keeps the older side's behaviour or refuses on purpose as a version gate and the PR says which ("every sender sets it" is no answer: the older side and stored data do not). A conflict that already exists between branches is a finding when the PR touches that definition; otherwise note it.

‎gensrc/thrift/PaloInternalService.thrift‎

Lines changed: 47 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -417,28 +417,22 @@ struct TQueryOptions {
417417
// Target file size in bytes for Iceberg write operations
418418
// Default 0 means use config::iceberg_sink_max_file_size
419419
178: optional i64 iceberg_write_target_file_size_bytes = 0;
420-
179: optional bool enable_parquet_filter_by_bloom_filter = true;
421420
180: optional i32 max_file_scanners_concurrency = 0;
422421
181: optional i32 min_file_scanners_concurrency = 0;
423422
182: optional i32 ivf_nprobe = 32;
424-
// Enable hybrid sorting: dynamically selects between PdqSort and TimSort based on
425-
// runtime profiling to choose the most efficient algorithm for the data pattern
426-
183: optional bool enable_use_hybrid_sort = false;
427-
184: optional i32 cte_max_recursion_depth;
428-
429-
185: optional bool enable_parquet_file_page_cache = true;
423+
179: optional bool enable_parquet_filter_by_bloom_filter = true;
424+
183: optional bool enable_aggregate_function_null_v2 = false;
430425

431426
186: optional bool enable_streaming_agg_hash_join_force_passthrough;
432427

433428
187: optional bool enable_distinct_streaming_agg_force_passthrough;
434429

435430
188: optional bool enable_broadcast_join_force_passthrough;
436431

437-
189: optional bool enable_aggregate_function_null_v2 = false;
438-
439432
195: optional bool enable_left_semi_direct_return_opt;
440433

441434
200: optional bool enable_adjust_conjunct_order_by_cost;
435+
442436
// Deprecated: the paimon-cpp reader has been removed. Retained for wire compatibility.
443437
201: optional bool enable_paimon_cpp_reader = false;
444438

@@ -447,6 +441,8 @@ struct TQueryOptions {
447441
// hash table expansion thresholds since all data is local.
448442
202: optional bool single_backend_query = false;
449443

444+
185: optional bool enable_parquet_file_page_cache = true;
445+
450446
203: optional bool enable_inverted_index_wand_query = true;
451447

452448
// Per-read/per-write buffer size used during spill I/O, in bytes. Controls the
@@ -459,7 +455,9 @@ struct TQueryOptions {
459455
// memory exceeds the corresponding threshold, it proactively spills to disk.
460456
// Default is 64MB for all three.
461457
205: optional i64 spill_join_build_sink_mem_limit_bytes = 67108864
458+
462459
206: optional i64 spill_aggregation_sink_mem_limit_bytes = 67108864
460+
463461
207: optional i64 spill_sort_sink_mem_limit_bytes = 67108864
464462

465463
// Total memory budget for the sort merge phase after spill. Divided by
@@ -473,52 +471,54 @@ struct TQueryOptions {
473471
// session variable `spill_repartition_max_depth` in FE. Default is 8.
474472
209: optional i32 spill_repartition_max_depth = 8
475473

476-
210: optional double max_scan_mem_ratio = 0.3;
474+
184: optional i32 cte_max_recursion_depth;
475+
476+
// Enable hybrid sorting: dynamically selects between PdqSort and TimSort based on
477+
// runtime profiling to choose the most efficient algorithm for the data pattern
478+
210: optional bool enable_use_hybrid_sort = false;
479+
477480
211: optional bool enable_adaptive_scan = false;
478481

479482
212: optional bool enable_local_exchange_before_agg = true;
480483

481-
213: optional i64 file_presigned_url_ttl_seconds = 3600;
484+
213: optional double max_scan_mem_ratio = 0.3;
485+
482486
214: optional i32 embed_max_batch_size = 5;
487+
483488
215: optional i64 ai_context_window_size = 131072;
484489

485490
// Use Rust-based Lance reader for FORMAT_LANCE scan ranges
486491
216: optional bool enable_rust_lance_reader = false; // deprecated
492+
487493
217: optional bool new_version_percentile = false
488494

489495
// Adaptive batch size: target output block size in bytes. Valid range [1MB, 512MB].
490496
// Default 8MB. Sent by FE session variable preferred_block_size_bytes.
491497
218: optional i64 preferred_block_size_bytes = 8388608
492498

493-
// Push LIMIT into SegmentIterator when safe.
494-
219: optional bool enable_segment_limit_pushdown = true
495-
496-
220: optional bool enable_ann_index_result_cache = true
497499
// ANN search falls back to exact vector distance evaluation when candidate rows
498500
// before ANN search are less than this value. 0 disables the absolute threshold.
499-
221: optional i64 ann_index_candidate_rows_threshold = 0
500-
// Candidate row ratio threshold against segment rows. Existing default is 0.3.
501-
222: optional double ann_index_candidate_rows_percent_threshold = 0.3
501+
219: optional i64 ann_index_candidate_rows_threshold = 0
502502

503-
// enable plan local exchange node in fe
504-
223: optional bool enable_local_shuffle_planner;
503+
// Candidate row ratio threshold against segment rows. Existing default is 0.3.
504+
220: optional double ann_index_candidate_rows_percent_threshold = 0.3
505505

506506
// Controls expression-based ZoneMap pruning for readers that honor this option.
507507
// FileScannerV2 always enables safe expression ZoneMap pruning.
508508
224: optional bool enable_expr_zonemap_filter = true
509509

510510
225: optional i64 runtime_filter_tree_publish_max_send_bytes = 268435456
511511

512-
226: optional bool enable_prune_nested_column = false;
513-
227: optional bool new_version_bitmap_op_count = false;
514-
228: optional bool enable_local_exchange_before_streaming_agg = false;
515-
// FE is the receiver of fragment reports, so BE must also honor its message limit.
516-
229: optional i32 coordinator_thrift_max_message_size;
517-
// FE can explicitly and idempotently acknowledge external-file commit reports.
518-
230: optional bool supports_external_file_report_ack = false;
512+
226: optional bool enable_local_exchange_before_streaming_agg = false;
513+
514+
227: optional i64 file_presigned_url_ttl_seconds = 3600;
515+
519516
// Fall back to RE2 when Hyperscan cannot compile a regular expression.
520-
231: optional bool enable_hyperscan_fallback = true;
521-
232: optional bool enable_runtime_filter_bucket_prune = true;
517+
228: optional bool enable_hyperscan_fallback = true;
518+
229: optional bool enable_paimon_rust_reader = false;
519+
520+
230: optional bool enable_prune_nested_column = false;
521+
522522
// For cloud, to control if the content would be written into file cache
523523
// In write path, to control if the content would be written into file cache.
524524
// In read path, read from file cache or remote storage when execute query.
@@ -540,6 +540,24 @@ struct TQueryOptions {
540540
// reads the data instead.
541541
// Defaults to false because an old FE never sends this field, and BE checked both cases before.
542542
1006: optional bool force_pushdown_zonemap_minmax = false
543+
544+
// enable plan local exchange node in fe
545+
223: optional bool enable_local_shuffle_planner;
546+
547+
// Push LIMIT into SegmentIterator when safe.
548+
231: optional bool enable_segment_limit_pushdown = true
549+
550+
232: optional bool enable_runtime_filter_bucket_prune = true;
551+
552+
233: optional bool enable_ann_index_result_cache = true
553+
554+
234: optional bool new_version_bitmap_op_count = false;
555+
556+
// FE is the receiver of fragment reports, so BE must also honor its message limit.
557+
235: optional i32 coordinator_thrift_max_message_size;
558+
559+
// FE can explicitly and idempotently acknowledge external-file commit reports.
560+
236: optional bool supports_external_file_report_ack = false;
543561
}
544562

545563

0 commit comments

Comments
 (0)