You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 899c581
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](thrift) Align TQueryOptions field ids with branch-4.2
### What problem does this PR solve?
Issue Number: None
Related PR: #68314, #68562
Problem Summary:
branch-4.1 and branch-4.2 assigned their own ids to TQueryOptions fields that were
picked from master, so the same option has different ids on the two lines and the same
id means different options:
master branch-4.2
183 enable_use_hybrid_sort 183 enable_aggregate_function_null_v2
189 enable_aggregate_function_null_v2 -
210 max_scan_mem_ratio 210 enable_use_hybrid_sort
213 file_presigned_url_ttl_seconds 213 max_scan_mem_ratio
219 enable_segment_limit_pushdown 219 ann_index_candidate_rows_threshold
220 enable_ann_index_result_cache 220 ann_index_candidate_rows_percent_threshold
221 ann_index_candidate_rows_threshold -
222 ann_index_candidate_rows_percent_threshold -
226 enable_prune_nested_column 226 enable_local_exchange_before_streaming_agg
227 new_version_bitmap_op_count 227 file_presigned_url_ttl_seconds
228 enable_local_exchange_before_streaming_agg 228 enable_hyperscan_fallback
229 coordinator_thrift_max_message_size 229 enable_paimon_rust_reader
230 supports_external_file_report_ack 230 enable_prune_nested_column
231 enable_hyperscan_fallback -
Thrift skips a field whose wire type differs from the receiver's definition, but a
bool that lands on another bool is read as that other option. During a rolling upgrade
from a 4.2 release to a release cut from master, a 4.2 FE talking to a new BE would turn
enable_aggregate_function_null_v2 into enable_use_hybrid_sort and read
enable_hyperscan_fallback as enable_local_exchange_before_streaming_agg, with no error
anywhere.
This renumbers master to the branch-4.2 ids for every shared field and moves the
master-only fields that sat on 4.2 ids to 233-236. 229 gets a placeholder
enable_paimon_rust_reader, the switch branch-4.1 and branch-4.2 carry at that id, so the id
cannot be handed to another field here; FE never sets it.
The declaration order is also changed to match branch-4.1/4.2, with the master-only
fields last. FE persists TQueryOptions into its image through Gson together with the
thrift isset bit vector, whose bit index is the position of each primitive field in the
struct, not its id; with the old order a 4.2 image with a pending schema change job
replayed on master with 34 shifted bits. Same ids and same order means the same bits.
FE and BE refer to these fields by name, so no code changes. master has no release that
uses the old ids; a mixed-version master dev cluster must upgrade FE and BE together
across this change.
To keep this from repeating, `gensrc/AGENTS.md` states the one rule behind the ids (every
definition means the same thing on both sides of every upgrade step, 3.1 → 4.0 → 4.1 → 4.2 →
master, and on disk) and how to review a change against it branch by branch; the root AGENTS.md
points to it, and the code-review skill gets a matching checkpoint, a module-guide row and a
protocol-compatibility item.
### Release note
None
### Check List (For Author)
- Test: Manual test
- thrift --gen cpp on PaloInternalService.thrift succeeds; a scan of all gensrc thrift and proto definitions reports no TQueryOptions id or name mismatch between this change and branch-4.2.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: .claude/skills/code-review/SKILL.md
+8Lines changed: 8 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -87,6 +87,7 @@ The following checkpoints must be **individually confirmed with conclusions** du
87
87
- Can affected processes detect the change promptly without restart?
88
88
- Does it involve incompatible changes like function symbols or storage formats? If yes:
89
89
- Is compatibility code added? Can it correctly handle requests during rolling upgrades?
90
+
- Does it change `gensrc/thrift` or `gensrc/proto`, or how FE/BE handle an unset thrift/proto field? If yes: read every touched definition, and every definition using a typedef, struct or enum it touches, on HEAD and on every release branch in the upgrade chain and compare id by id. A finding is the same id or enum value meaning something else on any branch, a reused id, a one-sided field that is not `optional`, a renamed or removed method an older side still calls, a stored definition whose names or primitive-field order differ from the latest release, or an unset handling that changes without keeping the older side's behaviour. (See `gensrc/AGENTS.md`)
90
91
- Are there functionally parallel code paths to the modified one? If yes:
91
92
- Should this modification be applied to other paths, and has it been?
92
93
- Are there special conditional checks? If yes:
@@ -205,6 +206,12 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
|`gensrc/AGENTS.md`| Thrift / protobuf definitions: same meaning for every id, enum value and method across the upgrade chain, on stored data and for external IDL copies |
214
+
208
215
---
209
216
210
217
## Part 3: Cross-Module Concerns
@@ -214,6 +221,7 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
214
221
-[ ] Do new `TPlanNodeType` values have matching BE handling?
215
222
-[ ] Are all FE-to-BE send paths updated when new transmitted variables are added?
216
223
-[ ] Is mixed-version compatibility preserved for serialization or function metadata changes?
224
+
-[ ] For `gensrc` changes: was every touched definition compared against every release branch, with no id or enum value meaning something else, no reused id, no new `required` field, and no stored definition whose names or primitive-field order differ from the latest release?
Copy file name to clipboardExpand all lines: AGENTS.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,10 @@ Every consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc /
18
18
19
19
Separately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.
20
20
21
+
## Thrift and Protobuf Field Ids
22
+
23
+
Every definition under `gensrc/thrift` and `gensrc/proto` must mean the same thing on both sides of every upgrade step (branch-3.1 → 4.0 → 4.1 → 4.2 → master), on stored data for every release that can still read it, and forever for the IDL copies compiled into UDF servers, connectors, brokers and ccr-syncer. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.
24
+
21
25
## When running in a WORKTREE directory
22
26
23
27
To ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.
Everything under `gensrc/thrift` and `gensrc/proto` is a wire format between FE, BE and the cloud services, FE↔FE and BE↔BE included during a rolling upgrade. The wire carries field ids, enum values and service method names, not field or enum names. A field that arrives on an id of the same wire type is read as that field, so same wire type is not enough: thrift container element types and proto integer width and signedness (int32, int64, uint32, bool, enum and sint* are all varints) must match too. A missing or mistyped `required` field fails the whole read. An enum value the receiver does not know is a raw integer in thrift C++ but unset, so the default, in thrift Java and proto2.
4
+
5
+
Some definitions are also stored, or compiled into software outside this repo:
6
+
7
+
- Stored: `olap_file.proto` and its imports `olap_common.proto` / `types.proto`, `segment_v2.proto`, `snii.proto`, `file_cache.proto`, `cloud.proto` (meta-service FoundationDB), and the thrift structs and enums FE writes into its image (`TBinlog` in binary; `TQueryOptions`, `TStorageMedium` and others through Gson, which stores field names, enum names and the position of every primitive field).
8
+
- External: `function_service.proto` and `types.proto` (remote UDF servers), `DorisExternalService.thrift` (connectors), `PaloBrokerService.thrift` (brokers), the ccr methods of `FrontendService.thrift` and `BackendService.thrift` (ccr-syncer). `parquet.thrift` is Apache Parquet's file; change it only by copying an upstream release.
9
+
10
+
## The rule
11
+
12
+
Every definition must mean the same thing on both sides of every upgrade step (branch-3.1 → 4.0 → 4.1 → 4.2 → master; 3.0 and 2.1 no longer take gensrc changes), for stored data on every release that can still read it, and for external consumers forever. That means:
13
+
14
+
- An id or enum value on both sides is the same field with the same declared type, requiredness and meaning, and the receiver does the same thing when the sender leaves it unset (the declared default or the `__isset` / `has_` fallback).
15
+
- An id or value on one side only is `optional` and unused on every other branch in the chain, not just the adjacent one. A dropped field keeps its id (`// Deprecated`, a commented-out line, proto `reserved`); `deprecated` is not permission to reuse. A new enum value reads as the default on older peers: add one only where that is acceptable, or map it for them as `MetaServiceCode` does.
16
+
- A method is renamed or removed only when no caller on the other side of any step still uses it.
17
+
- A stored definition keeps the field names, enum names and primitive-field order that the latest release has; they are on disk. Master follows the release, never the reverse.
18
+
- An id that no release carries may still be renumbered on master; say so in the PR, since mixed master builds break.
19
+
20
+
## Review
21
+
22
+
For every definition the PR touches, and every definition that uses a typedef, struct or enum it touches, read it on HEAD and on every branch in the chain (`git show <remote>/branch-4.2:<path>` with the `.thrift` or `.proto` path, after fetching the remote that points to apache/doris) and compare id by id and value by value, counting commented-out and `reserved` ids as taken. A finding is: the same id or value meaning something else on any branch; an id or value reused; a one-sided field that is not `optional`; a renamed or removed method an older side still calls; a stored definition whose names or primitive-field order differ from the latest release that has it; an unset handling that changes, unless it keeps the older side's behaviour or refuses on purpose as a version gate and the PR says which ("every sender sets it" is no answer: the older side and stored data do not). A conflict that already exists between branches is a finding when the PR touches that definition; otherwise note it.
0 commit comments