Skip to content

Commit 58c16ac

Browse files
committed
[chore](fe) Sync Flight table types with master
### What problem does this PR solve? Related PR: #67899 Problem Summary: Resolve the schema-helper import conflict while retaining master Arrow type support and the table-types metadata fix. ### Release note None; this commit synchronizes the existing PR with master. ### Check List (For Author) - Test: FE Checkstyle passed with zero violations. Official selected FE unit-test command attempted; blocked before execution by the Intel-only grpc-java 1.34.0 generator on this ARM macOS host. - Behavior changed: No additional behavior beyond the existing PR and master. - Does this need documentation: No. Signed-off-by: hutiefang76 <[email protected]>
2 parents 5fd7d98 + 011cff4 commit 58c16ac

2,370 files changed

Lines changed: 158025 additions & 20300 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.asf.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,27 @@ github:
6868
dismiss_stale_reviews: true
6969
require_code_owner_reviews: true
7070
required_approving_review_count: 1
71+
branch-4.2:
72+
required_status_checks:
73+
strict: false
74+
contexts:
75+
- License Check
76+
- Clang Formatter
77+
- CheckStyle
78+
- Build Broker
79+
- FE UT (Doris FE UT)
80+
- BE UT (Doris BE UT)
81+
- Cloud UT (Doris Cloud UT)
82+
- COMPILE (DORIS_COMPILE)
83+
- P0 Regression (Doris Regression)
84+
- NonConcurrent Regression (nonConcurrent)
85+
- External Regression (Doris External Regression)
86+
- cloud_p0 (Doris Cloud Regression)
87+
- vault_p0 (Doris Cloud Regression)
88+
required_pull_request_reviews:
89+
require_code_owner_reviews: true
90+
required_approving_review_count: 1
91+
dismiss_stale_reviews: true
7192
branch-4.1:
7293
required_status_checks:
7394
strict: false

‎.claude/skills/code-review/SKILL.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ The following checkpoints must be **individually confirmed with conclusions** du
8787
- Can affected processes detect the change promptly without restart?
8888
- Does it involve incompatible changes like function symbols or storage formats? If yes:
8989
- Is compatibility code added? Can it correctly handle requests during rolling upgrades?
90+
- Does it change `gensrc/thrift` or `gensrc/proto`? If yes: read every touched definition on HEAD and on each release branch and compare. An id or enum value that means something else on another branch, a reused id, or a new required field is a finding. (See `gensrc/AGENTS.md`)
9091
- Are there functionally parallel code paths to the modified one? If yes:
9192
- Should this modification be applied to other paths, and has it been?
9293
- Are there special conditional checks? If yes:
@@ -205,6 +206,12 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
205206
| `cloud/src/meta-service/AGENTS.md` | Cloud transaction commit paths, RPC retry contract, Cloud MoW contract |
206207
| `cloud/src/recycler/AGENTS.md` | Recycler safety, two-phase delete, packed-file ordering |
207208

209+
### Shared Definitions
210+
211+
| Directory | Coverage |
212+
|-----------|----------|
213+
| `gensrc/AGENTS.md` | Thrift / protobuf ids and enum values: same meaning on every branch, never reused, new fields optional |
214+
208215
---
209216

210217
## Part 3: Cross-Module Concerns
@@ -214,6 +221,7 @@ Detailed module review guides live in `AGENTS.md` files in each source directory
214221
- [ ] Do new `TPlanNodeType` values have matching BE handling?
215222
- [ ] Are all FE-to-BE send paths updated when new transmitted variables are added?
216223
- [ ] Is mixed-version compatibility preserved for serialization or function metadata changes?
224+
- [ ] For `gensrc` changes: does every id and enum value mean the same thing on every release branch, with no reused id and no new required field?
217225

218226
### 3.2 Cloud Mode vs Shared-Nothing Mode
219227

‎.github/scripts/emit_litefuse_otel_io.py‎

Lines changed: 91 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818

1919
import argparse
2020
import base64
21+
import gzip
2122
from datetime import datetime, timedelta, timezone
2223
import hashlib
2324
import json
@@ -389,6 +390,14 @@ def build_ingestion_payload(args, input_text, output_text, events):
389390
if key not in ("type", "_line_number")
390391
}
391392

393+
superseded = next((event for event in reversed(events)
394+
if event.get("type") == "review.superseded"), None)
395+
if superseded is not None:
396+
trace_metadata["review_superseded"] = {
397+
key: value for key, value in superseded.items()
398+
if key not in ("type", "_line_number")
399+
}
400+
392401
completed_items = [
393402
event
394403
for event in events
@@ -1450,11 +1459,16 @@ def encoded_size(candidate):
14501459
return otlp_chunks(chunk, next_limit, trace_body)
14511460

14521461

1453-
def post_payload_once(endpoint, public_key, secret_key, payload, timeout_seconds):
1462+
def post_payload_once(
1463+
endpoint, public_key, secret_key, payload, timeout_seconds, gzip_enabled=False
1464+
):
14541465
auth = base64.b64encode(f"{public_key}:{secret_key}".encode()).decode()
1466+
data = compact_json_bytes(payload)
1467+
if gzip_enabled:
1468+
data = gzip.compress(data, compresslevel=1, mtime=0)
14551469
request = urllib.request.Request(
14561470
endpoint,
1457-
data=compact_json_bytes(payload),
1471+
data=data,
14581472
headers={
14591473
"Content-Type": "application/json",
14601474
"Authorization": f"Basic {auth}",
@@ -1464,6 +1478,8 @@ def post_payload_once(endpoint, public_key, secret_key, payload, timeout_seconds
14641478
},
14651479
method="POST",
14661480
)
1481+
if gzip_enabled:
1482+
request.add_header("Content-Encoding", "gzip")
14671483
with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
14681484
body = response.read().decode()
14691485
detail = json.loads(body) if body else {}
@@ -1485,6 +1501,7 @@ def post_payload_once(endpoint, public_key, secret_key, payload, timeout_seconds
14851501
return {
14861502
"status": response.status,
14871503
"success_count": otlp_span_count(payload),
1504+
"wire_bytes": len(data),
14881505
}
14891506

14901507

@@ -1497,10 +1514,13 @@ def post_payload(
14971514
timeout_seconds,
14981515
retry_attempts,
14991516
retry_sleep_seconds,
1517+
gzip_enabled=False,
15001518
):
1519+
upload_started = time.monotonic()
15011520
statuses = []
15021521
success_count = 0
15031522
request_sizes = []
1523+
wire_request_sizes = []
15041524
payload_too_large_retry_count = 0
15051525
transport_retry_count = 0
15061526
http_retry_count = 0
@@ -1514,7 +1534,8 @@ def post_payload(
15141534
post_attempt_count += 1
15151535
try:
15161536
status = post_payload_once(
1517-
endpoint, public_key, secret_key, otlp_chunk, timeout_seconds
1537+
endpoint, public_key, secret_key, otlp_chunk, timeout_seconds,
1538+
gzip_enabled=gzip_enabled,
15181539
)
15191540
except urllib.error.HTTPError as exc:
15201541
if exc.code in OTLP_RETRYABLE_HTTP_STATUS_CODES:
@@ -1589,11 +1610,16 @@ def post_payload(
15891610
statuses.append(status["status"])
15901611
success_count += int(status.get("success_count") or 0)
15911612
request_sizes.append(request_size)
1613+
wire_request_sizes.append(status["wire_bytes"])
15921614
return {
1615+
"compression": "gzip" if gzip_enabled else "none",
1616+
"upload_seconds": round(time.monotonic() - upload_started, 3),
15931617
"statuses": statuses,
15941618
"request_count": len(statuses),
15951619
"post_attempt_count": post_attempt_count,
15961620
"request_sizes": request_sizes,
1621+
# Like request_sizes, these sizes cover successful requests only.
1622+
"wire_request_sizes": wire_request_sizes,
15971623
"max_request_size": max(request_sizes) if request_sizes else 0,
15981624
"payload_too_large_retries": payload_too_large_retry_count,
15991625
"transport_retries": transport_retry_count,
@@ -1779,44 +1805,15 @@ def observation_has_expected_io(observation, target):
17791805
return all(observation.get(field) is not None for field in fields)
17801806

17811807

1782-
def inspect_trace(args, public_key, secret_key, target, deadline):
1808+
def inspect_observations(
1809+
args, target, observations, legacy_detail, read_source, legacy_trace_error
1810+
):
17831811
result = target["result"]
1784-
trace_id = result["trace_id"]
17851812
subagent = target["subagent"]
17861813
root_name = "codex.subagent.review" if subagent else "codex.review"
17871814
required_observation_count = max(
17881815
1 if subagent else args.min_observations, result["observation_count"]
17891816
)
1790-
legacy_trace_error = ""
1791-
try:
1792-
legacy_detail = fetch_trace(
1793-
args.base_url, public_key, secret_key, trace_id, deadline=deadline
1794-
)
1795-
except Exception as exc:
1796-
legacy_detail = {}
1797-
legacy_trace_error = type(exc).__name__
1798-
try:
1799-
observations_payload = fetch_observations_legacy(
1800-
args.base_url, public_key, secret_key, trace_id,
1801-
max_pages=max(10, required_observation_count // 100 + 1),
1802-
deadline=deadline,
1803-
)
1804-
observations = observation_rows_from_v2(observations_payload)
1805-
read_source = "legacy_observations"
1806-
except Exception as exc:
1807-
try:
1808-
observations_payload = fetch_observations_v2(
1809-
args.base_url, public_key, secret_key, trace_id,
1810-
max_pages=max(10, required_observation_count // 1000 + 1),
1811-
deadline=deadline,
1812-
start_time=target.get("start_time"),
1813-
end_time=target.get("end_time"),
1814-
)
1815-
observations = observation_rows_from_v2(observations_payload)
1816-
read_source = "v2_observations"
1817-
except Exception:
1818-
observations = legacy_detail.get("observations") or []
1819-
read_source = f"legacy_trace_fallback:{type(exc).__name__}"
18201817
observations_missing_io = [
18211818
observation
18221819
for observation in observations
@@ -1948,6 +1945,59 @@ def inspect_trace(args, public_key, secret_key, target, deadline):
19481945
return ok, last_diagnostic
19491946

19501947

1948+
def inspect_trace(args, public_key, secret_key, target, deadline):
1949+
result = target["result"]
1950+
trace_id = result["trace_id"]
1951+
required_observation_count = max(
1952+
1 if target["subagent"] else args.min_observations, result["observation_count"]
1953+
)
1954+
legacy_trace_error = ""
1955+
try:
1956+
legacy_detail = fetch_trace(
1957+
args.base_url, public_key, secret_key, trace_id, deadline=deadline
1958+
)
1959+
except Exception as exc:
1960+
legacy_detail = {}
1961+
legacy_trace_error = type(exc).__name__
1962+
1963+
# Some deployments return all observation IO in trace detail. Apply the
1964+
# same complete validation before downloading those observations again.
1965+
ok, detail_diagnostic = inspect_observations(
1966+
args, target, legacy_detail.get("observations") or [], legacy_detail,
1967+
"legacy_trace", legacy_trace_error,
1968+
)
1969+
if ok:
1970+
return ok, detail_diagnostic
1971+
1972+
try:
1973+
observations_payload = fetch_observations_legacy(
1974+
args.base_url, public_key, secret_key, trace_id,
1975+
max_pages=max(10, required_observation_count // 100 + 1),
1976+
deadline=deadline,
1977+
)
1978+
observations = observation_rows_from_v2(observations_payload)
1979+
read_source = "legacy_observations"
1980+
except Exception as exc:
1981+
try:
1982+
observations_payload = fetch_observations_v2(
1983+
args.base_url, public_key, secret_key, trace_id,
1984+
max_pages=max(10, required_observation_count // 1000 + 1),
1985+
deadline=deadline,
1986+
start_time=target.get("start_time"),
1987+
end_time=target.get("end_time"),
1988+
)
1989+
observations = observation_rows_from_v2(observations_payload)
1990+
read_source = "v2_observations"
1991+
except Exception:
1992+
return False, {
1993+
**detail_diagnostic,
1994+
"read_source": f"legacy_trace_fallback:{type(exc).__name__}",
1995+
}
1996+
return inspect_observations(
1997+
args, target, observations, legacy_detail, read_source, legacy_trace_error
1998+
)
1999+
2000+
19512001
def verify_traces(args, public_key, secret_key, targets):
19522002
# All exported traces share polling rounds and one elapsed-time budget.
19532003
# Never restart the wait budget for each subagent.
@@ -2016,6 +2066,10 @@ def parse_args():
20162066
parser.add_argument("--post-timeout-seconds", type=int, default=120)
20172067
parser.add_argument("--post-retry-attempts", type=int, default=5)
20182068
parser.add_argument("--post-retry-sleep-seconds", type=int, default=5)
2069+
parser.add_argument(
2070+
"--gzip", action="store_true",
2071+
help="Gzip OTLP uploads; payload size limits still apply before compression",
2072+
)
20192073
parser.add_argument("--verify", action="store_true")
20202074
parser.add_argument("--dry-run", action="store_true")
20212075
parser.add_argument("--verify-attempts", type=int, default=24)
@@ -2105,6 +2159,7 @@ def main():
21052159
args.post_timeout_seconds,
21062160
args.post_retry_attempts,
21072161
args.post_retry_sleep_seconds,
2162+
gzip_enabled=args.gzip,
21082163
)
21092164
result["subagent_traces"] = []
21102165
for subagent_payload in subagent_payloads:
@@ -2117,6 +2172,7 @@ def main():
21172172
args.post_timeout_seconds,
21182173
args.post_retry_attempts,
21192174
args.post_retry_sleep_seconds,
2175+
gzip_enabled=args.gzip,
21202176
)
21212177
result["subagent_traces"].append(
21222178
{

‎.github/scripts/resolve_code_review_status.py‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,19 +73,30 @@ def resolve_status(
7373

7474
approved_contexts: set[tuple[int, str]] = set()
7575
blocked_contexts: set[tuple[int, str]] = set()
76+
incomplete_contexts: set[tuple[int, str]] = set()
7677
for context, item in latest_by_context.items():
7778
match = SOURCE_CONTEXT_RE.fullmatch(context)
7879
if match is not None and item["state"] == "success":
7980
approved_contexts.add((int(match.group(1)), match.group(2).casefold()))
8081
elif (match is not None and item["state"] == "failure"
8182
and context.startswith("code-review/source/automated/")):
82-
blocked_contexts.add((int(match.group(1)), match.group(2).casefold()))
83+
key = (int(match.group(1)), match.group(2).casefold())
84+
blocked_contexts.add(key)
85+
# State remains the sole gating signal. Use the runner's description
86+
# only to explain an already-blocking result, never to grant a pass.
87+
description = item.get("description")
88+
if isinstance(description, str) and description.startswith("Review did not converge after "):
89+
incomplete_contexts.add(key)
8390

8491
# Preserve the authorized local PASS / skip override. Otherwise a completed
85-
# automated review with P0/P1 blocks, rather than appearing to still be running.
92+
# automated review with P0/P1 or incomplete coverage blocks instead of pending.
8693
blocked = sorted((open_contexts - approved_contexts) & blocked_contexts)
8794
if blocked:
8895
number, base_sha = blocked[0]
96+
if (number, base_sha) in incomplete_contexts:
97+
return Resolution(
98+
"failure", f"Review did not converge for PR #{number} at {base_sha[:12]}.",
99+
)
89100
return Resolution(
90101
"failure", f"Review completed with P0/P1 for PR #{number} at {base_sha[:12]}.",
91102
)

0 commit comments

Comments
 (0)