Skip to content

Commit 7a621a1

Browse files
airborne12claude
andcommitted
[chore](nereids) Merge master with the inference fix of apache#69419
### What problem does this PR solve? Issue Number: None Related PR: apache#69292, apache#69419 Problem Summary: apache#69419 is now on master: predicate inference no longer copies a MATCH, multi_match or SEARCH predicate to an equal column. Review of this PR found that problem. With c1 = c2 and the same index on both columns, the copy added its BM25 score to the original, so a minimum score such as score() > 0.6 returned rows whose score is 0.47. Merge master, which applies without conflicts, so that this PR is tested with the fix. ### Release note None ### Check List (For Author) - Test: Unit Test and Regression test, on an ASAN build of the merge - FE: BindScoreToOlapScanTest, InferPredicateByReplaceTest, InferPredicatesTest, UnequalPredicateInferTest, CTEInlineTest, LogicalPlanDeepCopierTest, TopNRuntimeFilterTest, PullUpProjectExprUnderTopNTest, MaterializedViewUtilsTest and SearchExpressionTest pass (317 tests). Checkstyle passes. - Regression: the 20 score() suites of this PR, infer_match_predicate and the five suites from master that use score() pass with unchanged .out files. The query from review returns no rows, and without the minimum its rows score 0.4700036. - BE unit tests were not rerun: the merge does not change the BE files of this PR, and master's function_string_test.cpp (apache#69223) does not compile, as the last BE UT run of this PR showed. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
2 parents 854a54f + 709fe0a commit 7a621a1

259 files changed

Lines changed: 15537 additions & 1797 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# GitHub request diagnostics for code review
2+
3+
The runner, status aggregator and result-sync jobs load review_github_api.py
4+
from the immutable workflow commit before their first GitHub API request. The
5+
copy in RUNNER_TEMP survives the subsequent PR checkout and is shared by the
6+
submission helper and head watchdog.
7+
8+
Use python3 "$REVIEW_GH_HELPER" api ... in these workflow steps. Python helpers
9+
use run_gh with captured text output, check=True and their existing timeout;
10+
use decode_json when decoding its result. Request arguments, normal stdout,
11+
write deduplication, permission decisions and retry policy remain unchanged.
12+
13+
On a failure, the helper prints a short diagnostic and saves one JSON record
14+
under REVIEW_GITHUB_DIAGNOSTICS_DIR. Records include the command exit status or
15+
timeout, the last HTTP method/path/status, GitHub request ID, rate-limit and
16+
retry headers, gh version, and bounded response/stdout/stderr excerpts. A
17+
successful gh exit followed by a Python JSON decode error is also recorded.
18+
A transport failure without a response has status: null; it is not inferred
19+
to be an HTTP error.
20+
21+
The helper captures GH_DEBUG=api in memory because gh can discard a malformed
22+
error body and its HTTP status while parsing it. Outgoing headers, request
23+
bodies and argv are never archived. Response headers use an allowlist.
24+
Credential values and common credential fields are redacted before excerpts
25+
are truncated. The response body is gh's formatted debug excerpt, not exact
26+
wire bytes; gh may itself limit or omit it. Debug format changes may leave
27+
fields unavailable. Hard termination of the helper can prevent recording.
28+
29+
Each failure uses a separate file so later cleanup errors do not overwrite it.
30+
An always() artifact step runs after cleanup, including when an API failure
31+
is caught by an optional step. Artifacts are named
32+
github-api-errors-<job>-<run>-<attempt>; reusable aggregation jobs include the
33+
head SHA to distinguish current-head and previous-head executions. Diagnostic
34+
upload/storage failures do not replace the original request failure.
35+
36+
The existing failed-code-review-* submission artifacts remain available.
37+
These additions do not change the failure notification's content or posting
38+
arguments, or its existing failure classification.
39+
40+
Local validation:
41+
42+
~~~sh
43+
RUN_GH_HTTP_TESTS=1 python3 -m unittest discover -s .github/scripts -p 'test_*.py'
44+
~~~
45+
46+
The opt-in HTTP test uses real gh only against a loopback server, with fake
47+
tokens and a temporary config; it never sends a request to GitHub. Linux-only
48+
process-supervision tests require a Linux host.
Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
#!/usr/bin/env python3
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
19+
"""Capture gh HTTP failures without changing stdout, exit status, or retry policy."""
20+
21+
import json
22+
import os
23+
import re
24+
import subprocess
25+
import sys
26+
import time
27+
import uuid
28+
from datetime import datetime, timezone
29+
from pathlib import Path
30+
from urllib.parse import urlsplit
31+
32+
LIMIT = 65536
33+
HEADERS = {
34+
"content-type", "content-length", "date", "x-github-request-id",
35+
"retry-after", "x-ratelimit-limit", "x-ratelimit-remaining",
36+
"x-ratelimit-reset", "x-ratelimit-resource",
37+
}
38+
39+
40+
def redact(value):
41+
if isinstance(value, bytes):
42+
value = value.decode("utf-8", errors="replace")
43+
value = value or ""
44+
# Artifacts do not inherit Actions' secret masker. Never store the request
45+
# headers or body; also redact credentials echoed by an error response.
46+
for name, secret in os.environ.items():
47+
if re.search(r"(?:TOKEN|SECRET|PASSWORD|(?:^|_)KEY|(?:^|_)AK|(?:^|_)SK)$", name) and secret:
48+
for encoded in (secret, json.dumps(secret)[1:-1]):
49+
value = value.replace(encoded, "[REDACTED]")
50+
value = re.sub(r"\b(?:gh[pousr]_[A-Za-z0-9]+|github_pat_[A-Za-z0-9_]+)\b", "[REDACTED]", value)
51+
value = re.sub(
52+
r'(?i)(["\']?(?:access_token|refresh_token|id_token|token|api_key|secret|'
53+
r'client_secret|password|authorization|cookie|set-cookie)["\']?\s*[:=]\s*)'
54+
r'("(?:\\.|[^"\\])*"|[^\r\n,}]+)', r'\1"[REDACTED]"', value,
55+
)
56+
value = re.sub(r"(https?://)[^/\s:@]+:[^/@\s]+@", r"\1[REDACTED]@", value)
57+
value = re.sub(r"(?i)([?&](?:token|access_token|signature|sig|key)=)[^&\s\"']+", r"\1[REDACTED]", value)
58+
return value
59+
60+
61+
def excerpt(value):
62+
value = redact(value)
63+
return value[:LIMIT] + ("\n[truncated]" if len(value) > LIMIT else "")
64+
65+
66+
def split_trace(stderr):
67+
"""Keep only the final HTTP exchange; discard every outgoing header/body.
68+
69+
GH_DEBUG=api writes HTTP diagnostics to stderr, independently of gh api's
70+
response parser. Its body is a formatted, size-limited debug excerpt, not
71+
a byte-for-byte copy. Unknown debug formatting is discarded, never dumped.
72+
"""
73+
text = stderr.decode("utf-8", errors="replace") if isinstance(stderr, bytes) else (stderr or "")
74+
response = {"method": None, "path": None, "status": None, "headers": {},
75+
"body_excerpt": "", "body_source": "gh_debug", "request_count": 0}
76+
ordinary = []
77+
mode = "ordinary"
78+
body = []
79+
body_chars = 0
80+
for line in text.splitlines():
81+
if line.startswith("* Request at "):
82+
count = response["request_count"] + 1
83+
response = {**response, "method": None, "path": None, "status": None,
84+
"headers": {}, "body_excerpt": "", "request_count": count}
85+
body = []
86+
body_chars = 0
87+
response.pop("body_truncated", None)
88+
mode = "request"
89+
elif line.startswith("> "):
90+
match = re.match(r"> ([A-Z]+) (\S+) HTTP/", line)
91+
if match:
92+
response["method"] = match[1]
93+
response["path"] = redact(urlsplit(match[2]).path)
94+
mode = "request"
95+
elif re.match(r"< HTTP/\S+ \d{3}", line):
96+
response["status"] = int(line.split()[2])
97+
response["headers"] = {}
98+
body = []
99+
body_chars = 0
100+
mode = "headers"
101+
elif mode == "headers" and line.startswith("< "):
102+
name, sep, value = line[2:].partition(":")
103+
if sep and name.lower() in HEADERS:
104+
response["headers"][name.lower()] = excerpt(value.strip())
105+
elif line.startswith("* Request took "):
106+
mode = "ordinary"
107+
elif mode == "headers" and not line:
108+
mode = "body"
109+
elif mode == "body":
110+
# Bound retained response content even if debug output grows.
111+
line = redact(line) # Redact before truncation can split a secret.
112+
if body_chars < LIMIT:
113+
response["body_truncated"] = len(line) + body_chars > LIMIT
114+
body.append(line[:LIMIT - body_chars])
115+
body_chars += len(line) + 1
116+
else:
117+
response["body_truncated"] = True
118+
elif mode == "ordinary" and not line.startswith("* "):
119+
ordinary.append(line)
120+
response["body_excerpt"] = excerpt("\n".join(body).strip())
121+
if mode == "request":
122+
# A transport failure has no response; do not mistake request data for
123+
# its body. subprocess timeout/type still identifies the failure.
124+
response["body_excerpt"] = ""
125+
return response, excerpt("\n".join(ordinary).strip())
126+
127+
128+
def record_failure(command, error, stdout="", stderr="", *, diagnostic=None):
129+
response, ordinary = split_trace(stderr) if diagnostic is None else (diagnostic, excerpt(stderr))
130+
record = {
131+
"time": datetime.now(timezone.utc).isoformat(),
132+
"run_id": os.environ.get("GITHUB_RUN_ID"),
133+
"run_attempt": os.environ.get("GITHUB_RUN_ATTEMPT"),
134+
"job": os.environ.get("GITHUB_JOB"),
135+
"gh_version": os.environ.get("REVIEW_GH_VERSION"),
136+
"error_type": type(error).__name__,
137+
"returncode": getattr(error, "returncode", None),
138+
"timeout_seconds": getattr(error, "timeout", None),
139+
"http": response,
140+
"stderr": ordinary,
141+
"stdout_excerpt": excerpt(stdout),
142+
}
143+
# Do not record argv: -f/--body/--input can contain credentials or entire
144+
# reviews. Endpoint metadata comes only from the outgoing request line.
145+
summary = (
146+
f"GitHub request failed: {response['method'] or 'unknown'} {response['path'] or 'unknown'}; "
147+
f"HTTP={response['status']}; request_id={response['headers'].get('x-github-request-id', 'unknown')}; "
148+
f"{record['error_type']}; exit={record['returncode']}; timeout={record['timeout_seconds']}"
149+
)
150+
record["summary"] = redact(summary)
151+
directory = os.environ.get("REVIEW_GITHUB_DIAGNOSTICS_DIR")
152+
if directory:
153+
try:
154+
path = Path(directory)
155+
path.mkdir(parents=True, exist_ok=True, mode=0o700)
156+
data = json.dumps(record, ensure_ascii=True, indent=2) + "\n"
157+
# Unique files support concurrent helper calls without append races.
158+
filename = path / f"{time.time_ns()}-{uuid.uuid4().hex}.json"
159+
with open(filename, "x", opener=lambda p, f: os.open(p, f, 0o600)) as handle:
160+
handle.write(data)
161+
except OSError as exc:
162+
print(f"Could not persist GitHub diagnostic ({type(exc).__name__})", file=sys.stderr)
163+
print(record["summary"], file=sys.stderr, flush=True)
164+
return ordinary, record
165+
166+
167+
def run_gh(command, **kwargs):
168+
"""subprocess.run contract for captured text gh calls, with failure evidence."""
169+
env = dict(kwargs.pop("env", os.environ))
170+
env.update(GH_DEBUG="api", NO_COLOR="1")
171+
try:
172+
result = subprocess.run(command, env=env, **kwargs)
173+
except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as error:
174+
ordinary, record = record_failure(command, error, error.stdout, error.stderr)
175+
error.stderr = ordinary + "\n" + record["summary"]
176+
# Preserve the original return code/timeout and stdout for callers.
177+
raise
178+
except OSError as error:
179+
record_failure(command, error)
180+
raise
181+
response, ordinary = split_trace(result.stderr)
182+
result.stderr = ordinary
183+
result.github_response = response
184+
return result
185+
186+
187+
def decode_json(result, command):
188+
try:
189+
return json.loads(result.stdout)
190+
except json.JSONDecodeError as error:
191+
record_failure(command, error, result.stdout, result.stderr,
192+
diagnostic=result.github_response)
193+
raise
194+
195+
196+
def main():
197+
command = ["gh", *sys.argv[1:]]
198+
try:
199+
result = run_gh(command, capture_output=True, text=True, check=True)
200+
except subprocess.CalledProcessError as error:
201+
sys.stdout.write(error.stdout or "")
202+
print(error.stderr, file=sys.stderr)
203+
return error.returncode if error.returncode >= 0 else 128 - error.returncode
204+
except OSError as error:
205+
print(f"Could not execute gh ({type(error).__name__})", file=sys.stderr)
206+
return 1
207+
sys.stdout.write(result.stdout)
208+
if result.stderr:
209+
print(result.stderr, file=sys.stderr)
210+
return 0
211+
212+
213+
if __name__ == "__main__":
214+
sys.exit(main())

‎.github/scripts/run_review_with_resume.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
from datetime import datetime, timezone
3434
from pathlib import Path
3535

36+
from review_github_api import decode_json, run_gh
3637
from submit_review import RUN_FILE, RESULT_FILE, SUBMISSION_FILE, verify_completion, write_json
3738

3839
# Six same-session capacity retries, each after a fixed five-minute wait.
@@ -78,11 +79,12 @@ def check(self, *, force=False):
7879
# Include API time in the existing shared budget and polling interval.
7980
self.next_check = time.monotonic() + HEAD_POLL_SECONDS
8081
try:
81-
response = subprocess.run(
82-
["gh", "api", f"repos/{self.args.repository}/pulls/{self.args.pr_number}"],
82+
command = ["gh", "api", f"repos/{self.args.repository}/pulls/{self.args.pr_number}"]
83+
response = run_gh(
84+
command,
8385
check=True, capture_output=True, text=True, timeout=timeout,
8486
)
85-
check_head(json.loads(response.stdout), self.args.head_sha)
87+
check_head(decode_json(response, command), self.args.head_sha)
8688
except (ValueError, OSError, subprocess.SubprocessError) as exc:
8789
# An unavailable API is not evidence of a new commit. Strict target
8890
# checks before resume/submission still apply independently.

‎.github/scripts/submit_review.py‎

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@
3030
from datetime import datetime, timezone
3131
from pathlib import Path
3232

33+
from review_github_api import decode_json, redact, run_gh
34+
3335

3436
SUBMISSION_FILE = "review-submission.json"
3537
REJECTION_FILE = "review-submission-rejected.json"
@@ -96,16 +98,7 @@ def validate_submission(value):
9698

9799

98100
def diagnostic_text(value):
99-
if isinstance(value, bytes):
100-
value = value.decode("utf-8", errors="replace")
101-
value = value or ""
102-
# These are the credentials consumed by gh. Artifacts do not inherit the
103-
# GitHub Actions log masker, so remove them before persisting API output.
104-
for name in ("GH_TOKEN", "GITHUB_TOKEN", "GH_ENTERPRISE_TOKEN", "GITHUB_ENTERPRISE_TOKEN"):
105-
token = os.environ.get(name)
106-
if token:
107-
value = value.replace(token, "[REDACTED]")
108-
value = re.sub(r"\b(?:gh[pousr]_[A-Za-z0-9]+|github_pat_[A-Za-z0-9_]+)\b", "[REDACTED]", value)
101+
value = redact(value)
109102
return value[:65536] + ("\n[truncated after 65536 characters]" if len(value) > 65536 else "")
110103

111104

@@ -135,14 +128,14 @@ def api(path, remaining, *, payload=None, paginated=False):
135128
command += ["--method", "POST", "--input", "-"]
136129
method = "POST" if payload is not None else "GET"
137130
try:
138-
result = subprocess.run(
131+
result = run_gh(
139132
command, input=json.dumps(payload) if payload is not None else None,
140133
capture_output=True, text=True, check=True, timeout=min(30, remaining()),
141134
)
142135
except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as error:
143136
raise GitHubAPIError(path, method, error, error.stdout, error.stderr) from None
144137
try:
145-
value = json.loads(result.stdout)
138+
value = decode_json(result, command)
146139
except json.JSONDecodeError as error:
147140
raise GitHubAPIError(path, method, error, result.stdout, result.stderr) from None
148141
return [item for page in value for item in page] if paginated else value

‎.github/scripts/test_prepare_review_agents.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,7 @@ def test_actual_workflow_prepares_context_and_final_prompt(self):
187187
sed.write_text('#!/bin/bash\nif [ "$1" = -i ]; then shift; exec /usr/bin/sed -i "" "$@"; fi\nexec /usr/bin/sed "$@"\n')
188188
sed.chmod(0o755)
189189
env = {**os.environ, "PATH": f"{bin_dir}:{os.environ['PATH']}",
190+
"REVIEW_GH_HELPER": str(SCRIPTS / "review_github_api.py"),
190191
"REPO": "apache/doris", "PR_NUMBER": "66227", "HEAD_SHA": self.head,
191192
"BASE_SHA": self.base, "HELPER_REF": self.trusted,
192193
"REVIEW_CONTEXT_DIR": str(self.context), "REVIEW_CONTEXT_REL": self.context.name,

‎.github/scripts/test_review_auth_quarantine.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,12 +183,14 @@ def setUp(self):
183183
"REPO": "example/repo", "PR_NUMBER": "1", "HEAD_SHA": "a" * 40,
184184
"GITHUB_WORKSPACE": str(self.root),
185185
"BASE_SHA": "b" * 40, "HELPER_REF": "test-pin", "REVIEW_TIMEOUT_MINUTES": "120",
186+
"REVIEW_GH_HELPER": str(Path(__file__).resolve().with_name("review_github_api.py")),
186187
"FAKE_REVIEW_HELPER": str(Path(__file__).resolve().with_name("run_review_with_resume.py")),
187188
})
188189
self.new_runner()
189190

190191
def new_runner(self):
191192
runner = Path(tempfile.mkdtemp(dir=self.root))
193+
shutil.copyfile(self.env["REVIEW_GH_HELPER"], runner / "review_github_api.py")
192194
context = runner / "review-context"
193195
context.mkdir()
194196
(context / "codex_goal_prompt.txt").write_text("Fake review")
@@ -439,7 +441,7 @@ def test_sync_separates_execution_failure_from_blocking_findings(self):
439441
("success", "failure", "failure"), ("failure", "success", "pending")):
440442
with self.subTest(execution=execution, verdict=verdict):
441443
result = subprocess.run(["bash", "-eo", "pipefail", "-c",
442-
'gh() { printf "%s\\n" "$@"; };\n' + re.sub(r"\$\{\{.*?\}\}", "test", step_script("Sync Code Review check for current head"))],
444+
'python3() { shift; printf "%s\\n" "$@"; };\n' + re.sub(r"\$\{\{.*?\}\}", "test", step_script("Sync Code Review check for current head"))],
443445
env={**self.env, "JOB_STATUS": execution, "REVIEW_CONTEXT_OUTCOME": "success",
444446
"REVIEW_OUTCOME": execution, "REVIEW_STATE": verdict, "REVIEW_P0": "0", "REVIEW_P1": "1"},
445447
capture_output=True, text=True, check=True)

‎.github/scripts/test_review_convergence_workflow.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,8 +67,9 @@ def test_verified_result_flows_through_source_and_aggregate_status(self):
6767
source_script = step_script("Sync Code Review check for current head")
6868
# Replace the only external mutation with an argument capture. Run the
6969
# actual Bash/jq blocks, then feed their source into the real resolver.
70-
capture = """gh() {
71-
python3 -c 'import json, sys; print(json.dumps(sys.argv[1:]))' "$@"
70+
capture = """python3() {
71+
shift
72+
command python3 -c 'import json, sys; print(json.dumps(sys.argv[1:]))' "$@"
7273
}
7374
"""
7475
for converged, p0, p1, expected in (

0 commit comments

Comments
 (0)