Skip to content

Retry aggregate-diff review experiment for apache/doris#64683 - #140

Closed
zclllyybb wants to merge 1 commit into
codex-aggregate-64683-base-20260710from
codex-retry-64683-head-20260711
Closed

zclllyybb wants to merge 1 commit into
codex-aggregate-64683-base-20260710from
codex-retry-64683-head-20260711

Conversation

@zclllyybb

Copy link
Copy Markdown
Owner

Fresh replay PR for validating the current aggregate-diff review workflow. The base and head reproduce the referenced Apache Doris PR snapshot.

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Broadcast join runtime filters are globally merged by accepting the first producer, but every build-side BE still creates and sends the same broadcast runtime filter. This wastes runtime filter build CPU and network traffic. Add a Nereids-side producer allow-list controlled by runtime_filter_broadcast_join_producer_num so only up to the configured number of BEs produce broadcast runtime filters with remote targets. BE producer initialization skips disallowed broadcast remote runtime filters and leaves local-only broadcast filters and non-broadcast filters unchanged.

### Release note

Add session variable runtime_filter_broadcast_join_producer_num to limit Nereids broadcast join runtime filter producers.

### Check List (For Author)

- Test:
    - Unit Test: ./run-fe-ut.sh --run org.apache.doris.qe.runtime.RuntimeFiltersThriftBuilderTest
    - Unit Test: ./run-be-ut.sh --run --filter=RuntimeFilterMgrTest.*
    - Build: ./build.sh --fe
    - Static Analysis: build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN (blocked by pre-existing jni-util.h static_assert diagnostics and existing runtime_filter_mgr.cpp function-size diagnostic outside this change)
    - Manual Check: git diff --check
- Behavior changed: Yes. Nereids broadcast join runtime filter producers with remote targets are limited by runtime_filter_broadcast_join_producer_num, defaulting to 3.
- Does this need documentation: No

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one correctness gap in the new broadcast runtime filter producer limiting: the new session variable is only propagated through the Nereids distributed-plan thrift builder, while the still-reachable legacy Coordinator path ignores it.

Critical checkpoint conclusions:

  • Goal/test coverage: the PR adds FE/BE plumbing and unit tests for the Nereids path, but misses the legacy Coordinator path.
  • Scope: the implementation is focused, but the parallel runtime-filter serialization path was not updated.
  • Concurrency/lifecycle: no new BE race or dependency issue was substantiated; producer pruning happens before producer registration, and empty producer sets are no-ops.
  • Compatibility/configuration: the thrift field is optional and unset/non-positive values safely keep all producers; mixed-version behavior looks safe.
  • FE/BE protocol: incomplete because one FE send path never sets the new allow-list.
  • Tests: current tests cover the new builder and BE predicate behavior, but not the legacy Coordinator fallback.

No additional user focus was provided. Review was static only, per the review-only instructions; no builds or tests were run.

.getRuntimeFilterBroadcastJoinProducerNum();
RuntimeFiltersThriftBuilder runtimeFiltersThriftBuilder = RuntimeFiltersThriftBuilder.compute(
coordinatorContext.runtimeFilters, distributedPlans);
coordinatorContext.runtimeFilters, distributedPlans, broadcastRuntimeFilterProducerNum);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new session variable is only honored on the ThriftPlansBuilder path. When enable_nereids_distribute_planner is false, NereidsPlanner leaves distributed plans null and StmtExecutor falls back to the legacy Coordinator, whose runtime-filter params path builds TRuntimeFilterParams directly without setting broadcast_runtime_filter_producer_filter_ids. On BE an unset allow-list means should_build_runtime_filter_producer() returns true for every remote broadcast RF, so runtime_filter_broadcast_join_producer_num is silently ignored in that still-supported execution mode. Please either populate the same per-BE allow-list in the legacy Coordinator path or gate/scope the session variable so users cannot set it expecting it to work there.

zclllyybb added a commit to apache/doris that referenced this pull request Jul 12, 2026
…ability (#65178)

The Codex GitHub Actions review pipeline needed stronger guarantees that
agents review the intended PR snapshot, use one authoritative diff,
pursue high-risk cross-module mechanisms, and expose the work of spawned
subagents in Litefuse.

This PR:

- verifies the checked-out PR head and rechecks the live base/head after
context preparation;
- prepares an authoritative aggregate PR patch and changed-file list,
with atomic bounded retry (`2s`, `4s`, `8s`) for transient GitHub API
failures;
- directs the main agent to perform an initial risk scan, dispatch
focused subagents for suspicious mechanisms, maintain a shared
append-only ledger, validate/deduplicate every candidate, and converge
for at most three rounds;
- requires paths outside the authoritative diff/changed-file context to
be confirmed with `rg --files` before reading;
- records separate Litefuse traces for each Codex subagent session,
including command/tool I/O and non-empty encrypted-reasoning status,
instead of mixing subagent activity into the main trace;
- makes Litefuse ingestion more resilient to transport timeouts while
keeping telemetry upload non-blocking for the review outcome.

The repository review skill is kept focused on review invariants;
orchestration and completion policy live in the workflow prompt, where
they are available to the GitHub Actions invocation.

### Validation and experiment material

- Static checks: `python3 -m py_compile
.github/scripts/emit_litefuse_otel_io.py` and `git diff --check`.
- Litefuse uploader validation: real/synthetic subagent session
conversion, payload-limit coverage, and readback of independent subagent
traces.
- End-to-end replayed review experiments against fresh fork PRs built
from the original PR base/head commits:

| Replay PR | GitHub Actions run | Result |
| --- | --- | --- |
| [#136](zclllyybb#136) |
[29112823062](https://github.com/zclllyybb/doris/actions/runs/29112823062)
| Success; 2 inline comments |
| [#137](zclllyybb#137) |
[29114260631](https://github.com/zclllyybb/doris/actions/runs/29114260631)
| Success; 3 inline comments |
| [#138](zclllyybb#138) |
[29115946034](https://github.com/zclllyybb/doris/actions/runs/29115946034)
| Success; 1 inline comment |
| [#139](zclllyybb#139) |
[29150372758](https://github.com/zclllyybb/doris/actions/runs/29150372758)
| Success; 1 inline comment |
| [#140](zclllyybb#140) |
[29151057520](https://github.com/zclllyybb/doris/actions/runs/29151057520)
| Success; 1 inline comment |

The latest replay trace, `86be226391a40e5992bb8d3b4e125eca`, confirms
one authoritative `pr.diff` read, two changed-file-list reads, no `gh pr
diff` or `git diff` calls from the reviewing agent, and separate
subagent sessions. It also exposed one remaining prompt-compliance gap:
a guessed adjacent `.cpp` path was read without a preceding `rg --files`
lookup; this is documented test evidence rather than hidden by the
validation.
@zclllyybb
zclllyybb deleted the branch codex-aggregate-64683-base-20260710 July 13, 2026 00:31
@zclllyybb zclllyybb closed this Jul 13, 2026
@zclllyybb
zclllyybb deleted the codex-retry-64683-head-20260711 branch July 13, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants