Repository navigation
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Broadcast join runtime filters are globally merged by accepting the first producer, but every build-side BE still creates and sends the same broadcast runtime filter. This wastes runtime filter build CPU and network traffic. Add a Nereids-side producer allow-list controlled by runtime_filter_broadcast_join_producer_num so only up to the configured number of BEs produce broadcast runtime filters with remote targets. BE producer initialization skips disallowed broadcast remote runtime filters and leaves local-only broadcast filters and non-broadcast filters unchanged.
### Release note
Add session variable runtime_filter_broadcast_join_producer_num to limit Nereids broadcast join runtime filter producers.
### Check List (For Author)
- Test:
- Unit Test: ./run-fe-ut.sh --run org.apache.doris.qe.runtime.RuntimeFiltersThriftBuilderTest
- Unit Test: ./run-be-ut.sh --run --filter=RuntimeFilterMgrTest.*
- Build: ./build.sh --fe
- Static Analysis: build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN (blocked by pre-existing jni-util.h static_assert diagnostics and existing runtime_filter_mgr.cpp function-size diagnostic outside this change)
- Manual Check: git diff --check
- Behavior changed: Yes. Nereids broadcast join runtime filter producers with remote targets are limited by runtime_filter_broadcast_join_producer_num, defaulting to 3.
- Does this need documentation: No
There was a problem hiding this comment.
I found one correctness gap in the new broadcast runtime filter producer limiting: the new session variable is only propagated through the Nereids distributed-plan thrift builder, while the still-reachable legacy Coordinator path ignores it.
Critical checkpoint conclusions:
- Goal/test coverage: the PR adds FE/BE plumbing and unit tests for the Nereids path, but misses the legacy Coordinator path.
- Scope: the implementation is focused, but the parallel runtime-filter serialization path was not updated.
- Concurrency/lifecycle: no new BE race or dependency issue was substantiated; producer pruning happens before producer registration, and empty producer sets are no-ops.
- Compatibility/configuration: the thrift field is optional and unset/non-positive values safely keep all producers; mixed-version behavior looks safe.
- FE/BE protocol: incomplete because one FE send path never sets the new allow-list.
- Tests: current tests cover the new builder and BE predicate behavior, but not the legacy Coordinator fallback.
No additional user focus was provided. Review was static only, per the review-only instructions; no builds or tests were run.
| .getRuntimeFilterBroadcastJoinProducerNum(); | ||
| RuntimeFiltersThriftBuilder runtimeFiltersThriftBuilder = RuntimeFiltersThriftBuilder.compute( | ||
| coordinatorContext.runtimeFilters, distributedPlans); | ||
| coordinatorContext.runtimeFilters, distributedPlans, broadcastRuntimeFilterProducerNum); |
There was a problem hiding this comment.
The new session variable is only honored on the ThriftPlansBuilder path. When enable_nereids_distribute_planner is false, NereidsPlanner leaves distributed plans null and StmtExecutor falls back to the legacy Coordinator, whose runtime-filter params path builds TRuntimeFilterParams directly without setting broadcast_runtime_filter_producer_filter_ids. On BE an unset allow-list means should_build_runtime_filter_producer() returns true for every remote broadcast RF, so runtime_filter_broadcast_join_producer_num is silently ignored in that still-supported execution mode. Please either populate the same per-BE allow-list in the legacy Coordinator path or gate/scope the session variable so users cannot set it expecting it to work there.
…ability (#65178) The Codex GitHub Actions review pipeline needed stronger guarantees that agents review the intended PR snapshot, use one authoritative diff, pursue high-risk cross-module mechanisms, and expose the work of spawned subagents in Litefuse. This PR: - verifies the checked-out PR head and rechecks the live base/head after context preparation; - prepares an authoritative aggregate PR patch and changed-file list, with atomic bounded retry (`2s`, `4s`, `8s`) for transient GitHub API failures; - directs the main agent to perform an initial risk scan, dispatch focused subagents for suspicious mechanisms, maintain a shared append-only ledger, validate/deduplicate every candidate, and converge for at most three rounds; - requires paths outside the authoritative diff/changed-file context to be confirmed with `rg --files` before reading; - records separate Litefuse traces for each Codex subagent session, including command/tool I/O and non-empty encrypted-reasoning status, instead of mixing subagent activity into the main trace; - makes Litefuse ingestion more resilient to transport timeouts while keeping telemetry upload non-blocking for the review outcome. The repository review skill is kept focused on review invariants; orchestration and completion policy live in the workflow prompt, where they are available to the GitHub Actions invocation. ### Validation and experiment material - Static checks: `python3 -m py_compile .github/scripts/emit_litefuse_otel_io.py` and `git diff --check`. - Litefuse uploader validation: real/synthetic subagent session conversion, payload-limit coverage, and readback of independent subagent traces. - End-to-end replayed review experiments against fresh fork PRs built from the original PR base/head commits: | Replay PR | GitHub Actions run | Result | | --- | --- | --- | | [#136](zclllyybb#136) | [29112823062](https://github.com/zclllyybb/doris/actions/runs/29112823062) | Success; 2 inline comments | | [#137](zclllyybb#137) | [29114260631](https://github.com/zclllyybb/doris/actions/runs/29114260631) | Success; 3 inline comments | | [#138](zclllyybb#138) | [29115946034](https://github.com/zclllyybb/doris/actions/runs/29115946034) | Success; 1 inline comment | | [#139](zclllyybb#139) | [29150372758](https://github.com/zclllyybb/doris/actions/runs/29150372758) | Success; 1 inline comment | | [#140](zclllyybb#140) | [29151057520](https://github.com/zclllyybb/doris/actions/runs/29151057520) | Success; 1 inline comment | The latest replay trace, `86be226391a40e5992bb8d3b4e125eca`, confirms one authoritative `pr.diff` read, two changed-file-list reads, no `gh pr diff` or `git diff` calls from the reviewing agent, and separate subagent sessions. It also exposed one remaining prompt-compliance gap: a guessed adjacent `.cpp` path was read without a preceding `rg --files` lookup; this is documented test evidence rather than hidden by the validation.
Fresh replay PR for validating the current aggregate-diff review workflow. The base and head reproduce the referenced Apache Doris PR snapshot.