Skip to content

add missing file - #138

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:master
Nov 14, 2017
Merged

morningman merged 1 commit into
apache:masterfrom
morningman:master

Conversation

@morningman

Copy link
Copy Markdown
Contributor

ShowUserStmt.java

@morningman
morningman merged commit 44ed16c into apache:master Nov 14, 2017
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Jul 26, 2023
zclllyybb added a commit that referenced this pull request Jul 12, 2026
…ability (#65178)

The Codex GitHub Actions review pipeline needed stronger guarantees that
agents review the intended PR snapshot, use one authoritative diff,
pursue high-risk cross-module mechanisms, and expose the work of spawned
subagents in Litefuse.

This PR:

- verifies the checked-out PR head and rechecks the live base/head after
context preparation;
- prepares an authoritative aggregate PR patch and changed-file list,
with atomic bounded retry (`2s`, `4s`, `8s`) for transient GitHub API
failures;
- directs the main agent to perform an initial risk scan, dispatch
focused subagents for suspicious mechanisms, maintain a shared
append-only ledger, validate/deduplicate every candidate, and converge
for at most three rounds;
- requires paths outside the authoritative diff/changed-file context to
be confirmed with `rg --files` before reading;
- records separate Litefuse traces for each Codex subagent session,
including command/tool I/O and non-empty encrypted-reasoning status,
instead of mixing subagent activity into the main trace;
- makes Litefuse ingestion more resilient to transport timeouts while
keeping telemetry upload non-blocking for the review outcome.

The repository review skill is kept focused on review invariants;
orchestration and completion policy live in the workflow prompt, where
they are available to the GitHub Actions invocation.

### Validation and experiment material

- Static checks: `python3 -m py_compile
.github/scripts/emit_litefuse_otel_io.py` and `git diff --check`.
- Litefuse uploader validation: real/synthetic subagent session
conversion, payload-limit coverage, and readback of independent subagent
traces.
- End-to-end replayed review experiments against fresh fork PRs built
from the original PR base/head commits:

| Replay PR | GitHub Actions run | Result |
| --- | --- | --- |
| [#136](zclllyybb#136) |
[29112823062](https://github.com/zclllyybb/doris/actions/runs/29112823062)
| Success; 2 inline comments |
| [#137](zclllyybb#137) |
[29114260631](https://github.com/zclllyybb/doris/actions/runs/29114260631)
| Success; 3 inline comments |
| [#138](zclllyybb#138) |
[29115946034](https://github.com/zclllyybb/doris/actions/runs/29115946034)
| Success; 1 inline comment |
| [#139](zclllyybb#139) |
[29150372758](https://github.com/zclllyybb/doris/actions/runs/29150372758)
| Success; 1 inline comment |
| [#140](zclllyybb#140) |
[29151057520](https://github.com/zclllyybb/doris/actions/runs/29151057520)
| Success; 1 inline comment |

The latest replay trace, `86be226391a40e5992bb8d3b4e125eca`, confirms
one authoritative `pr.diff` read, two changed-file-list reads, no `gh pr
diff` or `git diff` calls from the reviewing agent, and separate
subagent sessions. It also exposed one remaining prompt-compliance gap:
a guessed adjacent `.cpp` path was read without a preceding `rg --files`
lookup; this is documented test evidence rather than hidden by the
validation.
Ryan19929 added a commit to Ryan19929/doris that referenced this pull request Nov 13, 2026
… restore DOWNLOAD journal

### What problem does this PR solve?

Issue Number: None

Related PR: apache#138

Problem Summary: The previous commits serialize a non-done restore job in the calling thread before it is queued to the EditLog flusher. For the DOWNLOAD journal this takes minutes, and the restore thread holds only the job lock meanwhile. A DROP TABLE FORCE (or DROP DATABASE FORCE) of a restored table, which the RESTORE table state does not stop, can take the db write lock and commit its DROP_TABLE journal first. Replaying DROP_TABLE -> DOWNLOAD ignores the drop because the table does not exist yet, then replayCheckAndPrepareMeta() registers the table again. If the FE restarts or the master changes before the CANCELLED journal is written, the dropped table comes back and the restore keeps running on it. The same race existed before with a much smaller window, because checkIfNeedCancel() does not hold the db lock either.

Fix: RestoreJob.logDownloadJournal() (also used by CloudRestoreJob) serializes the job without the db lock, then takes the db read lock, runs checkIfNeedCancel() again and logs the pre-serialized journal under that lock. A force drop either commits before the recheck, so the DOWNLOAD journal is not written and run() cancels the job, or waits for the db write lock until DOWNLOAD is logged. Holding the db read lock while waiting for the journal is lighter than DROP/CREATE TABLE, which wait for their journals under the db write lock; heartbeat, report, publish, checkpoint and BDB threads do not take db locks.

The caller-serialized bytes are now kept in SerializedJournalEntity, already in the JournalEntity format (op code + data), and JournalBatch.addJournal() uses its buffer directly. This removes the extra copy of a large job in the flusher, so peak memory stays at the level before serializing in the caller thread.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - run-fe-ut.sh on Linux: EditLogTest, RestoreJobTest, JournalBatchTest, BackupJobTest, BackupJobInfoTest, BackupRestoreJobStreamingJsonConfigTest, BDBJEJournalTest, 54 tests passed; fe-core checkstyle passed
    - New RestoreJobTest cases fail when the recheck under the db read lock is removed
- Behavior changed: No
- Does this need documentation: No
Ryan19929 added a commit to Ryan19929/doris that referenced this pull request Nov 13, 2026
… DOWNLOAD journal

### What problem does this PR solve?

Issue Number: None

Related PR: apache#138

Problem Summary: The master registers all restored tables, including views and other non-OLAP tables, when the restore job leaves PENDING, but the first journal that carries them is DOWNLOAD. A restored view can be dropped meanwhile with a plain DROP VIEW, since only OLAP tables are kept in the RESTORE state, and checkIfNeedCancel() only checks the OLAP tables in the file mapping, so the job goes on and logs DOWNLOAD after DROP_TABLE. A follower replaying DROP_TABLE ignores it because the view does not exist there yet, and then registers the view from the DOWNLOAD journal. The follower keeps a view the master no longer has, and it comes back after a master change.

Fix: under the db read lock in logDownloadJournal(), also check that every table in restoredTbls is still registered as the same object. If one has been dropped, the DOWNLOAD journal is not logged and the job is cancelled, as for a dropped OLAP table.

### Release note

Dropping a view that is being restored before the restore job reaches DOWNLOAD now cancels the restore job, instead of leaving the view on followers.

### Check List (For Author)

- Test: Unit Test
    - run-fe-ut.sh on Linux: EditLogTest, RestoreJobTest, JournalBatchTest, BackupJobTest, BackupJobInfoTest, BackupRestoreJobStreamingJsonConfigTest, BDBJEJournalTest, 55 tests passed; fe-core checkstyle passed
    - The new RestoreJobTest case fails when the restoredTbls check is removed
- Behavior changed: Yes. Dropping a restored view before DOWNLOAD cancels the restore job.
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants