[6.5.0] Bazel just hangs after successfully completed build for a long time #21568
Description
Description of the bug:
Sometimes after build successfully completed (at least if I look into buildbuddy invocation), bazel just hangs and prints out Build completed successfully for many times. After that bazel process completes successfully.
Piece of CI log here:
...
[2024-03-05T11:26:21.099Z] INFO: Elapsed time: 403.120s, Critical Path: 224.13s
[2024-03-05T11:26:21.099Z] INFO: 4107 processes: 4005 remote cache hit, 54 internal, 48 processwrapper-sandbox.
[2024-03-05T11:26:21.099Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:26:22.465Z] INFO: Streaming build results to: https://build-buddy.local/invocation/312f5fbb-782a-4544-b78a-3b80ead554f3
[2024-03-05T11:26:37.322Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:26:52.194Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:27:10.255Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:27:25.125Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:27:40.009Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:27:54.873Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:28:09.730Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:28:27.793Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:28:39.985Z] INFO: Build completed successfully, 4107 total actions
[2024-03-05T11:28:42.505Z] INFO: Build completed successfully, 4107 total actions
This may take a long time (10 mins or more).... How can I find what realy bazel do at that time?
Which category does this issue belong to?
Core
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
I can't understand under what conditions this behavior happens yet.
Which operating system are you running Bazel on?
Ubuntu 20.04
What is the output of bazel info release?
release 6.5.0
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse HEAD ?
No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response