Skip to content

[7.4.0] Output reuse#23674

Merged
iancha1992 merged 4 commits intobazelbuild:release-7.4.0from
fmeum:23377-cherry
Sep 19, 2024
Merged

[7.4.0] Output reuse#23674
iancha1992 merged 4 commits intobazelbuild:release-7.4.0from
fmeum:23377-cherry

Conversation

@fmeum
Copy link
Copy Markdown
Collaborator

@fmeum fmeum commented Sep 19, 2024

Cherry-picks the following changes to implement output reuse:

Fixes #23377
Fixes #23444
Fixes #23457

This fixes failures such as the following when a spawn, e.g. a reduced Java compilation spawn, doesn't create an optional output:

```
java.io.FileNotFoundException: /private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/execroot/_main/bazel-out/_tmp/actions/remote/175.tmp (No such file or directory)
	at com.google.devtools.build.lib.unix.NativePosixFiles.lstat(Native Method)
	at com.google.devtools.build.lib.unix.UnixFileSystem.statInternal(UnixFileSystem.java:212)
	at com.google.devtools.build.lib.unix.UnixFileSystem.stat(UnixFileSystem.java:201)
	at com.google.devtools.build.lib.vfs.Path.stat(Path.java:290)
	at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:456)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.moveOutputsToFinalLocation(RemoteExecutionService.java:878)
        ...
```

Also clean up temporary files in case of an exception.

Work towards bazelbuild#23288

Closes bazelbuild#23296.

PiperOrigin-RevId: 665744936
Change-Id: I89a409c7a6b28b2a5fa532bdb233dca9bc5bde73
When an action may modify a spawn's outputs after execution, the upload of outputs to the cache and reuse for deduplicated actions need to happen synchronously directly after spawn execution to avoid a race.

This commit implements this for cache uploads by marking all actions with this property and simply disabling async upload for all spawns executed by such actions.

For output reuse, all executions deduplicated against the first one register atomically upon deduplication and cause the cache upload to wait for all of them to complete reuse.

Fixes bazelbuild#22501
Fixes bazelbuild#23288
Work towards bazelbuild#21578
Closes bazelbuild#23307 (no longer needed)

Closes bazelbuild#23382.

PiperOrigin-RevId: 668101364
Change-Id: Ice75dbe14a7dd46e02ecb096d2b2a30940216356
@fmeum fmeum requested a review from a team as a code owner September 19, 2024 09:12
@github-actions github-actions bot added awaiting-review PR is awaiting review from an assigned reviewer team-Configurability platforms, toolchains, cquery, select(), config transitions team-Performance Issues for Performance teams team-Remote-Exec Issues and PRs for the Execution (Remote) team team-Rules-Java Issues for Java rules labels Sep 19, 2024
@fmeum fmeum requested a review from tjgq September 19, 2024 09:17
Closes bazelbuild#23422.

PiperOrigin-RevId: 669296792
Change-Id: If3a2519cbf57cc82f1f733900b93705065b37e63
@iancha1992 iancha1992 added this pull request to the merge queue Sep 19, 2024
Merged via the queue into bazelbuild:release-7.4.0 with commit d03dca6 Sep 19, 2024
@github-actions github-actions bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Sep 19, 2024
@fmeum fmeum deleted the 23377-cherry branch September 23, 2024 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team-Configurability platforms, toolchains, cquery, select(), config transitions team-Performance Issues for Performance teams team-Remote-Exec Issues and PRs for the Execution (Remote) team team-Rules-Java Issues for Java rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants