Wire up PathMapper in RemoteExecutionService
#19721
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Oct 4, 2023
Collaborator
Author
github-actions
bot
added
awaiting-review
gregestren
reviewed
Oct 4, 2023
src/main/java/com/google/devtools/build/lib/remote/RemoteExecutionService.java
Show resolved
Hide resolved
tjgq
reviewed
Oct 5, 2023
src/main/java/com/google/devtools/build/lib/remote/RemoteExecutionService.java
Outdated
Show resolved
Hide resolved
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 5, 2023
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` in `SpawnInputsExpander`, which takes care of this for inputs to `Spawn`s executed in a sandbox or remotely. Constructs specific to Blaze, filesets and archived tree artifacts, are not covered by this change. An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721. Work towards #6526 Closes #19718. PiperOrigin-RevId: 571109361 Change-Id: Ia38464011f658178ab2a1981a3ddaf5aead7c8fa
tjgq
requested changes
Oct 6, 2023
Contributor
tjgq
left a comment
tjgq
left a comment
There was a problem hiding this comment.
Can you please add two tests:
- One to verify that
buildRemoteActionmaps both inputs and outputs correctly. - One to verify that
downloadOutputscorrectly downloadsbazel-out/xxx/bin/fooeven when it gets path-stripped intobazel-out/bin/foo(pass a customRemoteOutputCheckerinto theRemoteExecutionService).
src/main/java/com/google/devtools/build/lib/remote/common/RemotePathResolver.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/common/RemotePathResolver.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/remote/RemoteExecutionService.java
Show resolved
Hide resolved
fmeum
force-pushed
the
6526-wire-up-remoteexecutionservice
branch
2 times, most recently
from
2a468b6 to
cadb5ef
Compare
tjgq
approved these changes
Oct 9, 2023
tjgq
added
awaiting-PR-merge
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 9, 2023
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` with sandbox outputs logic for sandboxed and worker sandboxed execution. An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721. Work towards #6526 Closes #19719. PiperOrigin-RevId: 571950903 Change-Id: I4a9d15da578da11f8677279b09e0864b8c33e9fc
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` in `RemoteExecutionService`, which ensures that paths of inputs and outputs are correctly mapped before being sent off to the remote executor and mapped back to the correct local paths when downloading the results. Work towards bazelbuild#6526
fmeum
force-pushed
the
6526-wire-up-remoteexecutionservice
branch
from
cadb5ef to
d54bc88
Compare
Collaborator
Author
|
I resolved the merge conflicts. |
Pavank1992
added
awaiting-review
tjgq
approved these changes
Oct 11, 2023
tjgq
added
awaiting-PR-merge
Collaborator
Author
|
@tjgq Is this still on track for getting merged? Happy to look into any issues it causes. |
Contributor
Sorry for the delay. It should be merged in today! |
github-actions
bot
removed
the
awaiting-PR-merge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PathMappers rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up thePathMapperused by a givenSpawninRemoteExecutionService, which ensures that paths of inputs and outputs are correctly mapped before being sent off to the remote executor and mapped back to the correct local paths when downloading the results.An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721.
Work towards #6526