fix: replace ctx.actions.symlink with cp for Bazel 9 linux-sandbox#294
Merged
xnox merged 1 commit intoApr 23, 2026
Merged
Conversation
Member
|
Also likely need to add 9 to the version test matrix in the actions (possibly separately to this PR, and mark it as non-required / non-blocking). |
Member
|
And tests are likely broken; due to using out of date versions that have likely been removed now; hence need to fix up tests. |
Contributor
Author
We see this internally too; Alpine dependencies get instantly deleted upstream. As a workaround, just for testing purposes I guess you could just recreate a lockfile for every run? |
This comment was marked as off-topic.
This comment was marked as off-topic.
ctx.actions.symlink outputs dangle when used as inputs to sandboxed
actions under Bazel 9 with linux-sandbox. The symlink points into the
previous sandbox directory which no longer exists.
Reproduce with Bazel 9 + linux-sandbox:
git checkout HEAD~1 # upstream v1.5.40
cd e2e/smoke && bazel build :lock # fails
git checkout - # this fix
cd e2e/smoke && bazel build :lock # works
Not reproducible with processwrapper-sandbox.
Upstream bugs:
bazelbuild/bazel#28953
bazelbuild/bazel#27068
Replace all ctx.actions.symlink calls with cp via copy_to_workdir.
xnox
force-pushed
the
fix/bazel9-replace-symlinks-with-copies
branch
from
April 23, 2026 15:48
c58d9f3 to
495d4fc
Compare
Member
|
@tiborv rebased your PR, things should be more green now! |
Member
But will work on 1 & 2 separately. |
xnox
approved these changes
Apr 23, 2026
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 24, 2026
The per-file copy_to_workdir() approach introduced in chainguard-dev#294 declares one output per cached content entry. For cache entries whose last two path segments share a basename (e.g. wolfi-signing.rsa.pub/wolfi-signing.rsa.pub), some REv2 executors (observed on BuildBuddy) fail to validate the output: the CopyFile action exits 0 but Bazel reports 'not all outputs were created or valid'. Collapse the N per-file CopyFile spawns into a single setup script that runs inline at the start of the apko build action. Only the setup script is a declared action output; the individual copies happen inside the apko action's sandbox and are never surfaced to REv2 output validation. This keeps the Bazel 9 linux-sandbox fix from chainguard-dev#294 intact (no ctx.actions.symlink for inputs) while avoiding the output-path shape that triggers the BB RBE bug. apko_config.bzl is untouched; its copy_to_workdir usage for config files does not hit the duplicated-basename pattern in practice.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends in its filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key and the stored filename are identical, producing on-disk paths shaped like: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file action output via ctx.actions.declare_file at exactly that path. Some Bazel sandbox and remote-execution implementations reject declared output paths whose basename equals their parent directory name, so the per-file copy actions either fail validation or are reported as producing missing/invalid outputs even though the cp command itself succeeded. Switch the cache-staging step to declare a single TreeArtifact for the whole cache directory and populate it in one ctx.actions.run_shell driven by a manifest written via ctx.actions.write. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. The lockfile and the apko binary remain individually declared via copy_to_workdir; their layout is fixed and never produces the duplicated-basename pattern.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
) chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp actions, declaring each cache entry via ctx.actions.declare_file. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends (observed on BuildBuddy) reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single TreeArtifact and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
) chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp actions, declaring each cache entry via ctx.actions.declare_file. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single TreeArtifact and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the TreeArtifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
…#294) chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp actions, declaring each cache entry via ctx.actions.declare_file. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single tree artifact (via ctx.actions.declare_directory) and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the tree artifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
…#294) chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp actions, declaring each cache entry via ctx.actions.declare_file. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single tree artifact (via ctx.actions.declare_directory) and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the tree artifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
…#294) After some more testing we noticed that the per-file cp staging introduced in chainguard-dev#294 - which replaced ctx.actions.symlink to fix Bazel 9 sandbox issues - regresses on remote execution. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single tree artifact (via ctx.actions.declare_directory) and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the tree artifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
tiborv
added a commit
to tiborv/rules_apko
that referenced
this pull request
Apr 27, 2026
…#294) After some more testing we noticed that the per-file cp staging introduced in chainguard-dev#294 - which replaced ctx.actions.symlink to fix Bazel 9 sandbox issues - regresses on remote execution. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single tree artifact (via ctx.actions.declare_directory) and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the tree artifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
xnox
pushed a commit
that referenced
this pull request
Apr 27, 2026
After some more testing we noticed that the per-file cp staging introduced in #294 - which replaced ctx.actions.symlink to fix Bazel 9 sandbox issues - regresses on remote execution. apko's cache is URL-keyed, so when a cached resource's URL ends in its own filename - typical for keyrings, e.g. https://packages.wolfi.dev/os/wolfi-signing.rsa.pub the cache key equals the filename and the entry lands at: .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub Some remote-execution backends reject declared output paths whose basename equals their parent directory name as REv2 validation failures, so cache staging fails on RBE even though the cp command itself succeeds. Declare the whole cache as a single tree artifact (via ctx.actions.declare_directory) and populate it in one ctx.actions.run_shell. apko sees the identical files at the identical relative paths, but the problematic paths are now contents of the tree artifact rather than individually declared outputs, so per-output validation no longer sees them. copy_to_workdir grows a list-of-Files branch for this; the lockfile, config, and apko binary continue through the existing single-file and directory paths unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ctx.actions.symlink outputs dangle when used as inputs to sandboxed actions under Bazel 9 with linux-sandbox. The symlink points into the previous sandbox directory which no longer exists.
Reproduce with Bazel 9 + linux-sandbox:
Not reproducible with processwrapper-sandbox.
Upstream bugs:
bazelbuild/bazel#28953
bazelbuild/bazel#27068
Replace all ctx.actions.symlink calls with cp via copy_to_workdir.