Skip to content

fix: replace ctx.actions.symlink with cp for Bazel 9 linux-sandbox#294

Merged
xnox merged 1 commit into
chainguard-dev:mainfrom
tiborv:fix/bazel9-replace-symlinks-with-copies
Apr 23, 2026
Merged

fix: replace ctx.actions.symlink with cp for Bazel 9 linux-sandbox#294
xnox merged 1 commit into
chainguard-dev:mainfrom
tiborv:fix/bazel9-replace-symlinks-with-copies

Conversation

@tiborv

@tiborv tiborv commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

ctx.actions.symlink outputs dangle when used as inputs to sandboxed actions under Bazel 9 with linux-sandbox. The symlink points into the previous sandbox directory which no longer exists.

Reproduce with Bazel 9 + linux-sandbox:

git checkout HEAD~1   # upstream v1.5.40
cd e2e/smoke && bazel build :lock   # fails

git checkout -        # this fix
cd e2e/smoke && bazel build :lock   # works

Not reproducible with processwrapper-sandbox.

Upstream bugs:
bazelbuild/bazel#28953
bazelbuild/bazel#27068

Replace all ctx.actions.symlink calls with cp via copy_to_workdir.

@xnox

xnox commented Apr 21, 2026

Copy link
Copy Markdown
Member

Also likely need to add 9 to the version test matrix in the actions (possibly separately to this PR, and mark it as non-required / non-blocking).

@xnox

xnox commented Apr 21, 2026

Copy link
Copy Markdown
Member

And tests are likely broken; due to using out of date versions that have likely been removed now; hence need to fix up tests.

@tiborv

tiborv commented Apr 22, 2026

Copy link
Copy Markdown
Contributor Author

And tests are likely broken; due to using out of date versions that have likely been removed now; hence need to fix up tests.

We see this internally too; Alpine dependencies get instantly deleted upstream. As a workaround, just for testing purposes I guess you could just recreate a lockfile for every run?

@xnox

This comment was marked as off-topic.

ctx.actions.symlink outputs dangle when used as inputs to sandboxed
actions under Bazel 9 with linux-sandbox. The symlink points into the
previous sandbox directory which no longer exists.

Reproduce with Bazel 9 + linux-sandbox:

    git checkout HEAD~1   # upstream v1.5.40
    cd e2e/smoke && bazel build :lock   # fails

    git checkout -        # this fix
    cd e2e/smoke && bazel build :lock   # works

Not reproducible with processwrapper-sandbox.

Upstream bugs:
  bazelbuild/bazel#28953
  bazelbuild/bazel#27068

Replace all ctx.actions.symlink calls with cp via copy_to_workdir.
@xnox
xnox force-pushed the fix/bazel9-replace-symlinks-with-copies branch from c58d9f3 to 495d4fc Compare April 23, 2026 15:48
@xnox

xnox commented Apr 23, 2026

Copy link
Copy Markdown
Member

@tiborv rebased your PR, things should be more green now!

@xnox

xnox commented Apr 23, 2026

Copy link
Copy Markdown
Member
  1. looks like i have other failures with bazel 9
  2. also we don't test the lock command; but we should
  3. existing CI passes, so this is good to go in

But will work on 1 & 2 separately.

@xnox
xnox merged commit 5d44496 into chainguard-dev:main Apr 23, 2026
15 checks passed
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 24, 2026
The per-file copy_to_workdir() approach introduced in chainguard-dev#294 declares one
output per cached content entry. For cache entries whose last two path
segments share a basename (e.g. wolfi-signing.rsa.pub/wolfi-signing.rsa.pub),
some REv2 executors (observed on BuildBuddy) fail to validate the output:
the CopyFile action exits 0 but Bazel reports 'not all outputs were
created or valid'.

Collapse the N per-file CopyFile spawns into a single setup script that
runs inline at the start of the apko build action. Only the setup
script is a declared action output; the individual copies happen inside
the apko action's sandbox and are never surfaced to REv2 output
validation.

This keeps the Bazel 9 linux-sandbox fix from chainguard-dev#294 intact (no
ctx.actions.symlink for inputs) while avoiding the output-path shape
that triggers the BB RBE bug.

apko_config.bzl is untouched; its copy_to_workdir usage for config
files does not hit the duplicated-basename pattern in practice.
@tiborv
tiborv deleted the fix/bazel9-replace-symlinks-with-copies branch April 27, 2026 13:13
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
apko's cache layout is URL-keyed. When a cached resource's URL ends
in its filename - typical for keyrings, e.g.
https://packages.wolfi.dev/os/wolfi-signing.rsa.pub - the cache key
and the stored filename are identical, producing on-disk paths
shaped like:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Since v1.5.42 (chainguard-dev#294) each such cache entry was declared as a per-file
action output via ctx.actions.declare_file at exactly that path. Some
Bazel sandbox and remote-execution implementations reject declared
output paths whose basename equals their parent directory name, so
the per-file copy actions either fail validation or are reported as
producing missing/invalid outputs even though the cp command itself
succeeded.

Switch the cache-staging step to declare a single TreeArtifact for
the whole cache directory and populate it in one ctx.actions.run_shell
driven by a manifest written via ctx.actions.write. apko sees the
identical files at the identical relative paths, but the problematic
paths are now contents of the TreeArtifact rather than individually
declared outputs, so per-output validation no longer sees them.

The lockfile and the apko binary remain individually declared via
copy_to_workdir; their layout is fixed and never produces the
duplicated-basename pattern.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
)

chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp
actions, declaring each cache entry via ctx.actions.declare_file.
apko's cache is URL-keyed, so when a cached resource's URL ends in
its own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends (observed on BuildBuddy) reject
declared output paths whose basename equals their parent directory
name as REv2 validation failures, so cache staging fails on RBE
even though the cp command itself succeeds.

Declare the whole cache as a single TreeArtifact and populate it in
one ctx.actions.run_shell. apko sees the identical files at the
identical relative paths, but the problematic paths are now contents
of the TreeArtifact rather than individually declared outputs, so
per-output validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file
and directory paths unchanged.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
)

chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp
actions, declaring each cache entry via ctx.actions.declare_file.
apko's cache is URL-keyed, so when a cached resource's URL ends in
its own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends reject declared output paths whose
basename equals their parent directory name as REv2 validation
failures, so cache staging fails on RBE even though the cp command
itself succeeds.

Declare the whole cache as a single TreeArtifact and populate it in
one ctx.actions.run_shell. apko sees the identical files at the
identical relative paths, but the problematic paths are now contents
of the TreeArtifact rather than individually declared outputs, so
per-output validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file
and directory paths unchanged.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
…#294)

chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp
actions, declaring each cache entry via ctx.actions.declare_file.
apko's cache is URL-keyed, so when a cached resource's URL ends in
its own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends reject declared output paths whose
basename equals their parent directory name as REv2 validation
failures, so cache staging fails on RBE even though the cp command
itself succeeds.

Declare the whole cache as a single tree artifact (via
ctx.actions.declare_directory) and populate it in one
ctx.actions.run_shell. apko sees the identical files at the
identical relative paths, but the problematic paths are now contents
of the tree artifact rather than individually declared outputs, so
per-output validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file
and directory paths unchanged.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
…#294)

chainguard-dev#294 switched cache staging from ctx.actions.symlink to per-file cp
actions, declaring each cache entry via ctx.actions.declare_file.
apko's cache is URL-keyed, so when a cached resource's URL ends in
its own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends reject declared output paths whose
basename equals their parent directory name as REv2 validation
failures, so cache staging fails on RBE even though the cp command
itself succeeds.

Declare the whole cache as a single tree artifact (via
ctx.actions.declare_directory) and populate it in one
ctx.actions.run_shell. apko sees the identical files at the
identical relative paths, but the problematic paths are now contents
of the tree artifact rather than individually declared outputs, so
per-output validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file
and directory paths unchanged.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
…#294)

After some more testing we noticed that the per-file cp staging
introduced in chainguard-dev#294 - which replaced ctx.actions.symlink to fix
Bazel 9 sandbox issues - regresses on remote execution.

apko's cache is URL-keyed, so when a cached resource's URL ends in
its own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends reject declared output paths whose
basename equals their parent directory name as REv2 validation
failures, so cache staging fails on RBE even though the cp command
itself succeeds.

Declare the whole cache as a single tree artifact (via
ctx.actions.declare_directory) and populate it in one
ctx.actions.run_shell. apko sees the identical files at the
identical relative paths, but the problematic paths are now contents
of the tree artifact rather than individually declared outputs, so
per-output validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file
and directory paths unchanged.
tiborv added a commit to tiborv/rules_apko that referenced this pull request Apr 27, 2026
…#294)

After some more testing we noticed that the per-file cp staging
introduced in chainguard-dev#294 - which replaced ctx.actions.symlink to fix
Bazel 9 sandbox issues - regresses on remote execution.

apko's cache is URL-keyed, so when a cached resource's URL ends in
its own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends reject declared output paths whose
basename equals their parent directory name as REv2 validation
failures, so cache staging fails on RBE even though the cp command
itself succeeds.

Declare the whole cache as a single tree artifact (via
ctx.actions.declare_directory) and populate it in one
ctx.actions.run_shell. apko sees the identical files at the
identical relative paths, but the problematic paths are now contents
of the tree artifact rather than individually declared outputs, so
per-output validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file
and directory paths unchanged.
xnox pushed a commit that referenced this pull request Apr 27, 2026
After some more testing we noticed that the per-file cp staging
introduced in #294 - which replaced ctx.actions.symlink to fix Bazel 9
sandbox issues - regresses on remote execution.

apko's cache is URL-keyed, so when a cached resource's URL ends in its
own filename - typical for keyrings, e.g.

    https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

the cache key equals the filename and the entry lands at:

    .../wolfi-signing.rsa.pub/wolfi-signing.rsa.pub

Some remote-execution backends reject declared output paths whose
basename equals their parent directory name as REv2 validation failures,
so cache staging fails on RBE even though the cp command itself
succeeds.

Declare the whole cache as a single tree artifact (via
ctx.actions.declare_directory) and populate it in one
ctx.actions.run_shell. apko sees the identical files at the identical
relative paths, but the problematic paths are now contents of the tree
artifact rather than individually declared outputs, so per-output
validation no longer sees them.

copy_to_workdir grows a list-of-Files branch for this; the lockfile,
config, and apko binary continue through the existing single-file and
directory paths unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants