Skip to content

Incremental bazel run + BwoB + --noexperimental_check_output_files = slow #20843

@tjgq

Description

@tjgq

This repro was gently provided by @purkhusid in #6862:

https://github.com/purkhusid/bazel_runfiles_repro/blob/main/reproduce.sh

In short: target T1 has a dependency on T2, which has associated runfiles. First populate a disk cache with the result of building T1. Then, starting clean, first build T1 against the disk cache without the bytes (--remote_download_minimal), then run it. The result is that the runfiles for T2 are missing at runtime (as in, the runfile symlinks dangle). Disabling output checking (--noexperimental_check_output_files) is required to reproduce.

The issue is that building without the bytes works by lying to the Skyframe machinery about the existence of files on disk; instead, the files are materialized when either (1) a consuming action prefetches them, or (2) a top-level target forces its outputs to be produced. bazel run is not a build action, so it can't rely on prefetching; instead, it relies on the target being considered top-level even in MINIMAL mode (see RemoteOutputChecker). Disabling output checking prevents Skyframe from doing the dirtyness check on the outputs and realize they're not actually there, which would cause the target to be rebuilt.

There are two possible fixes:

  1. Add input prefetching to bazel run
  2. Make --noexperimental_check_output_files a no-op for bazel run commands.

(1) is more correct but much trickier to implement, since all of the normal action-running machinery has already been torn down by the time the command is run; it would likely involve reimplementing bazel run as a special action injected into the build graph. I think the pragmatic thing to do is implement (2) until there's more evidence that (1) is worthwhile. (This will make bazel run slightly slower in cases where output checking is truly unnecessary, but it's not obvious that that would lead to a major regression; we're also covered by the fact that --noexperimental_check_output_files is, well, experimental and subject to change behavior at any time.)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions