Description of the problem / feature request:
Bazel does not deduplicate file uploads across actions, so it may attempt multiple uploads of the same content in parallel. If these are large files, then this causes significant network overhead.
For remote builds of TensorFlow over the public internet, this makes the build take a very long time.
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
[genrule(
name = "bar-%s" % i,
srcs = ["input.txt"],
outs = ["bar-%s.txt" % i],
cmd = "cp $< $@",
) for i in range(0, 100)]
Enable a remote gRPC cache or executor.
What operating system are you running Bazel on?
Linux.
Have you found anything relevant by searching the web?
This was previously also reported here, but I could not find a matching bug report:
bazelbuild/remote-apis#131 (comment)
@EdSchouten
Description of the problem / feature request:
Bazel does not deduplicate file uploads across actions, so it may attempt multiple uploads of the same content in parallel. If these are large files, then this causes significant network overhead.
For remote builds of TensorFlow over the public internet, this makes the build take a very long time.
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Enable a remote gRPC cache or executor.
What operating system are you running Bazel on?
Linux.
Have you found anything relevant by searching the web?
This was previously also reported here, but I could not find a matching bug report:
bazelbuild/remote-apis#131 (comment)
@EdSchouten