zig build: separate the maker process from the configurer process #35428
No reviewers
Labels
No labels
abi/f32
abi/ilp32
abi/sf
accepted
arch/21k
arch/6502
arch/aarch64
arch/alpha
arch/amdgcn
arch/arc
arch/arc32
arch/arc64
arch/arm
arch/avr
arch/bfin
arch/bpf
arch/colossus
arch/cris
arch/csky
arch/dlx
arch/epiphany
arch/fr30
arch/frv
arch/hexagon
arch/hppa
arch/hppa64
arch/ia64
arch/kalimba
arch/kvx
arch/lanai
arch/lm32
arch/loongarch32
arch/loongarch64
arch/m32r
arch/m68k
arch/m88k
arch/mcore
arch/microblaze
arch/mips
arch/mips64
arch/mmix
arch/moxie
arch/mrisc32
arch/msp430
arch/nds32
arch/ns32k
arch/nvptx
arch/or1k
arch/powerpc
arch/powerpc64
arch/propeller
arch/riscv32
arch/riscv64
arch/rl78
arch/rx
arch/s390x
arch/sh
arch/sparc
arch/sparc64
arch/spirv
arch/spu
arch/tricore
arch/v850
arch/vax
arch/vc4
arch/ve
arch/wasm
arch/x86
arch/x86_64
arch/xcore
arch/xtensa
autodoc
backend/c
backend/llvm
backend/self-hosted
binutils
breaking
build system
debug info
docs
error message
frontend
fuzzing
incremental
lib/c
lib/compiler-rt
lib/cxx
lib/std
lib/tsan
lib/ubsan-rt
lib/unwind
linking
miscompilation
os/android
os/contiki
os/dragonfly
os/driverkit
os/emscripten
os/freebsd
os/fuchsia
os/haiku
os/hermit
os/hurd
os/illumos
os/ios
os/linux
os/maccatalyst
os/macos
os/managarm
os/netbsd
os/ohos
os/openbsd
os/plan9
os/redox
os/rtems
os/serenity
os/tvos
os/uefi
os/visionos
os/wasi
os/watchos
os/windows
proposal
release notes
testing
zig cc
zig fmt
zig reduce
bounty
bug
contributor-friendly
downstream
enhancement
infra
optimization
question
regression
upstream
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
ziglang/zig!35428
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "build-runner-process"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
closes #31397
closes #31691
Release Notes
The primary motivation is to make
zig buildfaster, both in the sense that it will take less time to compile, because only the user'sbuild.ziglogic is being recompiled when it changes, and in the sense that the build runner is built with optimizations enabled, which is starting to become more valuable now that we have introduced--watchand--fuzz. Furthermore, the build.zig logic can be skipped sometimes depending on what CLI flags are used withzig build.This changeset heavily reworks the internal mechanism of the zig build system, however, it is mostly non-breaking from an API perspective, with the exceptions noted below.
Third-party tooling such as ZLS will benefit from consuming the serialized configuration file rather than maintaining a fork of the build runner.
New CLI Arguments
Run Step: Passthru Args
In the Run step, passthru args are all together now, not observable in
configure phase whether run args are provided.
⬇️
This removes a capability from build scripts since they can no longer observe
those arguments. In exchange, it means that when changing those arguments,
build scripts no longer must be rebuilt from source.
Fmt Step: Options
pathsandexclude_pathsare now LazyPath lists. There is a convenience method to create them:b.pathList.std.Build API
b.build_root(Directory) ->b.root(Path)ConfigHeader.Options:include_guard_override->include_guardLazyPath:getDisplayName->format("{f}")LazyPath.basename: removed since the value is not known until make phaseb.findProgramdivided intofindProgramandfindProgramLazyand API future-proofed.ConfigHeadernow properly reports unused values for all stylesstd.Target API
Build System Behavior
Perf Data Point:
zig build -h(cached)Perf Data Point: Running a subset of the toolchain test suite
Fresh cache except in both cases I let the build runner / maker / configurer
get cached beforehand.
master:
branch:
It's only one data point, but the new thing is faster and less memory. As is
expected from being compiled in optimized mode. That's the point of the
changeset.
Introduce the Concept of Configure Cache Poisoning
If the cache is poisoned means that the configure logic had side
effects, or otherwise did something that could not be tracked by the
cache system.
This is not to be confused with whether individual steps may have side
effects when being evaluated; it has to do with the logic inside build.zig
itself. For example, a
Runstep that prints "hello world" has sideeffects at make time and therefore does not warrant setting this flag,
while checking for the existence of
scdocat configure time in order tochoose the default value for a configuration option does.
Keeping the cache pure will make
zig buildfaster, bypassing theconfigurer process when identical configuration would be generated.
When the cache is poisoned, the maker process will delete the build
configuration file upon ingesting it since it cannot be reused.
Currently the only way to poison the cache is to call
findProgram.Advanced users can override the cache poisoning behavior with a CLI option:
findProgram
Immediately (in the configure phase), searches for an executable on the host
that has more than one possible name.
Names are searched in order, observing search prefixes first and then PATH
environment variable.
Calling this function poisons the configuration cache, so it is only
appropriate when the existence of the program or its output needs to be
observed by configuration logic. That's why there is a lazy variant of this function now.
findProgramLazy
Creates an anonymous
Stepthat searches for an executable on the host thathas more than one possible name.
Returns the
LazyPathof the found executable. The search only takes placeif the
LazyPathwill be used by a dependingStep.This API is useful in the following cases:
vs "python3").
globally installed and will therefore be possibly found in one of the
search prefix paths.
Removed Ability to Override Build Runner
There is no concept of a "build runner" any more; it has been split in two:
configurer and maker.
Users of this feature will likely want to no longer override any logic, and
instead consume the configuration file produced by configurer.
If overriding one or the other of these is desired, the feature will need to be
re-introduced (as --override-maker or --override-configurer).
Followup Issues
from the install step.
some values being strings and some values being LazyPaths.
will integrate properly with the cache system.
stat. if already exists and length zero, skip the open() call and track the
"all_cached" flag properly like the surrounding code.
directly rather than from relying on it being a Compile step. This will make
this logic work even for the edge case that the binary was produced by a
third party.
offer a command that will attempt to make a reproducer.
Is there a long-term reason that that Maker needs to be compiled at first use, and not simply being built in like
zig makeror whatever? (I understand that it is much easier to hack on it during development this way!). As it lives in global cache and not per-project cache, it doesn't depend on any project specific options, no?@bfredl I think the reasoning here would be the same as for other subcommands like
zig cc,zig libc,zig objcopyetc. being compiled on-demand instead of included in thezigbinary. As far as I'm aware:zigitself35a98f2ba70d5f203baf@andrewrk wrote in #35428 (comment):
Isn't this change going to cause transitive dependency DLLs to fail to load?
The difference is not in recursiveness, it is in whether the logic is done for only argv[0], or for argv[0] and also artifact args added to argv.
a355bd316ea27b0ba5c5a27b0ba5c5ecf7c2f621ecf7c2f6218ce996a8c98ce996a8c94438a3faa43beaf781ca9571cba45b9571cba45ba9e0eb5340zig buildCLI flag to print path to the configuration file #35498zig build#433