@scottwittenburg @tgamblin Any clue why this didn't result in a ton of builds in pipelines? Is it because pruning went a bit too far?

By a fast inspection of the e4s environment I see ~200 places where RelWithDebInfo turned into Release into this PR - with all the associated hash changes.

I think having -g is still very useful (which Release seems to lack). Can we somehow make it -O3 -g?

RelWithDebInfo is roughly -O2 -g, whereas Release is -O3 (no -g). See e.g. this answer on StackOverflow.

IMHO there are very few cases where I need the debug information in production code, I'm more concerned about the size of the install tree. If Release would significantly reduce that over RelWithDebInfo I would choose Release.

When I do need debug information (i.e. development codes), I would be fine with rebuilding the world, so Release makes sense as the default for my use cases.

-g is not just for development, it's also for tools like perf specifically for release builds.

Edit: this PR is not the place to discuss it, let's do that in the issue.

I don't think this is the correct thing to do. If anything, I think it is better to add Spack documentation that talks about configuring builds to be fully optimized for Release than it is to upset the apple cart by pulling debug info out from under all builds in spack by default. I get the incentive to use Release, but if people are to a point they trust a spack deployment enough to remove all debug info, they are likely also at a point they can add a couple lines in their packages.yaml to do the same thing.

Building with Release is not a good default.

I don't think it is very straightforward for the user to switch to packages: all: variants: build_type=Release. Using preferences, the concretizer doesn't care at all and we still get build_type=RelWithDebInfo all over the DAG for some reason I haven't figured out. I guess my question is packages: all: require: "build_type=Release" the up-to-date way I should do this? When trilinos builds as RelWithDebInfo we get 3GB executables, so that doesn't make RelWithDebInfo a good default either in my opinion. I've come to the conclusion there is no good default.

I've come to the conclusion there is no good default.

Depending on the use case for Spack different options are preferential. RelWithDebInfo is kind of the in-between that doesn't totally satisfy everyone but is also more or less acceptable in >90% of the cases. Release removes information that people may want and Debug removes basic optimizations that make a lot of the tools usable.

I guess my question is packages: all: require: "build_type=Release" the up-to-date way I should do this?

Yes, preferences that are actually requirements can be specified that way. You can also specify the requirement per-package too.

packages:
  trillinos:
    require: ["build_type=Release"]

Note, I would recommend using lists there instead of strings for composition purposes.

Here there are real numbers in terms of size reduction: llnl/smith#764

I just tried and it looks like:

packages:
  all:
    require: ["build_type=Release"]

doesn't work with non-CMakePackage packages. So it appears I'm stuck using preferences again, which the concretizer ignores. Unless I want to put requires: build_type=Release for every CMakePackage in my packages.yaml.

The size reduction argument is pretty persuasive to me...I am willing to let go on my stance for a 20x reduction in binary footprint, that is significantly more impactful than -O2 vs -O3 optimization.

+1

Sorry, I'm not convinced, even if there are 3 ✅s. How much of that "size argument" holds if you inject -gz?

You can keep debug symbols compressed. I still think we want debug symbols in CI, it's useful for debugging segfaults, it doesn't really matter if source is preserved or not, at least you have symbol names.

IMHO the upside of Release is enabling vectorization in GCC, even though that's already partially (?) enabled in GCC >= 12 in -O2. Losing debug info is really an annoyance.

Then for the CI, how about:

packages:
  all:
    variants: build_type=RelWithDebInfo

... at least you have symbol names.

CMake does not by default strip symbols from binaries even for Release (StackOverflow, demo). Spack would need to pass --strip to a manual call to cmake --install which it does not do. If the binary isn't stripped, all internal symbols are retained and will be available for a backtrace.

Just for a counter point, I have had CI builds fail due to running out of disk space. This mainly happened in Docker builds but the logs were lost and it was a giant pain to figure out.

IMO, debugging CI jobs directly from their output (except the logs to narrow it down locally) is usually not worth the time.

But more importantly, your CI should as closely reflect your user experience as possible.

I still think we want debug symbols in CI

I think for PR CI this can make sense, but for protected/develop/release CI I don't think there is is as much utility. Since those are really being used to generate build caches, I would argue that it is not really desired.

For CI, if we can figure out how to get require: ["build_type=RelWithDebugInfo"] to work reliably then it should be an easy thing to add in the generate script.

CMake does not by default strip symbols

CMake has nothing to do with the example you give. Your example doesn't declare the function static, so of course it will be part of the symbol table.

If the binary isn't stripped, all internal symbols are retained and will be available for a backtrace.

This is incorrect in the sense that there isn't a symbol for static functions, so you get very tricky if not useless backtraces.

but for protected/develop/release CI I don't think there is is as much utility.

I still think there is:

1. Binaries will segfault on a different system, and w/o debug symbols you get very poor bug reports ("it segfaults")
2. People run perf, debug symbols are not just for debugging

And to sum up the pros and cons:

build_type=Release

👍 higher optimization level, including loop vectorization on older GCC
👍 adds NDEBUG define, which disables assertions, which could cause speedups if assertions are in loops etc
👍 no -g means smaller install size
👎 bad backtraces
👎 perf reports may be useless
👎 no function arguments / local variables in debugger (could be of course)
👎 no file path / line numbers in debugger

Notice that debug symbols do not influence performance.

Other than that:

1. Nobody has reported yet on -gz file sizes, we could consider passing that flag by default through the compiler wrapper
2. I don't see why packages:all:require:build_type=... should fail at all, if I remember correctly we ensure that that requirement is only enforced if the requirement exists for the package; if the problem is that the value isn't validated, then we should probably do that too. I think this should Just Work as a requirement in all.

Your example doesn't declare the function static, so of course it will be part of the symbol table.

Symbols are not exposed by default for executables (see ld --export-dynamic), but I've updated the gist with a nm -D call and a static case to make the compiler's behavior more apparent.

This is incorrect in the sense that there isn't a symbol for static functions, so you get very tricky if not useless backtraces.

There are symbols for the outlined version of static functions. You are correct that there are no symbols for inlined functions, data about inlined functions is only present in the debug info. Without debug info inlined calls appear as part of their caller in the backtrace.

I still think there is:

1. Binaries will segfault on a different system, and w/o debug symbols you get very poor bug reports ("it segfaults")

I personally refuse to debug someone else's code without the original sources. The buildcache does not ship the sources (obviously), so I'd first rebuild from source. If I'm rebuilding from source anyway, I can set build_type=Debug.

Poor bug reports will be reported no matter what. There is (a lot of) software built outside of Spack, and (a lot of) it is stripped on installation (my common encounters involve GPU or MPI vendor libraries or Glibc itself). Keeping debug info here will not help the situation.

2. People run perf, debug symbols are not just for debugging

I do not think people who care about performance will use the Spack public buildcache for their main build. As a dev for HPCToolkit I've learned that they want software built -march=native. That means they're building most of their software stack from source. We already tell our users to build with -g for the parts they are interested in gathering performance for, we can tell them about build_type=RelWithDebInfo in the same breath.

FWIW, most application teams we interact with are only interested in detailed performance for their code, they really couldn't care less about the tens of inlined templates in their library dependencies (RAJA/Kokkos, *BLAS, etc.). They should only need debug info for a few (often one) element of a large Spack spec, more than that isn't worth the storage quota.

@alalazo fixed (and double-checked the rest of them).

@tgamblin addressed

@spackbot run pipeline

@spackbot run pipeline

I've started that pipeline for you!