I wasn't aware of this, but it appears that Package subclasses can override the variants of their parents.

FYI (should you be interested in the code) I added that back in #2623 together with the support for inheriting directives.

@scheibelp thanks for the quick fix! It seems the CMakePackage setup is a lot more convenient than I realized, it's really quite nice that you can just override the variant like that!

Honestly I would go with Release by default - currently we remove the sources we built from after a successful installation

@tgamblin explained this to me in #3966

Most tools you do want RelWithDebInfo. Consider something like OpenCV. As a developer, if everything is compiled as Release and you're trying to debug your code, you have no options. There are no debugging symbols, so you can't step through library boundaries. For most projects, the overhead of RelWithDebInfo is not noticeable. LLVM is a very special case. It is probably the most special snowflake of them all!

There may be other packages that should be defaulting to Release, but the LLVM one in particular should be something users don't even have to think about.

Most tools you do want RelWithDebInfo. Consider something like OpenCV. As a developer, if everything is compiled as Release and you're trying to debug your code, you have no options.

@svenevs I may agree with that (even though debugging a release build is a lysergic experience due to compiler optimizations). The fact is that we normally delete the source files after a build, or eventually we relocate them. I don't know how nicely that plays with the debug info in the binary objects.

The fact is that we normally delete the source files after a build, or eventually we relocate them. I don't know how nicely that plays with the debug info in the binary objects.

Noting first that my word is far from law on this, my experience as a C++ developer is this:

1. If a library I'm depending on is pure Release, I literally cannot even step through my own program when that library is getting called. This is because gdb / lldb have nowhere to go.

   • So the first try, your debugger will fail out on you.
   • The second try, you have to consciously step over (n), or setup a new breakpoint.

2. RelWithDebInfo makes it so that I can at least step through. This is particularly useful because if you are calling their API incorrectly, you can e.g. step through to the very point where (because you called their API wrong) they emit a segfault.

   • With just debugging symbols, it's a truly arcane process with C++ mangled names that are barely comprehensible.
   • But you can, at the very least, step through.

I think what you're getting confused about is the compilation process. Say I have two files A and B that are ultimately getting linked into libfoo.so. Separate compilations of A and B (producing say A.o and B.o) will indeed be deleted once the install is finished. However, in CMake because you did add_library with A and B resulting in libfoo.so, the picture you should have in mind is basically that A.o and B.o get concatenated together to form libfoo.so. This is why you are able to delete A.o and B.o and still use libfoo.so.

This is in many ways an embarrassing oversimplification, but the idea is that the final library that gets created still contains the debugging symbols, because when A and B were compiled we asked for them (via RelWithDebInfo).

Edit: or you're completely on the money, and my explanation only applies to static libraries (libfoo.a). So yeah, now I'm just as confused...

@svenevs

If a library I'm depending on is pure Release, I literally cannot even step through my own program when that library is getting called.

Not true. You can step into the library call and see machine code. Not sure how much that will be useful though. Anyhow you can still set breakpoints on functions exported in header files.

The fact is: as far as I know debugging symbols introduce references back to the original source files to map the current instruction to a line you coded. Try to play with:

// hello.h
void hello();

// hello.c
#include "hello.h"

#include <stdio.h>
#include <stdlib.h>

void hello(char * continuation) {
    char * greeting = "Hello";
    printf("%s %s\n", greeting, continuation);
}

// main.c
#include "hello.h"

int main() {
  char * final = "world!";
  hello(final);
  return 0;
}

and compile the thing like:

$ gcc -c -Wall -Werror -fPIC -O2 hello.c
$ gcc -shared -o libhello.so hello.o
$ gcc -L$PWD -Wl,-rpath=$PWD -Wall -Werror  -O2 -o test main.c -lhello

If you run gdb you can still set breakpoint, but you'll have no useful information from that.

(gdb) break main
Breakpoint 1 at 0x4005c0
(gdb) break hello
Breakpoint 2 at 0x4005a0
(gdb) run
Starting program: /home/mculpo/tmp/debug/test 

Breakpoint 1, 0x00000000004005c0 in main ()
(gdb) continue
Continuing.

Breakpoint 2, 0x00007ffff7bd86f0 in hello () from /home/mculpo/tmp/debug/libhello.so
(gdb) bt full
#0  0x00007ffff7bd86f0 in hello () from /home/mculpo/tmp/debug/libhello.so
No symbol table info available.
#1  0x00000000004005d0 in main ()
No symbol table info available.
(gdb) continue
Continuing.
Hello world!
[Inferior 1 (process 4866) exited normally]

Now if you compile with debugging info:

$ gcc -g -c -Wall -Werror -fPIC -O2 hello.c
$ gcc -shared -o libhello.so hello.o
$ gcc -g -L$PWD -Wl,-rpath=$PWD -Wall -Werror  -O2 -o test main.c -lhello

and then you rename or delete hello.c (which mimics sources that are not there anymore):

Reading symbols from ./test...done.
(gdb) break main
Breakpoint 1 at 0x4005c0: file main.c, line 3.
(gdb) break hello
Breakpoint 2 at 0x4005a0
(gdb) run
Starting program: /home/mculpo/tmp/debug/test 

Breakpoint 1, main () at main.c:3
3	int main() {
(gdb) continue
Continuing.

Breakpoint 2, hello (continuation=continuation@entry=0x400754 "world!") at hello.c:8
8	hello.c: No such file or directory.
(gdb) bt full
#0  hello (continuation=continuation@entry=0x400754 "world!") at hello.c:8
        greeting = 0x7ffff7bd8719 "Hello"
#1  0x00000000004005d0 in main () at main.c:5
        final = 0x400754 "world!"
(gdb) continue
Continuing.
Hello world!
[Inferior 1 (process 4914) exited normally]

The missing source file causes the hello.c: No such file or directory. message and so you cannot map the point of execution back to the source file (which is at least for me the most useful thing a debugger can provide you).

So my point is: should we generally pay for more space due to debug symbols if the state of things we are creating is that of the second example?

Thanks for the reviews! And thanks to @alalazo for making this easy!

@alalazo I stand both corrected and disgraced, humbled by your strictly superior understanding and prowess.

As it turns out the reason why I had this belief is because the only times I've actually gone through and debugged stuff like this is using git submodules, for which the source files were indeed still around!

This has all been quite illuminating.

Honestly I would go with Release by default - currently we remove the sources we built from after a successful installation

Even if we delete source after successful installation, RelWithDebInfo will be still useful because : Spack will tell use the exact version of the package, we download the source code and tell debugger to map the source code to newly downloaded source directory (e.g. gdb's set directories path-list or set substitute-path from to commands Or allinea ddt's --source-dirs command Or Totalview has dset command)

Unless I mistaken something...

@pramodk True, but did you consider:

1. it is far from ideal - even if you tell spack to copy the sources with --source you still need to configure your debugger properly
2. you'll be debugging a Release application - this means you'll see your program jumping around in the source code, or that x = y could very well modify x AND some other variable or other amenities
3. if you use the wrong sources after the fact the code will be mapped to the wrong lines

?

Anyhow, maybe it's better not to pollute this thread (sorry @scheibelp) and continue the discussion on Slack?`

maybe it's better to continue the discussion on Slack?`

I think a github issue might be better, since you have more formatting options. I'll add one and refer to this and #5258