Skip to content

stdenv: Enable PIE by default#252310

Closed
chivay wants to merge 1 commit intoNixOS:stagingfrom
chivay:enable-pie-kekw
Closed

stdenv: Enable PIE by default#252310
chivay wants to merge 1 commit intoNixOS:stagingfrom
chivay:enable-pie-kekw

Conversation

@chivay
Copy link
Member

@chivay chivay commented Aug 30, 2023

Description of changes

Almost exactly 7 years ago, a large security-related PR that introducted hardening was merged - #12895.

In 2016, enabling PIE was a bit controversial as it could break some packages and introduce overhead on 32 bit architectures.
Unfortunately, this decision lead us to the current state of nixpkgs where only a handful of packages (~20) enables PIE by default.

I think that we should revisit this discussion and enable PIE by default for all packages, leaving the option to opt-out of PIE if it causes crashes or broken packages. Other large distros have already done this, see:

https://wiki.ubuntu.com/Security/Features#pie
https://wiki.gentoo.org/wiki/Hardened/FAQ
https://wiki.archlinux.org/title/Arch_package_guidelines/Security#PIE

I'm aware that this PR won't be merged as-is since I don't have enough experience with stdenv, but it should be at least a humble beginning to a larger fix.

Kudos to @BonusPlay for pointing out this issue.

Fixes #205031

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@chivay chivay requested a review from a user August 30, 2023 09:14
@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Aug 30, 2023
@chivay chivay added the 1.severity: security Issues which raise a security issue, or PRs that fix one label Aug 30, 2023
@chivay
Copy link
Member Author

chivay commented Sep 4, 2023

After some digging, I've discovered the issue behind failed ofborg builds:

checking for gcc... gcc
checking whether the C compiler works... no
checking whether the C compiler works... no
configure: error: in `/build/bash-5.2':
configure: error: C compiler cannot create executables
See `config.log' for more details
configure: error: in `/build/bash-5.2':
configure: error: C compiler cannot create executables

It turns out that the bootstrap tools are missing a Scrt1.o file from glibc. After applying following patch and swapping the bootstrap files bash was built correctly.

$ nix build .#bash && checksec --file result/bin/bash
warning: Git tree '/home/chivay/repos/nixpkgs' is dirty
[*] '/home/chivay/repos/nixpkgs/result/bin/bash'
    Arch:     amd64-64-little
    RELRO:    Full RELRO
    Stack:    Canary found
    NX:       NX enabled
    PIE:      PIE enabled
    RUNPATH:  b'/nix/store/r01cc12xxvq8cn75ih6pq4spk6a1dyai-glibc-2.37-8/lib'
    FORTIFY:  Enabled

diff --git a/pkgs/stdenv/linux/make-bootstrap-tools.nix b/pkgs/stdenv/linux/make-bootstrap-tools.nix
index d6c4da0ab2be..03f218c5d143 100644
--- a/pkgs/stdenv/linux/make-bootstrap-tools.nix
+++ b/pkgs/stdenv/linux/make-bootstrap-tools.nix
@@ -79,6 +79,7 @@ in with pkgs; rec {
         cp -d ${libc.out}/lib/libnss*.so* $out/lib
         cp -d ${libc.out}/lib/libresolv*.so* $out/lib
         cp -d ${libc.out}/lib/crt?.o $out/lib
+        cp -d ${libc.out}/lib/Scrt*.o $out/lib

         # Hacky compat with our current unpack-bootstrap-tools.sh
         ln -s librt.so "$out"/lib/librt-dummy.so

@msm-code
Copy link
Contributor

msm-code commented Sep 7, 2023

Looking forward to this!

As a security engineer it's a bit weird to use NixOS as my main system, since it's clear that security is not the priority. Which is OK - plenty of security focused distros already, and we reproducibility fams may get Spectrum one day. But there are things one expects from "security focused" distros (like QubesOS), and there are things that constitute the status quo, the ground level. PIE is one of those things - it's a simple enhancement, supported by every major distribution already, and it's a major security improvement (TLDR: arbitrary write primitive is not enough for exploitation anymore, one needs to leak ASLR first so a separate vulnerability is usually needed).

I'm happy to help testing it if necessary. But can one of the maintainers weigh in on this enhancement first?

@chivay
Copy link
Member Author

chivay commented Sep 7, 2023

After some further experimentation I've discovered additional quirks - although bash is built correctly, some other packages like python3 end up without PIE (and for some unrelated reason also no stack canaries?)

$ checksec --file result/bin/python3
[*] '/home/chivay/repos/nixpkgs/result/bin/python3'
    Arch:     amd64-64-little
    RELRO:    Full RELRO
    Stack:    No canary found
    NX:       NX enabled
    PIE:      No PIE (0x400000)
    RUNPATH:  b'/nix/store/cn63ysfl4jk78hgkbz50qg626m0qnpid-python3-3.10.12/lib:/nix/store/y6139wxl897ghqwy002i54jqpzci2ygy-libxcrypt-4.4.36/lib:/nix/store/r01cc12xxvq8cn75ih6pq4spk6a1dyai-glibc-2.37-8/lib:/nix/store/4784z5x4jqqr24i0shizglvzvslkq1f5-gcc-12.3.0-lib/lib'

Lack of PIE seems to be caused by inconsistent NIX_HARDENING_ENABLE defaults used in shell scripts.

grepping over this variable reveals:

$ rg NIX_HARDENING_ENABLE= pkgs/build-support/
pkgs/build-support/cc-wrapper/fortran-hook.sh
8:: ${NIX_HARDENING_ENABLE="stackprotector pic strictoverflow relro bindnow"}

pkgs/build-support/bintools-wrapper/setup-hook.sh
68:: ${NIX_HARDENING_ENABLE="fortify stackprotector pic strictoverflow format relro bindnow"}

pkgs/build-support/cc-wrapper/setup-hook.sh
114:: ${NIX_HARDENING_ENABLE="fortify fortify3 stackprotector pic strictoverflow format relro bindnow"}

Adding pie to cc-wrapper/setup-hook.sh fixes the issue. Although IMO this should be defined in a nix file as a single source of truth.

@RaitoBezarius
Copy link
Member

In those situations, I advise adding at least the two following group of reviewers:

(a) low-level stdenv contributors/developers, e.g. @trofi @Artturin (@amjoseph-nixpkgs does a lot of work in this area those days too and is already included, @Ericson2314 is an historical contributor and is still doing work in those areas AFAIK)

(b) security team, e.g. @risicle who has been driving a lot of hardening enablement (fortify source 3 comes to mind)

(c) bonus, staging people because this one will probably a massive breakage incoming, cc @vcunat @mweinelt @K900 who were recently involved in staging rounds.

In either case, this should definitely get its own Hydra jobset for pre-evaluation of the breakage, and then the classical discussion ensues on how to sort out this and what to do.

For record, I am in favor of this, I just think we need the human resources to opt-out/fix the hardening for broken packages.

@trofi
Copy link
Contributor

trofi commented Sep 7, 2023

You might want to include all runtime *.o files from glibc (that's what we do for musl bootstrap):

$ ls /nix/store/905gkx2q1pswixwmi1qfhfl6mik3f22l-glibc-2.37-8/lib/*.o |& unnix
/<<NIX>>/glibc-2.37-8/lib/crt1.o
/<<NIX>>/glibc-2.37-8/lib/crti.o
/<<NIX>>/glibc-2.37-8/lib/crtn.o
/<<NIX>>/glibc-2.37-8/lib/gcrt1.o
/<<NIX>>/glibc-2.37-8/lib/grcrt1.o
/<<NIX>>/glibc-2.37-8/lib/Mcrt1.o
/<<NIX>>/glibc-2.37-8/lib/rcrt1.o
/<<NIX>>/glibc-2.37-8/lib/Scrt1.o
  • Scrt - PIE
  • rcrt - static-PIE
  • gcrt - profile-genrated
  • Mcrt - unused nowadays.

Otherweise we would need to re-cut bootstrapFiles again when someone finds that we could use static-pie or PGO builds in more contexts.

@risicle
Copy link
Contributor

risicle commented Sep 7, 2023

I echo the general approval of this idea.

Firstly we will probably find it helpful if more/most packages have tests or we find a way of automatically testing binaries for basic-execution-without-crashing.

Secondly as you find problems it may be good to try narrowing them down into a test which you can add to tests.hardeningFlags (recently merged, obligatory plug for own PR fixing them up a bit #253186)

chivay added a commit to chivay/nixpkgs that referenced this pull request Sep 7, 2023
Include all runtime object files in output package, enabling different
kinds of build modes - non-PIE, PIE, static PIE and profile-generated.

Suggested by @trofi:
NixOS#252310 (comment)
@risicle
Copy link
Contributor

risicle commented Oct 5, 2023

Also please see #259070 where I've refactored how defaultHardeningFlags is determined and would change the place you have to make this alteration to pkgs/build-support/bintools-wrapper/default.nix.

@LunNova
Copy link
Member

LunNova commented Oct 10, 2023

Lack of PIE seems to be caused by inconsistent NIX_HARDENING_ENABLE defaults used in shell scripts.

#206490 or a similar change should help with it. I never had the time to push that PR forward.

Copy link
Member

@LunNova LunNova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without #206490 (or #259070) this PR will do nothing to many derivations, as enabledHardeningOptions is not used unless hardeningDisable != [] || hardeningEnable != [] || stdenv.hostPlatform.isMusl, and it falls back to defaults in various build-support scripts.

@risicle
Copy link
Contributor

risicle commented Oct 12, 2023

and/or #259070 which unifies the set of defaults with those used by the build-support scripts ;)

@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025
@FliegendeWurst
Copy link
Member

With this change, some builds using clang log clang++: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument] per file compiled. I am not really sure why. Maybe the build already specifies -pie so our second -pie gets ignored?

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 4, 2025
@chivay
Copy link
Member Author

chivay commented Jan 4, 2025

With this change, some builds using clang log clang++: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument] per file compiled. I am not really sure why. Maybe the build already specifies -pie so our second -pie gets ignored?

I could be wrong but it seems that it might be happening when building shared libraries.

$ clang -shared -pie /dev/null
clang: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]

It is a bit ugly but won't break anything. Ideally, hardening scripts should drop this flag when -shared is passed to the compiler

@FliegendeWurst
Copy link
Member

I suppose we would pass -fpic instead, but that is already the default.

@LunNova
Copy link
Member

LunNova commented Jan 4, 2025

I think clang defaults to pie on now, (llvm/llvm-project@ca68038) so do we need to add an arg at all when the CC is clang?

Other distros have been shipping GCCs built with --enable-default-pie which makes gcc default pie on but we don't.

@FliegendeWurst
Copy link
Member

For clang version < 15, I guess. That commit re-applies a reverted commit, so I'm not sure what the default is for older versions.

@chivay
Copy link
Member Author

chivay commented Jan 4, 2025

do we need to add an arg at all when the CC is clang?

We probably don't need to in newer versions, but handling different compilers and versions could get messy.

@SigmaSquadron SigmaSquadron added the 1.severity: significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Jan 4, 2025
@emilazy
Copy link
Member

emilazy commented Jan 5, 2025

Might be best to just leave old Clang versions alone as they’re only used for picky packages anyway. Energy put into them is often best directed to unpinning them across the tree so we can eventually drop them.

@FliegendeWurst
Copy link
Member

Some packages even fail due to clang: error: argument unused during compilation: '-pie' [-Werror,-Wunused-command-line-argument], so it certainly needs to be addressed.
pkgs/build-support/cc-wrapper/add-hardening.sh already has some logic to detect -shared, so I'm not sure what the right fix would be.

@Pandapip1
Copy link
Member

Semi-Related: #356058; the python drv doesn't respect NIX_ variables

@chivay
Copy link
Member Author

chivay commented Jan 22, 2025

Some packages even fail due to clang: error: argument unused during compilation: '-pie' [-Werror,-Wunused-command-line-argument], so it certainly needs to be addressed. pkgs/build-support/cc-wrapper/add-hardening.sh already has some logic to detect -shared, so I'm not sure what the right fix would be.

@FliegendeWurst could you give an example of a failing package?

@emilazy
Copy link
Member

emilazy commented Jan 22, 2025

Probably better to just turn off the silly -Werror for such packages.

@FliegendeWurst
Copy link
Member

could you give an example of a failing package?

For example clang-uml or c3c.

@emilazy
Copy link
Member

emilazy commented Jan 22, 2025

Another option would be to just configure compilers to enable PIE by default rather than injecting -pie in our wrappers (and then explicitly disabling it when it’s in hardeningDisable). I would prefer we move in that direction in general, but of course it requires more work than this current diff.

@chivay
Copy link
Member Author

chivay commented Jan 22, 2025

From a quick test, seems non-trivial. After adding:

diff --git a/pkgs/development/compilers/gcc/common/configure-flags.nix b/pkgs/development/compilers/gcc/common/configure-flags.nix
index 25d4f1f53bae..83cd78f49fce 100644
--- a/pkgs/development/compilers/gcc/common/configure-flags.nix
+++ b/pkgs/development/compilers/gcc/common/configure-flags.nix
@@ -161,6 +161,7 @@ let
       "--without-included-gettext"
       "--with-system-zlib"
       "--enable-static"
+      "--enable-default-pie"
       "--enable-languages=${
         lib.concatStringsSep ","
           (  lib.optional langC        "c"

I'm getting a build failure in xgcc during configure stage

...
checking whether /build/build/./gcc/xgcc -B/build/build/./gcc/ -O2 -I/nix/store/137sa1w4glbbs69bv1xidc3v2vxwhg93-bootstrap-stage0-glibc-bootstrapFiles/include -B/nix/store/137sa1w4glbbs69bv>
checking for /build/build/./gcc/xgcc -B/build/build/./gcc/ -O2 -I/nix/store/137sa1w4glbbs69bv1xidc3v2vxwhg93-bootstrap-stage0-glibc-bootstrapFiles/include -B/nix/store/137sa1w4glbbs69bv1xid>
checking how to run the C preprocessor... /build/build/./gcc/xgcc -B/build/build/./gcc/ -O2 -I/nix/store/137sa1w4glbbs69bv1xidc3v2vxwhg93-bootstrap-stage0-glibc-bootstrapFiles/include -B/ni>
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking size of double... 8
checking size of long double... g++  -fPIE  -o g++-mapper-server server.o resolver.o ../libcody/libcody.a ../libiberty/pic/libiberty.a
/nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/ld: cannot find ../libiberty/pic/libiberty.a: No such file or directory
...

@azahi azahi removed their request for review April 26, 2025 22:33
@nyabinary

This comment was marked as off-topic.

"strictoverflow"
] ++ lib.optional (with stdenvNoCC;
# Musl-based platforms will keep "pie", other platforms will not.
# If you change this, make sure to update section `{#sec-hardening-in-nixpkgs}`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't help but notice that this is not updated in this PR. Will it be updated somewhere else?

@vcunat
Copy link
Member

vcunat commented Oct 7, 2025

I think this got superseded by PR #439314 which is now in nixos-unstable (25.11).

@vcunat vcunat closed this Oct 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.severity: security Issues which raise a security issue, or PRs that fix one 1.severity: significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. 2.status: merge conflict This PR has merge conflicts with the target branch 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.