Triaged in Nix team meeting:

• @roberth tried to work around some failures and also had design questions
  • it should probably follow the convention we already have for the unit tests, even if we're not building much
  • @Ericson2314: instead of copying the sources to the build directory we could actually "install" them
• @roberth: this also needs to work in the VM test
  • can definitely be done
  • would be fantastic if we could just package the tests, but maybe it's just not how it works
  • @Ericson2314: we could absolutely do both, but there's still a lot of reworking to get the functional tests how we want them

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-07-10-nix-team-meeting-minutes-160/49101/1

@GoldsteinE might you be interested in helping out with this? I think I just (hackily) fixed the CI eval issue so the actual interesting failures will show up.

The differences between the Meson and Make bash environments for the functional tests is...very subtle! :( It makes debugging these failures hard.

I’ll take a look, but I don’t really know Meson.

Unrelated, but I think one of the end goals here should be rewriting functional tests to something that is not bash. Writing logic in bash is extremely error-prone and I think replacing it with e.g. Python would make all of this logic much less brittle.

@GoldsteinE Thank you!!

Unrelated, but I think one of the end goals here should be rewriting functional tests to something that is not bash.

I agree! I wrote up #10823 for this.

The PRs that this depends on should be reviewed separately, but then it's good to go!

🎉 All dependencies have been resolved !

It seems that the macOS failure is due to libcurl is failing to initialize:

2024-08-13T14:04:56.2756290Z nix-functional-tests> building '/private/tmp/nix-build-nix-functional-tests-2.25.0pre20240813_1df9fb0.drv-0/nix-test/check/store/98i8amk2i6hj4h9w9fk6hcnmbkcspa3y-eval-okay-xml.exp.xml.drv'...
2024-08-13T14:04:56.2757530Z nix-functional-tests> objc[25124]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
2024-08-13T14:04:56.2759080Z nix-functional-tests> objc[25124]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
2024-08-13T14:04:56.2761140Z nix-functional-tests> error: builder for '/private/tmp/nix-build-nix-functional-tests-2.25.0pre20240813_1df9fb0.drv-0/nix-test/check/store/98i8amk2i6hj4h9w9fk6hcnmbkcspa3y-eval-okay-xml.exp.xml.drv' failed due to signal 6 (Abort trap: 6)

from the last run. Restarting to see if at least Linux will finish now.

Unfortunately, I could not reproduce these failures in a macOS dev shell: those tests just passed normally.

That is a known issue. You can apply #10164 to work around it.

@roberth Look good now?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-14-nix-team-meeting-minutes-169/50633/1