NixOS · wolfgangwalther · Jan 10, 2025 · Oct 9, 2024 · Dec 19, 2024 · Dec 19, 2024
diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md
@@ -183,7 +183,15 @@ following are specific to `buildPythonPackage`:
   [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
   the binary. Additional arguments here can allow a developer to set environment
   variables which will be available when the binary is run. For example,
-  `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
+  `makeWrapperArgs = ["--set" "FOO" "BAR" "--set" "BAZ" "QUX"]`.
+
+  ::: {.note}
+  When `__structuredAttrs = false`, the attribute `makeWrapperArgs` is passed as a space-separated string to the build script. Developers should use `prependToVar` or `appendToVar` to add arguments to it in build phases, or use `__structuredAttrs = true` to ensure that `makeWrapperArgs` is passed as a Bash array.
+
+  For compatibility purposes,
+  when `makeWrapperArgs` shell variable is specified as a space-separated string (instead of a Bash array) in the build script, the string content is Bash-expanded before concatenated into the `wrapProgram` command. Still, developers should not rely on such behaviours, but use `__structuredAttrs = true` to specify flags containing spaces (e.g. `makeWrapperArgs = [ "--set" "GREETING" "Hello, world!" ]`), or use -pre and -post phases to specify flags with Bash-expansions (e.g. `preFixup = ''makeWrapperArgs+=(--prefix PATH : "$SOME_PATH")`'').
+  :::
+
 * `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
   defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
 * `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
@@ -1273,7 +1281,7 @@ Using the example above, the analogous `pytestCheckHook` usage would be:
   ];
 
   # requires additional data
-  pytestFlagsArray = [
+  pytestFlags = [
     "tests/"
     "--ignore=tests/integration"
   ];
@@ -1433,7 +1441,7 @@ automatically add `pythonRelaxDepsHook` if either `pythonRelaxDeps` or
     unittestCheckHook
   ];
 
-  unittestFlagsArray = [
+  unittestFlags = [
     "-s" "tests" "-v"
   ];
 }
@@ -2002,7 +2010,7 @@ Occasionally packages don't make use of a common test framework, which may then
 
 * Non-working tests can often be deselected. Most Python modules
   do follow the standard test protocol where the pytest runner can be used.
-  `pytest` supports the `-k` and `--ignore` parameters to ignore test
+  `pytest` supports the `-k` and `--ignore-glob` parameters to ignore test
   methods or classes as well as whole files. For `pytestCheckHook` these are
   conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
 
@@ -2019,11 +2027,17 @@ Occasionally packages don't make use of a common test framework, which may then
     ];
 
     disabledTestPaths = [
-      "this/file.py"
+      "path/to/performance.py"
+      "path/to/connect-*.py"
     ];
   }
   ```
 
+  ::: {.note}
+  If the test path to disable contains characters like `*`, `?`, `[`, and `]`,
+  quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally.
+  :::
+
 * Tests that attempt to access `$HOME` can be fixed by using the following
   work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
 * Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`.

diff --git a/nixos/doc/manual/release-notes/rl-2505.section.md b/nixos/doc/manual/release-notes/rl-2505.section.md
@@ -22,6 +22,8 @@
 
 - The default Elixir version has been updated to 1.18.
 
+- `buildPythonPackage`, `buildPythonApplication` and the Python building setup hooks now support both `__structuredAttrs = true` and `__structuredAttrs = false`.
+
 - `services.dex` now restarts upon changes to the `.environmentFile` or entries in `.settings.staticClients[].secretFile` when the entry is a `path` type.
 
 - `nixos-rebuild-ng`, a full rewrite of `nixos-rebuild` in Python, is available for testing. You can enable it by setting [system.rebuild.enableNg](options.html#opt-system.rebuild.enableNg) in your configuration (this will replace the old `nixos-rebuild`), or by adding `nixos-rebuild-ng` to your `environment.systemPackages` (in this case, it will live side-by-side with `nixos-rebuild` as `nixos-rebuild-ng`). It is expected that the next major version of NixOS (25.11) will enable `system.rebuild.enableNg` by default.
@@ -120,6 +122,16 @@
   instead of the python tester launcher. You can still refer to the python
   launcher via `python3Packages.toPythonApplication python3Packages.playwright`
 
+- The representation of the flags attributes as shell/environment variables for most Python building setup hooks are now the same as `stdenv.mkDerivation` and other build helpers -- they are space-separated environment variables when `__structuredAttrs = false` and Bash arrays when `__structuredAttrs = true`, and are concatenated to the command without Bash-evaluation. The following behaviour changes are introduced during the conversion:
+
+  - The following flags are no longer Bash-expanded before concatenated to the command:
+    - `disabledTests` and `disabledTestPaths` for `pytestCheckHook`. (`disabledTestPaths` used to be expanded twice before concatenation.)
+    - `setupPyBuildFlags` and `setupPyGlobalFlags` for `setuptoolsBuildHook`.
+
+  - `pytestFlags` and `unittestFlags` replace `pytestFlagsArray` and `unittestFlagsArray` and become the new and conforming interface.
+
+  - `pytestFlagsArray` and `unittestFlagsArray` are kept for compatibility purposes. They continue to be Bash-expanded before concatenated. This compatibility layer will be removed in future releases.
+
 - `strawberry` has been updated to 1.2, which drops support for the VLC backend and Qt 5. The `strawberry-qt5` package
   and `withGstreamer`/`withVlc` override options have been removed due to this.
 

diff --git a/pkgs/development/interpreters/python/hooks/pytest-check-hook.sh b/pkgs/development/interpreters/python/hooks/pytest-check-hook.sh
@@ -1,33 +1,42 @@
 # Setup hook for pytest
-echo "Sourcing pytest-check-hook"
+# shellcheck shell=bash
 
-declare -ar disabledTests
-declare -a disabledTestPaths
+echo "Sourcing pytest-check-hook"
 
 function pytestCheckPhase() {
     echo "Executing pytestCheckPhase"
     runHook preCheck
 
     # Compose arguments
-    args=" -m pytest"
-    if [ -n "$disabledTests" ]; then
+    local -a flagsArray=(-m pytest)
+    if [ -n "${disabledTests[*]-}" ]; then
         disabledTestsString="not $(concatStringsSep " and not " disabledTests)"
-        args+=" -k \""$disabledTestsString"\""
-    fi
-
-    if [ -n "${disabledTestPaths-}" ]; then
-        eval "disabledTestPaths=($disabledTestPaths)"
+        flagsArray+=(-k "$disabledTestsString")
     fi
 
-    for path in ${disabledTestPaths[@]}; do
-        if [ ! -e "$path" ]; then
-            echo "Disabled tests path \"$path\" does not exist. Aborting"
-            exit 1
-        fi
-        args+=" --ignore=\"$path\""
+    local -a _pathsArray=()
+    concatTo _pathsArray disabledTestPaths
+    for path in "${_pathsArray[@]}"; do
+        # Check if every path glob matches at least one path
+        @pythonCheckInterpreter@ <(cat <<EOF
+import glob
+import sys
+path_glob=sys.argv[1]
+if not len(path_glob):
+    sys.exit('Got an empty disabled tests path glob. Aborting')
+if next(glob.iglob(path_glob), None) is None:
+    sys.exit('Disabled tests path glob "{}" does not match any paths. Aborting'.format(path_glob))
+EOF
+        ) "$path"
+        flagsArray+=("--ignore-glob=$path")
     done
-    args+=" ${pytestFlagsArray[@]}"
-    eval "@pythonCheckInterpreter@ $args"
+
+    # Compatibility layer to the obsolete pytestFlagsArray
+    eval "flagsArray+=(${pytestFlagsArray[*]-})"
+
+    concatTo flagsArray pytestFlags
+    echoCmd 'pytest flags' "${flagsArray[@]}"
+    @pythonCheckInterpreter@ "${flagsArray[@]}"
 
     runHook postCheck
     echo "Finished executing pytestCheckPhase"

diff --git a/pkgs/development/interpreters/python/hooks/setuptools-build-hook.sh b/pkgs/development/interpreters/python/hooks/setuptools-build-hook.sh
@@ -1,25 +1,29 @@
 # Setup hook for setuptools.
+# shellcheck shell=bash
+
 echo "Sourcing setuptools-build-hook"
 
 setuptoolsBuildPhase() {
     echo "Executing setuptoolsBuildPhase"
-    local args setuptools_has_parallel=@setuptools_has_parallel@
+    local setuptools_has_parallel=@setuptools_has_parallel@
     runHook preBuild
 
     cp -f @setuppy@ nix_run_setup
-    args=""
-    if [ -n "$setupPyGlobalFlags" ]; then
-        args+="$setupPyGlobalFlags"
+    local -a flagsArray=()
+    if [ -n "${setupPyGlobalFlags[*]-}" ]; then
+        concatTo flagsArray setupPyGlobalFlags
     fi
     if [ -n "$enableParallelBuilding" ]; then
         if [ -n "$setuptools_has_parallel" ]; then
-            setupPyBuildFlags+=" --parallel $NIX_BUILD_CORES"
+            appendToVar setupPyBuildFlags --parallel "$NIX_BUILD_CORES"
         fi
     fi
-    if [ -n "$setupPyBuildFlags" ]; then
-        args+=" build_ext $setupPyBuildFlags"
+    if [ -n "${setupPyBuildFlags[*]-}" ]; then
+        flagsArray+=(build_ext)
+        concatTo flagsArray setupPyBuildFlags
     fi
-    eval "@pythonInterpreter@ nix_run_setup $args bdist_wheel"
+    echoCmd 'setup.py build flags' "${flagsArray[@]}"
+    @pythonInterpreter@ nix_run_setup "${flagsArray[@]}" bdist_wheel
 
     runHook postBuild
     echo "Finished executing setuptoolsBuildPhase"

diff --git a/pkgs/development/interpreters/python/hooks/unittest-check-hook.sh b/pkgs/development/interpreters/python/hooks/unittest-check-hook.sh
@@ -1,17 +1,26 @@
 # Setup hook for unittest.
+# shellcheck shell=bash
+
 echo "Sourcing unittest-check-hook"
 
 unittestCheckPhase() {
     echo "Executing unittestCheckPhase"
     runHook preCheck
 
-    eval "@pythonCheckInterpreter@ -m unittest discover $unittestFlagsArray"
+    local -a flagsArray=()
+
+    # Compatibility layer to the obsolete unittestFlagsArray
+    eval "flagsArray+=(${unittestFlagsArray[*]-})"
+
+    concatTo flagsArray unittestFlags
+    echoCmd 'unittest flags' "${flagsArray[@]}"
+    @pythonCheckInterpreter@ -m unittest discover "${flagsArray[@]}"
 
     runHook postCheck
     echo "Finished executing unittestCheckPhase"
 }
 
-if [ -z "${dontUseUnittestCheck-}" ] && [ -z "${installCheckPhase-}" ]; then
+if [[ -z "${dontUseUnittestCheck-}" ]] && [[ -z "${installCheckPhase-}" ]]; then
     echo "Using unittestCheckPhase"
     appendToVar preDistPhases unittestCheckPhase
 fi
diff --git a/pkgs/development/interpreters/python/mk-python-derivation.nix b/pkgs/development/interpreters/python/mk-python-derivation.nix
@@ -106,7 +106,9 @@ let
     "format"
     "disabledTestPaths"
     "disabledTests"
+    "pytestFlags"
     "pytestFlagsArray"
+    "unittestFlags"
     "unittestFlagsArray"
     "outputs"
     "stdenv"
@@ -434,16 +436,20 @@ let
       }
       // optionalAttrs (attrs.doCheck or true) (
         optionalAttrs (disabledTestPaths != [ ]) {
-          disabledTestPaths = escapeShellArgs disabledTestPaths;
+          disabledTestPaths = disabledTestPaths;
         }
         // optionalAttrs (attrs ? disabledTests) {
-          # `escapeShellArgs` should be used as well as `disabledTestPaths`,
-          # but some packages rely on existing raw strings.
           disabledTests = attrs.disabledTests;
         }
+        // optionalAttrs (attrs ? pytestFlags) {
+          pytestFlags = attrs.pytestFlags;
+        }
         // optionalAttrs (attrs ? pytestFlagsArray) {
           pytestFlagsArray = attrs.pytestFlagsArray;
         }
+        // optionalAttrs (attrs ? unittestFlags) {
+          unittestFlags = attrs.unittestFlags;
+        }
         // optionalAttrs (attrs ? unittestFlagsArray) {
           unittestFlagsArray = attrs.unittestFlagsArray;
         }

diff --git a/pkgs/development/python-modules/conda/default.nix b/pkgs/development/python-modules/conda/default.nix
@@ -26,6 +26,7 @@
   defaultPkgPath ? "~/.conda/pkgs", # default path to store download conda packages
 }:
 buildPythonPackage rec {
+  __structuredAttrs = true;
   pname = "conda";
   version = "24.7.1";
   pyproject = true;
@@ -63,9 +64,17 @@ buildPythonPackage rec {
   patches = [ ./0001-conda_exe.patch ];
 
   makeWrapperArgs = [
-    "--set CONDA_EXE ${placeholder "out"}/bin/conda"
-    ''--set-default CONDA_ENVS_PATH "${defaultEnvPath}"''
-    ''--set-default CONDA_PKGS_DIRS "${defaultPkgPath}"''
+    "--set"
+    "CONDA_EXE"
+    "${placeholder "out"}/bin/conda"
+
+    "--set-default"
+    "CONDA_ENVS_PATH"
+    defaultEnvPath
+
+    "--set-default"
+    "CONDA_PKGS_DIRS"
+    defaultPkgPath
   ];
 
   pythonImportsCheck = [ "conda" ];

diff --git a/pkgs/development/python-modules/mysql-connector/default.nix b/pkgs/development/python-modules/mysql-connector/default.nix
@@ -18,10 +18,11 @@ buildPythonPackage rec {
   disabled = pythonOlder "3.7";
 
   setupPyBuildFlags = [
-    "--with-mysql-capi=\"${mysql80}\""
-    "--with-openssl-include-dir=\"${openssl.dev}/include\""
-    "--with-openssl-lib-dir=\"${lib.getLib openssl}/lib\""
-    "-L \"${lib.getLib pkgs.zstd}/lib:${lib.getLib mysql80}/lib\""
+    "--with-mysql-capi=${mysql80}"
+    "--with-openssl-include-dir=${openssl.dev}/include"
+    "--with-openssl-lib-dir=${lib.getLib openssl}/lib"
+    "-L"
+    "${lib.getLib pkgs.zstd}/lib:${lib.getLib mysql80}/lib"
   ];
 
   src = fetchFromGitHub {

diff --git a/pkgs/development/python-modules/pytest-forked/setup-hook.sh b/pkgs/development/python-modules/pytest-forked/setup-hook.sh
@@ -1,7 +1,5 @@
 pytestForkedHook() {
-    pytestFlagsArray+=(
-        "--forked"
-    )
+    appendToVar pytestFlags "--forked"
 
     # Using --forked on darwin leads to crashes when fork safety is
     # enabled. This often happens when urllib tries to request proxy

diff --git a/pkgs/development/python-modules/pytest-xdist/default.nix b/pkgs/development/python-modules/pytest-xdist/default.nix
@@ -47,7 +47,7 @@ buildPythonPackage rec {
 
   # pytest can already use xdist at this point
   preCheck = ''
-    appendToVar pytestFlagsArray "--numprocesses=$NIX_BUILD_CORES"
+    appendToVar pytestFlags "--numprocesses=$NIX_BUILD_CORES"
   '';
 
   # access file system

diff --git a/pkgs/development/python-modules/pytest-xdist/setup-hook.sh b/pkgs/development/python-modules/pytest-xdist/setup-hook.sh
@@ -1,7 +1,5 @@
 pytestXdistHook() {
-    pytestFlagsArray+=(
-        "--numprocesses=$NIX_BUILD_CORES"
-    )
+    appendToVar pytestFlags "--numprocesses=$NIX_BUILD_CORES"
 }
 
 if [ -z "${dontUsePytestXdist-}" ] && [ -z "${dontUsePytestCheck-}" ]; then

diff --git a/pkgs/development/python-modules/tensorflow/default.nix b/pkgs/development/python-modules/tensorflow/default.nix
@@ -607,6 +607,7 @@ let
   };
 in
 buildPythonPackage {
+  __structuredAttrs = true;
   inherit version pname;
   disabled = pythonAtLeast "3.12";
 
@@ -636,7 +637,10 @@ buildPythonPackage {
     rm $out/bin/tensorboard
   '';
 
-  setupPyGlobalFlags = [ "--project_name ${pname}" ];
+  setupPyGlobalFlags = [
+    "--project_name"
+    pname
+  ];
 
   # tensorflow/tools/pip_package/setup.py
   propagatedBuildInputs = [

diff --git a/pkgs/development/python-modules/vowpalwabbit/default.nix b/pkgs/development/python-modules/vowpalwabbit/default.nix
@@ -42,7 +42,7 @@ buildPythonPackage rec {
 
   # As we disable configure via cmake, pass explicit global options to enable
   # spdlog and fmt packages
-  setupPyGlobalFlags = [ "--cmake-options=\"-DSPDLOG_SYS_DEP=ON;-DFMT_SYS_DEP=ON\"" ];
+  setupPyGlobalFlags = [ "--cmake-options=-DSPDLOG_SYS_DEP=ON;-DFMT_SYS_DEP=ON" ];
 
   propagatedBuildInputs = [
     numpy