test(linstor): fake_linstor_clock rewrites a shared kubectl stub that parallel bats tests execute #5298

One test in hack/linstor-storage-mode_test.bats failed once with Text file busy on a full unit run under bats -j 4. The same test passed on the five full runs before it, and the file is not touched by the change that was being tested.

not ok 1141 a suite that fails before recording its baseline cannot reuse the previous suite's
# (from function `fake_linstor_clock' in file hack/linstor-storage-mode_test.bats, line 22,
#  in test file hack/linstor-storage-mode_test.bats, line 458)
#   `fake_linstor_clock' failed
# .../hack/linstor-storage-mode_test.bats: line 22: _out/tmp/linstor-bin/kubectl: Text file busy

fake_linstor_clock is called by 12 tests in that file. Each call does mkdir -p _out/tmp/linstor-bin, then printf ... >_out/tmp/linstor-bin/kubectl, then chmod +x, and puts that directory first on PATH. The path is relative to the repository root and the same for every test, so all of them write one file and execute it.

The file is run as kubectl by the code under test, which goes through timeout and sh -c and so finds the stub on PATH. On Linux, opening a file for writing while another process is executing it fails with ETXTBSY. printf > truncates the existing file in place, so a test that reaches line 22 while a sibling test is running the stub gets the error. Tests in one file run concurrently under bats -j N unless --no-parallelize-within-files is given, and make bats-unit-tests sets -j to nproc when GNU parallel is installed.

Run alone, the file passes 27 of 27 with -j 1, and six runs with -j 4 did not fail either. The window is short, so the failure is rare.

The stub content never changes, so it only needs to be written once per test, in a place no other test shares. Putting it under $BATS_TEST_TMPDIR fixes the race and also stops leaving _out/tmp/linstor-bin behind. Writing it to a temporary name and renaming it into place would also work, because rename replaces the directory entry and does not touch an inode that is being executed.

I grepped hack/*.bats for other tests that write an executable under _out/tmp. This is the only one; the other uses of _out/tmp there are data files named with $$, which are not executed.