Fix leftover processes/hangs in tests

azat · azat · commit 72fa58e19217 · 2023-12-23T16:40:33.000+01:00
One of such cases is 02479_race_condition_between_insert_and_droppin_mv [1], yes it can be fixed (by using fixed number of iterations, or with some bash trickery), but it is better to fix them completelly, eventually such tests will be submitted and pass review anyway. By allocating process group for each test we can kill all the processes in this process group, and this what this patch does. This will also fix some test hangs (like in [1]) as well as some possible issues in stress tests. [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/e2c1230b00386c4d0096a245396ab3be7ce60950/stateless_tests__release__analyzer_/run.log Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
diff --git a/tests/clickhouse-test b/tests/clickhouse-test
@@ -1031,7 +1031,7 @@ class TestCase:
         if proc:
             if proc.returncode is None:
                 try:
-                    proc.kill()
+                    os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
                 except OSError as e:
                     if e.errno != ESRCH:
                         raise
@@ -1307,7 +1307,7 @@ class TestCase:
 
         command = pattern.format(**params)
 
-        proc = Popen(command, shell=True, env=os.environ)
+        proc = Popen(command, shell=True, env=os.environ, start_new_session=True)
 
         while (
             datetime.now() - start_time
diff --git a/tests/queries/shell_config.sh b/tests/queries/shell_config.sh
@@ -185,3 +185,11 @@ function query_with_retry
     done
     echo "Query '$query' failed with '$result'"
 }
+
+# Add --foreground to avoid running setpgid() in timeout, otherwise
+# clickhouse-test will not kill those processes in case of timeout
+function timeout()
+{
+    command timeout --foreground "$@"
+}
+export -f timeout
