Hidden failed tests due to OOME #314
Description
Hidden failed tests due to OOME
We are facing an issue where failed tests are not being detected by Jenkins, causing the build to remain green despite test failures.
We have found this OOME during investigation:
12:35:21 [junit4] <JUnit4> says ¡Hola! Main seed: A3622C02CB25DA98
12:39:57 [junit4] JVM J4: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J4-20250307_123531_0013519913414711688511.syserr
12:39:57 [junit4] JVM J3: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J3-20250307_123531_00115795041469580267028.syserr
12:39:57 [junit4] JVM J6: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J6-20250307_123531_00115333636689315299013.syserr
12:39:57 [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J2-20250307_123531_0015475062217785742379.syserr
12:39:57 [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J0-20250307_123531_00115150551702856997639.syserr
12:40:29 [junit4] JVM J5: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J5-20250307_123531_00115978591693253400979.syserr
12:40:51 [junit4] Event bus dispatch error: java.lang.OutOfMemoryError: Java heap space
12:40:51 [junit4] java.lang.OutOfMemoryError: Java heap space
12:40:51 [junit4] at java.base/java.util.Arrays.copyOf(Arrays.java:3541)
12:40:51 [junit4] at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:242)
12:40:51 [junit4] at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:744)
12:40:51 [junit4] at java.base/java.lang.StringBuffer.append(StringBuffer.java:410)
12:40:51 [junit4] at java.base/java.io.StringWriter.write(StringWriter.java:99)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.carrotsearch.randomizedtesting.WriterOutputStream.flushOutput(WriterOutputStream.java:284)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.carrotsearch.randomizedtesting.WriterOutputStream.write(WriterOutputStream.java:200)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.LocalForkedJvmStreamHandler$OnDiskStreamEvent.copyTo(LocalForkedJvmStreamHandler.java:202)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.ForkedJvmInfo.decodeStreams(ForkedJvmInfo.java:139)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.buildModel(AntXmlReport.java:236)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.onSuiteResult(AntXmlReport.java:182)
12:40:51 [junit4] at java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
12:40:51 [junit4] at java.base/java.lang.invoke.LambdaForm$MH/0x00007fe2b0129400.invoke(LambdaForm$MH)
12:40:51 [junit4] at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
12:40:51 [junit4] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
12:40:51 [junit4] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
12:40:51 [junit4] at java.base/java.lang.reflect.Method.invoke(Method.java:580)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:144)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:67)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.Dispatcher$PerThreadQueuedDispatcher.dispatch(Dispatcher.java:108)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:212)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveSuiteEnd(AggregatingListener.java:158)
12:40:51 [junit4] at java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(DirectMethodHandle$Holder)
12:40:51 [junit4] at java.base/java.lang.invoke.LambdaForm$MH/0x00007fe2b012a400.invoke(LambdaForm$MH)
12:40:51 [junit4] at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
12:40:51 [junit4] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
12:40:51 [junit4] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
12:40:51 [junit4] at java.base/java.lang.reflect.Method.invoke(Method.java:580)
12:40:51 [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
12:42:13 [junit4] JVM J7: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J7-20250307_123530_9983752796699895042005.syserr
12:42:25 [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/zzz.git-test_master/test/work/junit4-J1-20250307_123531_00114547635880582419092.syserr
12:42:25 [junit4] JVM J0: 1.14 .. 261.22 = 260.08s
12:42:25 [junit4] JVM J1: 1.20 .. 412.73 = 411.53s
12:42:25 [junit4] JVM J2: 1.11 .. 260.70 = 259.60s
12:42:25 [junit4] JVM J3: 1.07 .. 259.56 = 258.49s
12:42:25 [junit4] JVM J4: 1.00 .. 259.40 = 258.40s
12:42:25 [junit4] JVM J5: 1.00 .. 293.46 = 292.46s
12:42:25 [junit4] JVM J6: 1.16 .. 260.61 = 259.45s
12:42:25 [junit4] JVM J7: 1.19 .. 397.72 = 396.53s
12:42:25 [junit4] Execution time total: 6 minutes 52 seconds
12:42:25 [junit4] Tests summary: 689 suites (7 ignored), 6647 tests, 9 failures, 32 ignored
12:42:25 [echo] 2025-03-07 12:42:23> JUnit end
Jenkins does not detect failed tests despite the test summary reporting failures:
12:42:25 [junit4] Tests summary: 689 suites (7 ignored), 6647 tests, 9 failures, 32 ignored
Excessive test output (>50MB) contributes to the OOME, preventing test results from being processed correctly.
Reducing test output resolves the OOME, allowing test failures to be properly reported and causing the build to fail as expected. However, it would be safer if failures like this also caused the JUnit4 task and the Jenkins build to fail.
Environment: junit4-ant-2.8.2