Skip to content

Commit b9d817e

Browse files
morningmanclaude
andcommitted
[fix](paimon) Keep paimon pool threads from exiting the BE process
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Paimon's `ExecutorThreadFactory` gives every thread it makes `FatalExitExceptionHandler` unless the caller names another handler, and that handler logs, dumps the threads and calls `System.exit(-17)`. That suits the Flink or Spark worker paimon is written for. Inside BE it is an `::exit()` from a JVM thread, the same abort the fluss client's copy of the handler caused (previous commit). The paimon plugin runs such threads on its read path: `AsyncRecordReader` reads each large data file of a merge read on a thread of one of these pools, and `ParallelExecution` runs another. A task's own exception goes to whoever waits on its future, but an `OutOfMemoryError` that hits an idle worker while it waits for work kills the thread outside any task - and paimon's merge reads are the readers that exhaust BE's JVM heap most easily: sixteen of them at once overflow the default 2 GB on a 30M-row primary-key table that is not compacted. Not seen to fire inside BE yet (found by listing the `System.exit` callers of every jar BE loads); a standalone JVM that throws an `OutOfMemoryError` on a thread from paimon's `ExecutorThreadFactory` exits with status 239. The paimon plugin now ships a Doris copy of `org.apache.paimon.utils.FatalExitExceptionHandler`, with every member paimon's declares, whose handler only logs the exception. It lives in a new module, `paimon-common-patch`, built and shipped the way `fluss-client-patch` is: its jar names the class in `Doris-Shadows-Classes`, so PluginRuntime searches it ahead of the copies in `paimon-common` and in the bundled `paimon-hive-connector`, and `paimon-scanner` depends on it, declared before the paimon artifacts. The thread that threw is lost, its pool starts another, and BE stays up. ### Release note A paimon thread inside BE that dies of an uncaught error, such as running the BE's JVM out of heap, no longer brings down the BE process. ### Check List (For Author) - Test: Unit Test - `PaimonThreadDeathTest` (new, 2 tests): a thread from paimon's `ExecutorThreadFactory` dies of an uncaught `OutOfMemoryError`, and a pool built on that factory keeps serving after one of its workers dies that way. With paimon's handler the JVM running them exits with status 239. paimon-scanner: 66 tests pass. - Built with the Maven build cache on, the module and plugin `target/` removed, built again: the plugin's `target/lib` still holds the patch jar with its `Doris-Shadows-Classes` entry; `check_plugin_layout.py` and checkstyle pass. - Behavior changed: Yes. An uncaught exception in a paimon thread inside BE is logged instead of exiting the process. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 7b9be9d commit b9d817e

8 files changed

Lines changed: 230 additions & 1 deletion

File tree

‎build.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -885,6 +885,7 @@ if [[ "${BUILD_BE_JAVA_EXTENSIONS}" -eq 1 ]]; then
885885
# complete enumeration.
886886
modules+=("be-java-extensions/plugin-toolkit")
887887
modules+=("be-java-extensions/fluss-client-patch")
888+
modules+=("be-java-extensions/paimon-common-patch")
888889
modules+=("be-java-extensions/hive-udf-shade")
889890
modules+=("be-java-extensions/hive-apache-shade")
890891

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<parent>
24+
<artifactId>be-java-extensions</artifactId>
25+
<groupId>org.apache.doris</groupId>
26+
<version>${revision}</version>
27+
</parent>
28+
<modelVersion>4.0.0</modelVersion>
29+
30+
<artifactId>paimon-common-patch</artifactId>
31+
<name>Doris BE paimon common patch</name>
32+
<description>
33+
Doris's copy of org.apache.paimon.utils.FatalExitExceptionHandler, the handler paimon gives
34+
its pool threads, which ends the process when one of them dies (the class header says why).
35+
Not a plugin: the paimon plugin declares this artifact, so its jar lands in the plugin
36+
directory, and the jar's Doris-Shadows-Classes manifest entry is what makes PluginRuntime
37+
search it ahead of paimon-common's copy and of the one paimon-hive-connector bundles.
38+
39+
A module of its own, not a second jar built by paimon-scanner, because of the Maven build
40+
cache: a cache hit restores a module's own jar and nothing else, while copy-dependencies,
41+
forced to run on a hit (fe/.mvn/maven-build-cache-config.xml), copies the jars of the
42+
modules a plugin depends on. A second jar that paimon-scanner wrote into its own target/lib
43+
was missing from every cache-restored build, and the plugin deployed without it. The
44+
layout check (tools/be-java-plugins/check_plugin_layout.py) requires a shadowing jar to
45+
hold nothing but the classes it names, which is all this module builds.
46+
</description>
47+
48+
<properties>
49+
<maven.compiler.source>8</maven.compiler.source>
50+
<maven.compiler.target>8</maven.compiler.target>
51+
</properties>
52+
53+
<dependencies>
54+
<!-- The artifact this class replaces, provided by the plugin that deploys it. -->
55+
<dependency>
56+
<groupId>org.apache.paimon</groupId>
57+
<artifactId>paimon-common</artifactId>
58+
<scope>provided</scope>
59+
</dependency>
60+
</dependencies>
61+
62+
<build>
63+
<plugins>
64+
<plugin>
65+
<groupId>org.apache.maven.plugins</groupId>
66+
<artifactId>maven-jar-plugin</artifactId>
67+
<configuration>
68+
<archive>
69+
<!-- Merged with the parent's Doris-Jni-Plugin-Api-Version entry, not replacing it. -->
70+
<manifestEntries>
71+
<Doris-Shadows-Classes>org.apache.paimon.utils.FatalExitExceptionHandler</Doris-Shadows-Classes>
72+
</manifestEntries>
73+
</archive>
74+
</configuration>
75+
</plugin>
76+
</plugins>
77+
</build>
78+
</project>
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.paimon.utils;
19+
20+
import org.slf4j.Logger;
21+
import org.slf4j.LoggerFactory;
22+
23+
/**
24+
* Doris's copy of the handler paimon gives its threads for uncaught exceptions: this one logs them,
25+
* where paimon's stops the process.
26+
*
27+
* <p>Paimon hands {@link #INSTANCE} to every thread its {@code ExecutorThreadFactory} makes unless the
28+
* caller names another handler - among them the pool behind {@code AsyncRecordReader}, which reads
29+
* each large data file of a merge read on a thread of its own, and the pools of
30+
* {@code ParallelExecution}. Paimon's own version calls {@code System.exit(-17)}, which suits the Flink
31+
* or Spark worker paimon is written for. Inside BE it is an {@code ::exit()} from a JVM thread: the C++
32+
* global destructors run under a BE that is still serving, the failure {@code jvm_launcher.cpp}
33+
* describes for the JVM's signal handlers, and BE aborts. The fluss client's handler of the same
34+
* name did exactly that when concurrent reads exhausted the JVM heap, and paimon's merge reads are
35+
* the readers that exhaust it most easily. A pool thread is cheap to lose: a task's own exception goes
36+
* to whoever waits on its future, so this handler only sees what kills a thread between tasks - an
37+
* {@code OutOfMemoryError} while an idle thread waits for work - and its pool starts another.
38+
*
39+
* <p>Found ahead of paimon-common's copy, and of the one paimon-hive-connector bundles, because the
40+
* jar it ships in names it in {@code Doris-Shadows-Classes} (see this module's pom). It declares every
41+
* member paimon's version does, so the paimon classes compiled against that one link to this one.
42+
*/
43+
public final class FatalExitExceptionHandler implements Thread.UncaughtExceptionHandler {
44+
45+
public static final FatalExitExceptionHandler INSTANCE = new FatalExitExceptionHandler();
46+
47+
/** The status paimon's version exits with. Nothing exits with it here; it stays for linkage. */
48+
public static final int EXIT_CODE = -17;
49+
50+
private static final Logger LOG = LoggerFactory.getLogger(FatalExitExceptionHandler.class);
51+
52+
@Override
53+
public void uncaughtException(Thread thread, Throwable e) {
54+
LOG.error("Paimon thread '{}' died of an uncaught exception. Paimon would stop the process"
55+
+ " here; inside BE the thread is lost and the process carries on.", thread.getName(), e);
56+
}
57+
}

‎fe/be-java-extensions/paimon-scanner/pom.xml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,28 @@ under the License.
6666
<version>${project.version}</version>
6767
</dependency>
6868

69+
<!--
70+
Doris's copy of the paimon handler that would end BE when a pool thread dies
71+
(FatalExitExceptionHandler; see that module). copy-dependencies puts its jar in this
72+
plugin's directory, where its Doris-Shadows-Classes manifest entry makes PluginRuntime
73+
search it ahead of paimon-common and of the copy paimon-hive-connector bundles. Declared
74+
BEFORE the paimon artifacts because surefire's classpath follows declaration order, so the
75+
tests run on the copy the deployed plugin runs on - PaimonThreadDeathTest depends on that.
76+
Every transitive is excluded: the module's own dependency is provided, so all that could
77+
come along are the logging artifacts the fe parent pom declares on every module.
78+
-->
79+
<dependency>
80+
<groupId>org.apache.doris</groupId>
81+
<artifactId>paimon-common-patch</artifactId>
82+
<version>${project.version}</version>
83+
<exclusions>
84+
<exclusion>
85+
<groupId>*</groupId>
86+
<artifactId>*</artifactId>
87+
</exclusion>
88+
</exclusions>
89+
</dependency>
90+
6991
<dependency>
7092
<groupId>org.apache.paimon</groupId>
7193
<artifactId>paimon-core</artifactId>
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.paimon;
19+
20+
import org.apache.paimon.utils.ExecutorThreadFactory;
21+
import org.apache.paimon.utils.FatalExitExceptionHandler;
22+
import org.junit.jupiter.api.Assertions;
23+
import org.junit.jupiter.api.Test;
24+
25+
import java.util.concurrent.ExecutorService;
26+
import java.util.concurrent.Executors;
27+
import java.util.concurrent.TimeUnit;
28+
29+
/**
30+
* A paimon pool thread that dies must not take the process with it. Paimon's own handler for these
31+
* threads calls {@code System.exit}, and inside BE that runs the C++ global destructors under a BE
32+
* that is still serving; the plugin ships a handler of the same name that only logs.
33+
*
34+
* <p>Should paimon's handler come back - the patched class dropped, or found after paimon-common's -
35+
* these tests do not fail with an assertion: the JVM running them exits, and surefire reports a
36+
* forked VM that terminated without saying goodbye.
37+
*/
38+
public class PaimonThreadDeathTest {
39+
40+
@Test
41+
public void threadThatDiesOfAnUncaughtErrorLeavesTheProcessRunning() throws Exception {
42+
// The factory behind the pools of AsyncRecordReader and ParallelExecution.
43+
Thread thread = new ExecutorThreadFactory("doris-fatal-exit-test").newThread(() -> {
44+
throw new OutOfMemoryError("simulated: Java heap space");
45+
});
46+
Assertions.assertSame(FatalExitExceptionHandler.INSTANCE, thread.getUncaughtExceptionHandler(),
47+
"paimon no longer hands its threads this handler; the patch may not be needed any more");
48+
49+
thread.start();
50+
thread.join(TimeUnit.SECONDS.toMillis(60));
51+
Assertions.assertFalse(thread.isAlive(), "the thread was meant to die");
52+
}
53+
54+
@Test
55+
public void poolKeepsServingAfterOneOfItsThreadsDies() throws Exception {
56+
ExecutorService pool = Executors.newCachedThreadPool(new ExecutorThreadFactory("doris-fatal-exit-pool"));
57+
try {
58+
// Thrown outside any future, the way an OutOfMemoryError hits a worker waiting for work:
59+
// it reaches the thread's uncaught exception handler and ends that worker.
60+
pool.execute(() -> {
61+
throw new OutOfMemoryError("simulated: Java heap space");
62+
});
63+
Assertions.assertEquals("served", pool.submit(() -> "served").get(60, TimeUnit.SECONDS));
64+
} finally {
65+
pool.shutdownNow();
66+
}
67+
}
68+
}

‎fe/be-java-extensions/pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ under the License.
2525
<module>jni-bootstrap</module>
2626
<module>plugin-toolkit</module>
2727
<module>fluss-client-patch</module>
28+
<module>paimon-common-patch</module>
2829
<module>iceberg-metadata-scanner</module>
2930
<module>hadoop-hudi-scanner</module>
3031
<module>hive-udf-shade</module>

‎fe/check/checkstyle/suppressions.xml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ under the License.
5656
live in fluss's packages, which import-control.xml (rooted at org.apache.doris) does not cover; every other
5757
check still applies -->
5858
<suppress files="fluss-client-patch[\\/]src[\\/]main[\\/]java[\\/]org[\\/]apache[\\/]fluss[\\/]" checks="ImportControl" />
59+
<!-- paimon-common-patch's replacement for paimon's FatalExitExceptionHandler, likewise in paimon's package -->
60+
<suppress files="paimon-common-patch[\\/]src[\\/]main[\\/]java[\\/]org[\\/]apache[\\/]paimon[\\/]" checks="ImportControl" />
5961
<suppress files="org[\\/]apache[\\/]doris[\\/]load[\\/]loadv2[\\/]dpp[\\/]ColumnParser\.java" checks="OneTopLevelClass" />
6062
<suppress files="org[\\/]apache[\\/]doris[\\/]load[\\/]loadv2[\\/]dpp[\\/]SparkRDDAggregator\.java" checks="OneTopLevelClass" />
6163
<suppress files="org[\\/]apache[\\/]doris[\\/]catalog[\\/]FunctionSet\.java" checks="LineLength" />

‎run-fe-ut.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ FE_MODULES+=("fe-connector/fe-connector-fluss")
219219
# Listed one by one rather than as the aggregator: -pl on an aggregator selects that pom and none
220220
# of its children. Keep in sync with the module list in build.sh, which is the other complete
221221
# enumeration of this directory.
222-
for be_java_extension in jni-spi jni-bootstrap plugin-toolkit fluss-client-patch \
222+
for be_java_extension in jni-spi jni-bootstrap plugin-toolkit fluss-client-patch paimon-common-patch \
223223
hive-apache-shade hive-udf-shade \
224224
hadoop-deps iceberg-metadata-scanner hadoop-hudi-scanner java-udf jdbc-scanner paimon-scanner \
225225
fluss-scanner max-compute-connector trino-connector-scanner java-writer; do

0 commit comments

Comments
 (0)