Handle `--jvm` flag in process with JNI instantiated JVM by JaroslavTulach · Pull Request #12843 · enso-org/enso

JaroslavTulach · 2025-04-14T03:57:06Z

Pull Request Description

Fixes #12842 by designing a JNI OS environment, providing its implementations for supported platforms and using it to handle --jvm flag. All the Standard Library Tests (jvm) use this new launch-in-the-same-process technique.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

All code follows the
Scala,
Java,
Unit tests have been written where possible.

engine/runner/src/main/java/org/enso/runner/Main.java

JaroslavTulach · 2025-04-22T17:34:51Z

Now the problem is in executing Generic_JDBC_Tests - e.g. the switch to JVM mode isn't yet working correctly:

[org.enso.runner.JavaFinder] No appropriate JDK found in the distribution runtimes. Trying system-wide JDK.
Error occurred during initialization of boot layer
java.lang.UnsatisfiedLinkError: 'int sun.nio.fs.UnixNativeDispatcher.init()'

Unsatisfied link problem on Linux (update on May 7, 2025: got it working after removing java.desktop module from native image). We have linkage problem on Windows:

jvm.lib(jni.obj) : error LNK2005: JNI_GetDefaultJavaVMInitArgs already defined in enso.obj
jvm.lib(jni.obj) : error LNK2005: JNI_CreateJavaVM already defined in enso.obj
jvm.lib(jni.obj) : error LNK2005: JNI_GetCreatedJavaVMs already defined in enso.obj
net.lib(net_util_md.obj) : error LNK2005: DllMain already defined in jvm.lib(os_windows.obj)

Update on May 2, 2025: the Windows version works on GraalVM for JDK21, but not on any newer version!

On Mac the --jvm option already works OK.

JaroslavTulach · 2025-04-30T03:27:49Z

Linux

Trying to isolate the UnsatisfiedLinkError...

Error occurred during initialization of boot layer
java.lang.UnsatisfiedLinkError: 'int sun.nio.fs.UnixNativeDispatcher.init()'

looks like the set of enabled modules is the cause of the problem! With:

diff --git build.sbt build.sbt
diff --git build.sbt build.sbt
index bbdeef42ce..00f114fa64 100644
$ git diff
diff --git build.sbt build.sbt
diff --git build.sbt build.sbt
index bbdeef42ce..764f9bf079 100644
--- build.sbt
+++ build.sbt
@@ -3912,7 +3912,7 @@ lazy val `engine-runner` = project
       val NI_MODULES =
         "org.graalvm.nativeimage,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.base,org.graalvm.nativeimage.driver,org.graalvm.nativeimage.librarysupport,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,com.oracle.graal.graal_enterprise,com.oracle.svm.svm_enterprise"
       val JDK_MODULES =
-        "java.desktop,java.naming,java.net.http,jdk.charsets,jdk.crypto.ec,jdk.localedata,jdk.httpserver,java.rmi"
+        "java.naming,java.net.http,jdk.charsets,jdk.crypto.ec,jdk.localedata,jdk.httpserver,java.rmi"
       val DEBUG_MODULES  = "jdk.jdwp.agent"
       val PYTHON_MODULES = "jdk.security.auth,java.naming"

it is possible to execute

enso$ ./built-distribution/enso-engine-*/enso-*/bin/enso --jvm --run test/Base_Tests

after building native image executable with ENSO_LAUNCHER=-ls,fast,debug.

…Linux

JaroslavTulach · 2025-05-08T08:43:42Z

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/JNIDirectives.java

+
+  @Override
+  public List<String> getLibraries() {
+    return List.of("jvm");


This may be the core cause of [Native Image] Dynamic loading of jvm.dlldoesn't link - regression since GraalVM21? oracle/graal#11152

the jvm.lib or jvm.obj then appears multiple times in the linkage

Let's see if it works in da05d8d!?

Use the generated executable JaroslavTulach/LaunchJdkViaJni#3 indicates it could work

JaroslavTulach · 2025-05-09T07:20:40Z

build.sbt

+    ),
+    assemblyMergeStrategy := { case _ =>
+      MergeStrategy.preferProject
+    }


the commit 967e2ce starts with following idea

define poi-wrapper fat JAR

use that poi-wrapper in std-table instead of direct dependecy on poi

make sure the poi-wrapper contains the necessary patches to avoid dependency on java.desktop

I was hoping to find inspiration at Updating snakeyaml and removing no longer needed patches #11207 - but it doesn't define reusable fat JAR either!

done by @Akirathan at 179aff3

thank you - it's a complicated change - I could never do it myself

but the CI is green!

finally we can move pass this JNI struggle...

JaroslavTulach · 2025-05-09T07:22:32Z

lib/java/poi-wrapper/src/main/java/org/apache/poi/hssf/util/HSSFColor.java

+  private static Map<Integer, HSSFColor> indexHash;
+  private static Map<HSSFColorPredefined, HSSFColor> enumList;
+
+  private final int index;


Patches to avoid dependency on java.desktop by Apache POI

proposal sent to upstream project

Avoiding needless java.awt.Color dependency in HSSFColor apache/poi#806

once it is integrated

let's switch to new POI version and remove this copy of HSSFColor!

…ar, not other poi jars.

Akirathan

So if I understand it correctly, all the Standard Library Tests (jvm) use the new launch-in-the-same-process technique?

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/JVM.java

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/PosixJVM.java

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/WindowsJVM.java

…ulach/Jni12842

JaroslavTulach · 2025-05-11T14:23:09Z

This change is already present in this Nightly Release:

thanks to Check NI size after it is built #12996 we can check the current sizes
Linux 404MB
MacOSX Arm 392MB
MacOSX Intel 377MB
Windows 390MB

These values should be smaller, because we removed java.desktop dependency from native image.

#12843 (comment)

Decreasing the maximum expected NI size of Linux by 30 MB. The new sizes (as reported in #12843 (comment)): - [Linux](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442578#step:10:3067) **404MB** - [MacOSX Arm](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442393#step:10:3076) **392MB** - [MacOSX Intel](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442260#step:10:2883) **377MB** - [Windows](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442715#step:10:3129) **390MB**

Fixes #12842 by designing a _JNI OS environment_, providing its implementations for supported platforms and using it to handle `--jvm` flag. All the _Standard Library Tests (jvm)_ use this new launch-in-the-same-process technique.

Decreasing the maximum expected NI size of Linux by 30 MB. The new sizes (as reported in #12843 (comment)): - [Linux](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442578#step:10:3067) **404MB** - [MacOSX Arm](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442393#step:10:3076) **392MB** - [MacOSX Intel](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442260#step:10:2883) **377MB** - [Windows](https://github.com/enso-org/enso/actions/runs/14951454883/job/42001442715#step:10:3129) **390MB**

Removing obsoleted DesktopEnvironment

b2a04ed

JaroslavTulach added CI: No changelog needed Do not require a changelog entry for this PR. -compiler labels Apr 14, 2025

JaroslavTulach self-assigned this Apr 14, 2025

JaroslavTulach requested review from 4e6, Akirathan and hubertp as code owners April 14, 2025 03:57

JaroslavTulach added 7 commits April 14, 2025 08:05

Load a class via JNI

0c4d7e9

Invoke Short.parseShort method

b3f5894

Enabling dynamic linking on Linux

f8b9234

Works (without sighandler) on Windows

45f45c1

Need to disable segfault handler on Windows

14aabfa

Create JVM with options

ed2ce35

Invoke Main.main via JVM and its JNI

ebb49cb

JaroslavTulach changed the title ~~API for access to JVM via JNI~~ Handle --jvm flag in process with JNI instantiated JVM Apr 14, 2025

Verify behavior of JVM.executeMain method

1efcc8f

JaroslavTulach force-pushed the wip/jtulach/Jni12842 branch from aa4dae1 to 1efcc8f Compare April 15, 2025 08:23

Compute factorial in NI and JVM and compare the results

d3faacf

JaroslavTulach commented Apr 15, 2025

View reviewed changes

engine/runner/src/main/java/org/enso/runner/Main.java Outdated Show resolved Hide resolved

JaroslavTulach added 2 commits April 22, 2025 18:12

Repeat the test just five times

41c19f6

Derive Java home from java executable File

369912b

JaroslavTulach added 5 commits April 22, 2025 22:51

Handle JAVA_OPTS in JVM class

d29b6ae

Don't pass (wrong) JAVA_OPTS to test-os-env subprocess

94565c5

Merge remote-tracking branch 'origin/develop' into wip/jtulach/Jni12842

764cbc8

Specifying --module-path rather than class path

179307b

Merge remote-tracking branch 'origin/develop' into wip/jtulach/Jni12842

17b1691

Avoiding dependency on java.desktop unlocks --jvm via JNI support on …

ee85301

…Linux

JaroslavTulach requested a review from jdunkerley as a code owner April 30, 2025 05:16

JaroslavTulach added 2 commits May 8, 2025 10:42

No need for loading in jvm.dll

7a0846a

Handling --jvm flag via JNI works (locally) on Windows

da05d8d

JaroslavTulach commented May 9, 2025

View reviewed changes

JaroslavTulach added 2 commits May 9, 2025 09:11

Disable segfault handler when running os-environment/test

0ab5f22

Attempt to define poi-wrapper (like to be) uber JAR with patches

967e2ce

JaroslavTulach commented May 9, 2025

View reviewed changes

JaroslavTulach requested a review from Akirathan May 9, 2025 07:46

std-table polyglot directory contains only poi-wrapper-assembly fat j…

179aff3

…ar, not other poi jars.

Akirathan approved these changes May 9, 2025

View reviewed changes

jdunkerley approved these changes May 9, 2025

View reviewed changes

JaroslavTulach added 4 commits May 9, 2025 17:31

Use Platform.getOperatingSystem() to check for a platform

923f4ec

Removing outdated comments & co.

2191b84

Merge branch 'wip/jtulach/Jni12842' of enso:enso-org/enso into wip/jt…

71410be

…ulach/Jni12842

Let's use JNI all the time

e24a687

JaroslavTulach added the CI: Ready to merge This PR is eligible for automatic merge label May 9, 2025

mergify bot merged commit 59eb1f2 into develop May 9, 2025
78 of 80 checks passed

mergify bot deleted the wip/jtulach/Jni12842 branch May 9, 2025 17:40

enso-bot bot mentioned this pull request May 10, 2025

Use JNI to handle --jvm invocation "in process" #12842

Closed

JaroslavTulach mentioned this pull request May 10, 2025

Avoiding needless java.awt.Color dependency in HSSFColor apache/poi#806

Closed

JaroslavTulach mentioned this pull request May 11, 2025

Check NI size after it is built #12996

Merged

5 tasks

Akirathan pushed a commit that referenced this pull request May 12, 2025

Reflect smaller Linux NI size.

25c28d5

#12843 (comment)

Akirathan mentioned this pull request May 12, 2025

Update NI size check - smaller Linux NI. #13081

Merged

3 tasks

JaroslavTulach mentioned this pull request May 27, 2025

Dual NI + JVM Mode for Loading Libraries #13172

Closed

3 tasks

jtulach mentioned this pull request Oct 18, 2025

Make sure netbeans in embedded terminal means itself apache/netbeans#8756

Merged

JaroslavTulach mentioned this pull request Feb 19, 2026

Read package.yaml jvm option when starting language server #14785

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle `--jvm` flag in process with JNI instantiated JVM#12843

Handle `--jvm` flag in process with JNI instantiated JVM#12843
mergify[bot] merged 65 commits intodevelopfrom
wip/jtulach/Jni12842

JaroslavTulach commented Apr 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

JaroslavTulach commented Apr 22, 2025 •

edited

Loading

Uh oh!

JaroslavTulach commented Apr 30, 2025 •

edited

Loading

Uh oh!

JaroslavTulach May 8, 2025

Uh oh!

JaroslavTulach May 9, 2025 •

edited

Loading

Uh oh!

JaroslavTulach May 9, 2025 •

edited

Loading

Uh oh!

JaroslavTulach May 9, 2025

Uh oh!

JaroslavTulach May 9, 2025 •

edited

Loading

Uh oh!

Akirathan left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JaroslavTulach commented May 11, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

JaroslavTulach commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Checklist

Uh oh!

Uh oh!

JaroslavTulach commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linux

Uh oh!

JaroslavTulach May 8, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 9, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Akirathan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JaroslavTulach commented May 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JaroslavTulach commented Apr 14, 2025 •

edited

Loading

JaroslavTulach commented Apr 22, 2025 •

edited

Loading

JaroslavTulach commented Apr 30, 2025 •

edited

Loading

JaroslavTulach May 9, 2025 •

edited

Loading

JaroslavTulach May 9, 2025 •

edited

Loading

JaroslavTulach May 9, 2025 •

edited

Loading

JaroslavTulach commented May 11, 2025 •

edited

Loading