Skip to content

Make Turbine x86_64 native image more compatible#20306

Closed
fmeum wants to merge 1 commit intobazelbuild:masterfrom
fmeum:20298-compatible-turbine
Closed

Make Turbine x86_64 native image more compatible#20306
fmeum wants to merge 1 commit intobazelbuild:masterfrom
fmeum:20298-compatible-turbine

Conversation

@fmeum
Copy link
Copy Markdown
Collaborator

@fmeum fmeum commented Nov 23, 2023

Work towards #20298

@fmeum fmeum requested a review from a team as a code owner November 23, 2023 19:49
@fmeum fmeum requested a review from cushon November 23, 2023 19:49
@github-actions github-actions Bot added team-Rules-Java Issues for Java rules awaiting-review PR is awaiting review from an assigned reviewer labels Nov 23, 2023
@fmeum fmeum removed the request for review from a team November 23, 2023 19:50
@fmeum fmeum mentioned this pull request Nov 23, 2023
Closed
@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Nov 23, 2023

@bazel-io flag

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Nov 23, 2023
@keertk
Copy link
Copy Markdown
Member

keertk commented Nov 23, 2023

@fmeum is this for 7.0.0 too?

@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Nov 23, 2023

@keertk No, this one can go into 7.1.0.

@keertk
Copy link
Copy Markdown
Member

keertk commented Nov 23, 2023

@bazel-io fork 7.1.0

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Nov 23, 2023
@bazel-io bazel-io mentioned this pull request Nov 23, 2023
Closed
@davido
Copy link
Copy Markdown
Contributor

davido commented Nov 23, 2023 •
edited
Loading

@fmeum

Thanks a lot for doing it.

How can I verify that this PR actually does fix #20298 on my Laptop?

And if it's does, why we don't try to ship a new java_rules release with this fix included in 7.0.0 release to avoid disruption for the folks in the wild with outdated hardware?

@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Nov 23, 2023

How can I verify that this PR actually does fix #20298 on my Laptop?

Could you build //src:java_tools_prebuilt_zip from this branch, unpack it and try to run the java_tools/turbine_direct_graal executable in it without arguments? If it prints a usage message rather than an error, this PR should fix your issue.

And if it's does, why we don't try to ship a new java_rules release with this fix included in 7.0.0 release to avoid disruption for the folks in the wild with outdated hardware?

Bazel 7.0.0 ships with rules_java 7.1.0, which doesn't even contain a native image of Turbine and thus shouldn't be affected by this issue. Once this PR and #20294 have been merged, I would ask for a new patch release that will hopefully become the default in Bazel 7.1.0.

@davido
Copy link
Copy Markdown
Contributor

davido commented Nov 23, 2023 •
edited
Loading

Could you build //src:java_tools_prebuilt_zip from this branch [...]

I cannot build this branch out of the box on my outdated laptop, I guess I need to revert java_rules upgrade on Bazel@HEAD:

davido@localhost:~/projects/bazel (20298-compatible-turbine %)$ bazelisk build //src:java_tools_prebuilt_zip
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_java/releases/download/7.3.0/rules_java-7.3.0.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //src:java_tools_prebuilt_zip (7 packages loaded, 674 targets configured).
INFO: Found 1 target...
ERROR: /home/davido/.cache/bazel/_bazel_davido/0fa756dec521553dbe2dde6b6eac99b4/external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/BUILD:1:13: Compiling Java headers external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/libzip-hjar.jar (1 source file) [for tool] failed: (Exit 1): turbine_direct_graal failed: error executing command (from target @rules_jvm_external~5.2//private/tools/java/com/github/bazelbuild/rules_jvm_external/zip:zip) external/rules_java~7.3.0~toolchains~remote_java_tools_linux/java_tools/turbine_direct_graal --output ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA].
Please rebuild the executable with an appropriate setting of the -march option.Target //src:java_tools_prebuilt_zip failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/davido/.cache/bazel/_bazel_davido/0fa756dec521553dbe2dde6b6eac99b4/external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/BUILD:3:12 Building external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/AddJarManifestEntry.jar (1 source file) [for tool] failed: (Exit 1): turbine_direct_graal failed: error executing command (from target @rules_jvm_external~5.2//private/tools/java/com/github/bazelbuild/rules_jvm_external/zip:zip) external/rules_java~7.3.0~toolchains~remote_java_tools_linux/java_tools/turbine_direct_graal --output ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 20.273s, Critical Path: 0.54s
INFO: 28 processes: 26 internal, 2 linux-sandbox.
FAILED: Build did NOT complete successfully

Of course, I have access to other machines, where I could build it though...and then test the result on my outdated laptop.

Bazel 7.0.0 ships with rules_java 7.1.0, which doesn't even contain a native image of Turbine and thus shouldn't be affected by this issue.

Thanks for clarifying, I assumed for some reason, that if java_rules 7.3.0 release is bumped on Bazel@HEAD, that the final 7.0.0 release would also ship this java_rules release 7.3.0 per default.

@davido
Copy link
Copy Markdown
Contributor

davido commented Nov 23, 2023

@fmeum

Confirmed!

I've built from the branch on another machine:

davido@wizball:~/projects/bazel (20298-compatible-turbine %)$ bazelisk build //src:java_tools_prebuilt_zip
2023/11/23 22:31:20 Downloading https://releases.bazel.build/6.4.0/release/bazel-6.4.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_java/releases/download/7.3.0/rules_java-7.3.0.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
DEBUG: /home/dos/.cache/bazel/_bazel_dos/953b7366ee92774e569a6b4daf38af93/external/rules_jvm_external~5.2/private/extensions/maven.bzl:141:14: The maven repository 'maven' is used in two different bazel modules, originally in 'bazel' and now in 'protobuf'
DEBUG: /home/dos/.cache/bazel/_bazel_dos/953b7366ee92774e569a6b4daf38af93/external/rules_jvm_external~5.2/coursier.bzl:600:18: Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.9.0, 2.8.9
    com.google.errorprone:error_prone_annotations has multiple versions 2.23.0, 2.3.2
    com.google.truth:truth has multiple versions 1.1.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.4.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not get unexpected artifact versions
INFO: Analyzed target //src:java_tools_prebuilt_zip (99 packages loaded, 2764 targets configured).
INFO: Found 1 target...
INFO: From Native Image (fastbuild) //src/java_tools/buildjar/java/com/google/devtools/build/java/turbine:turbine_direct_graal:
========================================================================================================================
GraalVM Native Image: Generating 'turbine_direct_graal' (static executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (8.2s @ 0.17GB)
 Java version: 20.0.2+9, vendor version: GraalVM CE 20.0.2+9.1
 Graal compiler: optimization level: b, target machine: x86-64-v2
 C compiler: gcc (null, null, 0.0.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
[2/8] Performing analysis...  [******]                                                                  (23.9s @ 0.42GB)
   5,260 (80.11%) of  6,566 types reachable
   7,362 (56.64%) of 12,999 fields reachable
  26,876 (49.31%) of 54,507 methods reachable
   1,770 types,    83 fields, and   846 methods registered for reflection
      57 types,    55 fields, and    52 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                               (4.0s @ 0.60GB)
[4/8] Parsing methods...      [**]                                                                       (2.6s @ 0.57GB)
[5/8] Inlining methods...     [***]                                                                      (1.5s @ 0.64GB)
[6/8] Compiling methods...    [***]                                                                     (11.6s @ 0.51GB)
[7/8] Layouting methods...    [**]                                                                       (3.5s @ 0.84GB)
[8/8] Creating image...       [**]                                                                       (4.5s @ 0.65GB)
  10.65MB (47.24%) for code area:    16,239 compilation units
  10.85MB (48.11%) for image heap:  138,983 objects and 5 resources
   1.05MB ( 4.65%) for other data
  22.55MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   5.94MB java.base                                            2.44MB byte[] for code metadata
   1.34MB processed_turbine-0.3.0.jar                          1.35MB java.lang.String
   1.07MB svm.jar (Native Image)                               1.21MB java.lang.Class
 892.06kB libcore.jar                                          1.11MB byte[] for general heap data
 334.65kB guava-32.1.1-jre.jar                                 1.08MB byte[] for java.lang.String
 288.66kB liblite_runtime_only.jar                           452.03kB com.oracle.svm.core.hub.DynamicHubCompanion
 250.32kB jdk.zipfs                                          273.70kB java.util.HashMap$Node
 115.98kB java.logging                                       266.89kB java.lang.String[]
  60.13kB jdk.crypto.ec                                      256.35kB java.lang.Object[]
  56.01kB org.graalvm.nativeimage.base                       253.14kB byte[] for reflection metadata
 236.19kB for 16 more packages                                 2.00MB for 1169 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
                        3.5s (5.7% of total time) in 105 GCs | Peak RSS: 1.67GB | CPU load: 5.63
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /home/dos/.cache/bazel/_bazel_dos/953b7366ee92774e569a6b4daf38af93/sandbox/linux-sandbox/384/execroot/_main/bazel-out/k8-fastbuild/bin/src/java_tools/buildjar/java/com/google/devtools/build/java/turbine/turbine_direct_graal (executable)
========================================================================================================================
Finished generating 'turbine_direct_graal' in 1m 0s.
Target //src:java_tools_prebuilt_zip up-to-date:
  bazel-bin/src/java_tools_prebuilt.zip
INFO: Elapsed time: 242.243s, Critical Path: 115.14s
INFO: 421 processes: 34 internal, 381 linux-sandbox, 6 worker.
INFO: Build completed successfully, 421 total actions

Then I extracted and copied the artifact: turbine_direct_graal to my outdated laptop, and now I'm getting the help prompt, when running it:

davido@localhost:~/projects/bazel (20298-compatible-turbine %)$ file turbine_direct_graal
turbine_direct_graal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=0bc5c26095e615292a2bba22a9ba669aeb3ef463, for GNU/Linux 3.2.0, with debug_info, not stripped

davido@localhost:~/projects/bazel (20298-compatible-turbine %)$ ./turbine_direct_graal
at least one of --output, --gensrc_output, or --resource_output is required

Usage: turbine [options]

Options:
  --output
    The jar output file.
  --sources
    The sources to compile.
  --source_jars
    jar archives of sources to compile.
  --classpath
    The compilation classpath.
  --bootclasspath
    The compilation bootclasspath.
  --help
    Print this usage statement.
  @<filename>
    Read options and filenames from file.

Thanks again for fixing it, really appreciated!

@meteorcloudy meteorcloudy added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Nov 24, 2023
@github-actions github-actions Bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Nov 27, 2023
@fmeum fmeum deleted the 20298-compatible-turbine branch November 27, 2023 09:02
copybara-service Bot pushed a commit to google/google-java-format that referenced this pull request Dec 8, 2023
@cushon
Make g-j-f native image more compatible
a2067bc 
I was reminded of this by bazelbuild/bazel#20306

PiperOrigin-RevId: 585447776
@copybara-service copybara-service Bot mentioned this pull request Dec 8, 2023
Merged
copybara-service Bot pushed a commit to google/google-java-format that referenced this pull request Dec 8, 2023
@cushon
Make g-j-f native image more compatible
425c924 
I was reminded of this by bazelbuild/bazel#20306

PiperOrigin-RevId: 585447776
copybara-service Bot pushed a commit to google/google-java-format that referenced this pull request Dec 8, 2023
@cushon
Make g-j-f native image more compatible
dc8b461 
I was reminded of this by bazelbuild/bazel#20306

PiperOrigin-RevId: 589210771
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@meteorcloudy meteorcloudy meteorcloudy approved these changes

@cushon cushon Awaiting requested review from cushon

Assignees

No one assigned

Labels

team-Rules-Java Issues for Java rules

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@fmeum @keertk @davido @meteorcloudy @bazel-io