GH-15265: [Java] Publish SBOM artifacts by dongjoon-hyun · Pull Request #15267 · apache/arrow

dongjoon-hyun · 2023-01-09T09:18:40Z

This closes #15265

Closes: [Java] Publish SBOM artifacts #15265

github-actions · 2023-01-09T09:19:03Z

Closes: [Java] Publish SBOM artifacts #15265

github-actions · 2023-01-09T09:19:05Z

⚠️ GitHub issue #15265 has been automatically assigned in GitHub to PR creator.

assignUser · 2023-01-09T12:12:05Z

Should the sbom be published with the nightlies?

kou · 2023-01-09T12:26:26Z

Could you also update ci/scripts/java_full_build.sh, dev/tasks/java-jars/github.yml and dev/tasks/tasks.yml (the following diff is incomplete for this file) like the following to publish them?

diff --git a/ci/scripts/java_full_build.sh b/ci/scripts/java_full_build.sh
index 1c07971bcc..2734f3e9db 100755
--- a/ci/scripts/java_full_build.sh
+++ b/ci/scripts/java_full_build.sh
@@ -65,7 +65,13 @@ find . \
      -exec echo {} ";" \
      -exec cp {} $dist_dir ";"
 find ~/.m2/repository/org/apache/arrow \
-     "(" -name "*.jar" -o -name "*.zip" -o -name "*.pom" ")" \
+     "(" \
+     -name "*.jar" -o \
+     -name "*.json" -o \
+     -name "*.pom" -o \
+     -name "*.xml" -o \
+     -name "*.zip" \
+     ")" \
      -exec echo {} ";" \
      -exec cp {} $dist_dir ";"
 
diff --git a/dev/tasks/java-jars/github.yml b/dev/tasks/java-jars/github.yml
index cfa1dbed49..c42c9e6f4d 100644
--- a/dev/tasks/java-jars/github.yml
+++ b/dev/tasks/java-jars/github.yml
@@ -204,5 +204,7 @@ jobs:
             $GITHUB_WORKSPACE/arrow \
             $GITHUB_WORKSPACE/arrow/java-dist
       {{ macros.github_upload_releases(["arrow/java-dist/*.jar",
+                                        "arrow/java-dist/*.pson",
                                         "arrow/java-dist/*.pom",
+                                        "arrow/java-dist/*.xml",
                                         "arrow/java-dist/*.zip"])|indent }}
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index 66df61e215..8437ad0778 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -801,6 +801,9 @@ tasks:
     ci: github
     template: java-jars/github.yml
     artifacts:
+      - arrow-algorithm-{no_rc_snapshot_version}-cyclonedx.json
+      - arrow-algorithm-{no_rc_snapshot_version}-cyclonedx.xml
+      - arrow-algorithm-{no_rc_snapshot_version}-javadoc.jar
       - arrow-algorithm-{no_rc_snapshot_version}-javadoc.jar
       - arrow-algorithm-{no_rc_snapshot_version}-sources.jar
       - arrow-algorithm-{no_rc_snapshot_version}-tests.jar
# NOTE!!! We need to add more entries for *-cyclonedx.{json,xml}

dongjoon-hyun · 2023-01-09T17:45:25Z

Thank you so much, @assignUser and @kou . I addressed your comments.

viirya · 2023-01-09T18:21:53Z

ci/scripts/java_full_build.sh

Hmm, I think the previous comment diff points to L73 instead of L63?

diff --git a/ci/scripts/java_full_build.sh b/ci/scripts/java_full_build.sh index 1c07971bcc..2734f3e9db 100755 --- a/ci/scripts/java_full_build.sh +++ b/ci/scripts/java_full_build.sh @@ -65,7 +65,13 @@ find . \ -exec echo {} ";" \ -exec cp {} $dist_dir ";" find ~/.m2/repository/org/apache/arrow \ - "(" -name "*.jar" -o -name "*.zip" -o -name "*.pom" ")" \ + "(" \ + -name "*.jar" -o \ + -name "*.json" -o \ + -name "*.pom" -o \ + -name "*.xml" -o \ + -name "*.zip" \ + ")" \ -exec echo {} ";" \ -exec cp {} $dist_dir ";"

I.e., the one finding the files in local repo dir.

Oh, you are right. I'll fix it soon.

dongjoon-hyun · 2023-01-09T19:00:00Z

dev/tasks/tasks.yml

Oh, do we need to repeat this for all jars like arrow-avro/arrow-c-data/arrow-compression/...?

I think so. That is why @kou said the diff is incomplete for dev/tasks/tasks.yml and there is a # NOTE!!! We need to add more entries for *-cyclonedx.{json,xml}.

Got it. Thank you for the confirmation.

I added them all at de44a62

assignUser · 2023-01-09T23:34:14Z

@github-actions crossbow submit java-jars

assignUser · 2023-01-09T23:36:13Z

To have them uploaded to nightlies.apache.org the nightly java job needs to be updated to: https://github.com/apache/arrow/blob/master/.github/workflows/java_nightly.yml#L110

github-actions · 2023-01-09T23:37:02Z

Revision: de44a621698683f1891e0e1886a23569d2998eb3

Submitted crossbow builds: ursacomputing/crossbow @ actions-588eb7d929

Task	Status
java-jars

dongjoon-hyun · 2023-01-10T00:32:15Z

Thank you, @assignUser . .json and .xml patterns are added.

dongjoon-hyun · 2023-01-10T21:39:42Z

The python failure looks irrelevant to this Java plugin PR. Please let me know if there is some thing I need to do more, @assignUser , @kou , @viirya . Thank you in advance.

viirya

Yea, the failure looks unrelated. This looks good to me.

assignUser · 2023-01-10T23:55:31Z

Yeah the failure of the crossbow job is due to C++ stuff but it also means that the changed logic won't be tested. As this won't succeed and the changes from this PR wont be used (unless built locally ofc) until that is fixed I would prefer to wait for #25633 to be closed so we can properly test this.

For the SBOM files to be uploaded with the official releases we will also need to update https://github.com/apache/arrow/blob/-/dev/release/06-java-upload.sh which could be done as a follow up if the creation of the sbom artifacts for local builds is important for you.

kou · 2023-01-11T06:16:49Z

For the SBOM files to be uploaded with the official releases we will also need to update https://github.com/apache/arrow/blob/-/dev/release/06-java-upload.sh

It seems that we don't need it. https://github.com/apache/arrow/blob/master/dev/release/06-java-upload.sh#L123 will collect new files.

dongjoon-hyun · 2023-01-15T07:05:31Z

According to the above comment about 06-java-upload.sh#L123, we don't need additional change. *-cyclone* will be considered.

dongjoon-hyun · 2023-01-15T07:06:01Z

Please let me know if there is something I can help, @assignUser .

assignUser · 2023-01-17T04:31:16Z

@dongjoon-hyun please rebase, the fix for the c++ issue was merged. Afterwards please trigger the crossbow with @github-actions crossbow submit java-jars as a comment

assignUser · 2023-01-17T04:32:37Z

Once the job complets succesfully you can take a look at the artifacts if they correctly contian the sbom. I will merge then and it will be part of 11.0.0
cc @raulcd

dongjoon-hyun · 2023-01-17T04:32:42Z

Got it, @assignUser !

dongjoon-hyun · 2023-01-17T04:33:42Z

@github-actions crossbow submit java-jars

github-actions · 2023-01-17T04:36:27Z

Revision: 97af0c9

Submitted crossbow builds: ursacomputing/crossbow @ actions-18dd62a120

Task	Status
java-jars

assignUser

SBOM artifacts are uploaded to releases and will thus be picked up by the release scripts https://github.com/ursacomputing/crossbow/releases/tag/actions-18dd62a120-github-java-jars

This closes #15265 * Closes: #15265 Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Jacob Wujciak-Jens <[email protected]>

ursabot · 2023-01-18T09:24:01Z

Benchmark runs are scheduled for baseline = 641d1da and contender = 5580f27. 5580f27 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.03%] test-mac-arm
[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x
[Finished ⬇️0.44% ⬆️0.34%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 5580f27f ec2-t3-xlarge-us-east-2
[Failed] 5580f27f test-mac-arm
[Finished] 5580f27f ursa-i9-9960x
[Finished] 5580f27f ursa-thinkcentre-m75q
[Finished] 641d1da6 ec2-t3-xlarge-us-east-2
[Finished] 641d1da6 test-mac-arm
[Finished] 641d1da6 ursa-i9-9960x
[Finished] 641d1da6 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

This closes apache#15265 * Closes: apache#15265 Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Jacob Wujciak-Jens <[email protected]>

github-actions bot added the Component: Java label Jan 9, 2023

viirya reviewed Jan 9, 2023

View reviewed changes

dongjoon-hyun commented Jan 9, 2023

View reviewed changes

viirya approved these changes Jan 10, 2023

View reviewed changes

dongjoon-hyun added 5 commits January 16, 2023 20:33

GH-15265: [Java] Publish SBOM artifacts

035ee69

Address comment

785ceac

fix

a2f604e

enumerate for all jars

0bd0900

Add .json and .xml

97af0c9

dongjoon-hyun requested review from assignUser, kou, lidavidm and raulcd as code owners January 17, 2023 04:33

assignUser approved these changes Jan 17, 2023

View reviewed changes

assignUser merged commit 5580f27 into apache:master Jan 17, 2023

dongjoon-hyun deleted the GH-15265 branch January 17, 2023 06:21

raulcd pushed a commit that referenced this pull request Jan 18, 2023

GH-15265: [Java] Publish SBOM artifacts (#15267)

7b759bc

This closes #15265 * Closes: #15265 Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Jacob Wujciak-Jens <[email protected]>

kou mentioned this pull request Apr 12, 2023

[Java][CI] Failed to generate SBOM by CycloneDX #35086

Closed

Conversation

dongjoon-hyun commented Jan 9, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 9, 2023

Uh oh!

github-actions bot commented Jan 9, 2023

Uh oh!

assignUser commented Jan 9, 2023

Uh oh!

kou commented Jan 9, 2023

Uh oh!

dongjoon-hyun commented Jan 9, 2023

Uh oh!

viirya Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

viirya Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 9, 2023

Choose a reason for hiding this comment

Uh oh!

assignUser commented Jan 9, 2023

Uh oh!

assignUser commented Jan 9, 2023

Uh oh!

github-actions bot commented Jan 9, 2023

Uh oh!

dongjoon-hyun commented Jan 10, 2023

Uh oh!

dongjoon-hyun commented Jan 10, 2023

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

assignUser commented Jan 10, 2023

Uh oh!

kou commented Jan 11, 2023

Uh oh!

dongjoon-hyun commented Jan 15, 2023

Uh oh!

dongjoon-hyun commented Jan 15, 2023

Uh oh!

assignUser commented Jan 17, 2023

Uh oh!

assignUser commented Jan 17, 2023

Uh oh!

dongjoon-hyun commented Jan 17, 2023

Uh oh!

dongjoon-hyun commented Jan 17, 2023

Uh oh!

github-actions bot commented Jan 17, 2023

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Jan 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dongjoon-hyun commented Jan 9, 2023 •

edited by github-actions bot

Loading