Skip to content

Adding main method for LaunchDataIngestionJobCommand for spark-submit command #8168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiangfu0 merged 1 commit into from
Feb 10, 2022
Merged

Adding main method for LaunchDataIngestionJobCommand for spark-submit command #8168

xiangfu0 merged 1 commit into from
Feb 10, 2022

Conversation

@xiangfu0
Copy link
Contributor

@xiangfu0 xiangfu0 commented Feb 8, 2022 •
edited
Loading

Description

Adding main method for LaunchDataIngestionJobCommand for spark-submit command

Got this exception when try to use class: org.apache.pinot.tools.admin.PinotAdministrator

22/02/08 13:51:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.apache.pinot.tools.admin.command.StartKafkaCommand.<init>(StartKafkaCommand.java:51)
	at org.apache.pinot.tools.admin.PinotAdministrator.<clinit>(PinotAdministrator.java:98)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:806)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.NoSuchElementException
	at java.base/java.util.ServiceLoader$2.next(ServiceLoader.java:1308)
	at java.base/java.util.ServiceLoader$2.next(ServiceLoader.java:1296)
	at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1394)
	at org.apache.pinot.tools.utils.KafkaStarterUtils.getKafkaConnectorPackageName(KafkaStarterUtils.java:54)
	at org.apache.pinot.tools.utils.KafkaStarterUtils.<clinit>(KafkaStarterUtils.java:46)
	... 12 more

Upgrade Notes

Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

  • Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

  • Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

  • New configuration options
  • Deprecation of configurations
  • Signature changes to public methods/interfaces
  • New plugins added or old plugins removed
  • Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

@xiangfu0 xiangfu0 force-pushed the adding_main_method_for_LaunchDataIngestionJobCommand branch 2 times, most recently from fd6f00e to 1974bd0 Compare February 8, 2022 23:03
@codecov-commenter
Copy link

codecov-commenter commented Feb 8, 2022 •
edited
Loading

Codecov Report

Merging #8168 (78b069d) into master (8d867ed) will increase coverage by 0.03%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #8168      +/-   ##
============================================
+ Coverage     71.35%   71.38%   +0.03%     
+ Complexity     4306     4303       -3     
============================================
  Files          1623     1623              
  Lines         84314    84314              
  Branches      12640    12640              
============================================
+ Hits          60163    60191      +28     
+ Misses        20035    20000      -35     
- Partials       4116     4123       +7
Flag Coverage Δ
integration1 28.80% <ø> (-0.04%) ⬇️
integration2 27.57% <ø> (+0.02%) ⬆️
unittests1 67.89% <ø> (-0.04%) ⬇️
unittests2 14.20% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...a/manager/realtime/RealtimeSegmentDataManager.java 50.00% <0.00%> (-25.00%) ⬇️
...nt/local/startree/v2/store/StarTreeDataSource.java 40.00% <0.00%> (-13.34%) ⬇️
...ot/segment/local/startree/OffHeapStarTreeNode.java 72.22% <0.00%> (-5.56%) ⬇️
...core/startree/operator/StarTreeFilterOperator.java 87.41% <0.00%> (-3.50%) ⬇️
...gregation/function/StUnionAggregationFunction.java 73.52% <0.00%> (-2.95%) ⬇️
...roller/helix/core/relocation/SegmentRelocator.java 75.67% <0.00%> (-2.71%) ⬇️
.../java/org/apache/pinot/spi/data/TimeFieldSpec.java 88.63% <0.00%> (-2.28%) ⬇️
.../org/apache/pinot/core/startree/StarTreeUtils.java 69.89% <0.00%> (-2.16%) ⬇️
...lix/core/realtime/PinotRealtimeSegmentManager.java 82.72% <0.00%> (-1.05%) ⬇️
...manager/realtime/LLRealtimeSegmentDataManager.java 70.57% <0.00%> (-1.05%) ⬇️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d867ed...78b069d. Read the comment docs.

@xiangfu0 xiangfu0 force-pushed the adding_main_method_for_LaunchDataIngestionJobCommand branch from 1974bd0 to 78b069d Compare February 10, 2022 05:13
@xiangfu0 xiangfu0 merged commit b404e4a into apache:master Feb 10, 2022
@xiangfu0 xiangfu0 deleted the adding_main_method_for_LaunchDataIngestionJobCommand branch February 10, 2022 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jackie-Jiang Jackie-Jiang Jackie-Jiang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xiangfu0 @codecov-commenter @Jackie-Jiang