[ZEPPELIN-1258] Add Spark packages support to Livy interpreter#1255
[ZEPPELIN-1258] Add Spark packages support to Livy interpreter#1255mfelgamal wants to merge 5 commits intoapache:masterfrom
Conversation
update from original
merge master
|
I would not recommend user to do that to add extra library. Because most of time zeppelin user don't know the cluster details like which/where extra library are located in the cluster machines. I would suggest livy to support dynamic library loading as native spark interpreter. |
|
@zjffdu - Livy doesn't support dynamic library loading right now. So, I don't think there's a ZeppelinContext available in Livy. Would this be acceptable until dynamic library loading is added to Livy and a Zeppelin Context becomes available? As of now, Livy is not that much use if you cannot add libraries to your Spark app. |
|
@jimdowling Right, there's no ZeppelinContext in livy right now. Would allowing user to specify jars/packages when creating livy session more general ? (like --jars/--packages in spark-submit) |
|
Do you mean by modifying the UI on notebook creation or using a param to selecting the interpreter like |
|
No, I mean adding interpreter property to allow user to specify jars/packages. |
|
I think that the livy itself doesn't support add (--jars/--packages property in spark-submit) |
|
You can use |
|
@zjffdu I added |
|
@zjffdu do you have any further comments ? |
docs/interpreter/livy.md
Outdated
| <table class="table-configuration"> | ||
| <tr> | ||
| <th>Property</th> | ||
| <th>Default</th> |
There was a problem hiding this comment.
this wouldn't be the "Default" right?
There was a problem hiding this comment.
True, it's just example.
|
could you update the title of this PR and JIRA ZEPPELIN-1258, if we are changing how we would approach this? |
|
@felixcheung I suggests this title "Adding packages to livy interpreter", Is it Ok ? |
|
how about "Add Spark packages support to Livy interpreter"? |
|
@felixcheung done. |
|
@felixcheung do you have any further comments ? |
|
There is a test failure? |
|
@felixcheung : This PR does not change the code. It is just a change in the configurations. So, it should not affect the tests |
|
That's a fair point. I'll merge if there is no more discussion. |
What is this PR for?
Adding extra libraries to livy interpreter which isn't exist by default.
What type of PR is it?
[ Improvement ]
Todos
What is the Jira issue?
How should this be tested?
livy.spark.jars.packagesto list of maven coordinates of jars. The format for the coordinates should be groupId:artifactId:version.Screenshots (if appropriate)
Questions: