[ZEPPELIN-773] Adding support for spark interpreter to work with yarn…#903
[ZEPPELIN-773] Adding support for spark interpreter to work with yarn…#903mfelgamal wants to merge 1 commit intoapache:masterfrom
Conversation
… cluster mode through livy server
|
Hi @mfelgamal, A livy interpreter has recently been added in zeppelin with #827. It works with both the installation of https://github.com/cloudera/hue/tree/master/apps/spark/java and https://github.com/cloudera/livy, is enabled with user impersonation as well. Can you take a look at that as well, and in case if you see short coming in it, may be file a jira for it. |
|
@mfelgamal thank you for your contributions. As you can see we started a PR on Livy interpreter about a month and a half ago, and it has recently been merged. You are very welcome to take a look at that and integrate additional features that we might have missed. More contributions are welcome! |
|
Hi @prabhjyotsingh @felixcheung , the PR in #827 sound like it is the same as our PR. So, I agree with you that we might be duplicating the feature. Anyway, I have suggested a small change in your PR. You can find it at #944 . It mainly adds some important configuration to spark in Yarn mode. |
3 participants
What is this PR for?
Adding support for zeppelin to run in yarn-cluster mode through Livy
What type of PR is it?
Feature
Todos
What is the Jira issue?
How should this be tested?
%livy.sparkto spark,%livy.pysparkto python,%livy.sparkrto R)_If you run on a cluster, the cluster should support yarn, and run livy-server in yarn mode_
Screenshots (if appropriate)
Questions: