[ZEPPELIN-759] Spark 2.0 support by Leemoonsoo · Pull Request #1195 · apache/zeppelin

Leemoonsoo · 2016-07-16T02:50:43Z

What is this PR for?

This PR implement spark 2.0 support based on #747.
This PR has approach from #980 which is reimplementing code in scala.

You can try build this branch

mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests

What type of PR is it?

Improvements

Todos

- Spark 2.0 support
- Rebase after [ZEPPELIN-605] Add support for Scala 2.11 #747 merge
- Update LICENSE file
- Update related document (build)

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-759

How should this be tested?

Build and try

mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests

Screenshots (if appropriate)

Questions:

Does the licenses files need update? yes
Is there breaking changes for older versions? no
Does this needs documentation? yes

Leemoonsoo · 2016-07-18T03:12:02Z

Now CI is green. Review this please.

CI profile for spark 2.0 force spark version 2.0.0-preview instead of 2.0.0, because of spark 2.0.0 is not yet released. If this PR merges before spark 2.0.0 release, i'll create a separate issue to change 2.0.0-preview to 2.0.0.

bzz · 2016-07-19T00:21:30Z

Looks great to me

minahlee · 2016-07-19T08:24:26Z

Shall we also change spark version in https://github.com/apache/zeppelin/blob/master/r/pom.xml#L386 and https://github.com/apache/zeppelin/blob/master/r/pom.xml#L398 ?

Leemoonsoo · 2016-07-20T02:16:08Z

@minahlee I was trying to make 'zeppelin-spark' interpreter support spark 2.0 in this PR which includes Sparkr support. If there're user demands, spark 2.0 support in 'zeppelin-zrinterpreter' can be addressed in separate issue.

minahlee · 2016-07-20T02:55:47Z

After I look close the pom files, I don't see clear reason why spark.version is specified in scala-2.11 and scala-2.10 profiles.
https://github.com/apache/zeppelin/blob/master/spark-dependencies/pom.xml#L347
https://github.com/apache/zeppelin/blob/master/r/pom.xml#L386
https://github.com/apache/zeppelin/blob/master/r/pom.xml#L398

And I think it's always good to ship default spark version as the latest one. Can we also update below lines?
https://github.com/apache/zeppelin/blob/master/spark-dependencies/pom.xml#L39
https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L41

echarles · 2016-07-20T16:33:20Z

Tested this with spark 2.0 and scala 2.11.
Spark 2.0 works well with scala, R and python.

@minahlee comment makes sense - we should ship the latest version.

minahlee · 2016-07-22T08:52:03Z

Changes looks good. Could you rebase to master?

…e test

Leemoonsoo · 2016-07-23T12:57:22Z

CI is green. Merge it to master and branch-0.6 if there're no more discussions

This PR implement spark 2.0 support based on #747. This PR has approach from #980 which is reimplementing code in scala. You can try build this branch ``` mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests ``` Improvements * [x] - Spark 2.0 support * [x] - Rebase after #747 merge * [x] - Update LICENSE file * [x] - Update related document (build) https://issues.apache.org/jira/browse/ZEPPELIN-759 Build and try ``` mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests ``` ![spark2](https://cloud.githubusercontent.com/assets/1540981/16771611/fe804038-4805-11e6-8447-3fa4258bb51d.gif) * Does the licenses files need update? yes * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Lee moon soo <[email protected]> Closes #1195 from Leemoonsoo/spark-20 and squashes the following commits: d78b322 [Lee moon soo] trigger ci 8017e8b [Lee moon soo] Remove unnecessary spark.version property e3141bd [Lee moon soo] restart sparkcluster before sparkr test 1493b2c [Lee moon soo] print spark standalone cluster log when ci test fails a208cd0 [Lee moon soo] Debug sparkRTest 31369c6 [Lee moon soo] Update license 293896a [Lee moon soo] Update build instruction 862ff6c [Lee moon soo] Make ZeppelinSparkClusterTest.java work with spark 2 839912a [Lee moon soo] Update SPARK_HOME directory detection pattern for 2.0.0-preview in the test 3413707 [Lee moon soo] Update .travis.yml 02bcd5d [Lee moon soo] Update SparkSqlInterpreterTest f06a2fa [Lee moon soo] Spark 2.0 support (cherry picked from commit 8546666) Signed-off-by: Lee moon soo <[email protected]>

…ala 211 artifacts ### What is this PR for? - Update publish_release.sh to publish scala 2.10 and scala 2.11 artifacts to maven repository. - Set build spark version to 2.0 This PR supposed to work once #1186 and #1195 is merged ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-1180](https://issues.apache.org/jira/browse/ZEPPELIN-1180) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes #1209 from minahlee/ZEPPELIN-1180 and squashes the following commits: 40e5718 [Mina Lee] Use underscore as word separator in file name for consistency 0a91932 [Mina Lee] Include all module for scala 2.11 artifact build c1d6439 [Mina Lee] publish 2.11 artifact to maven a7918f0 [Mina Lee] Fix mod from 755 to 644 bca1af1 [Mina Lee] Fix indentation

…ala 211 artifacts ### What is this PR for? - Update publish_release.sh to publish scala 2.10 and scala 2.11 artifacts to maven repository. - Set build spark version to 2.0 This PR supposed to work once #1186 and #1195 is merged ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-1180](https://issues.apache.org/jira/browse/ZEPPELIN-1180) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes #1209 from minahlee/ZEPPELIN-1180 and squashes the following commits: 40e5718 [Mina Lee] Use underscore as word separator in file name for consistency 0a91932 [Mina Lee] Include all module for scala 2.11 artifact build c1d6439 [Mina Lee] publish 2.11 artifact to maven a7918f0 [Mina Lee] Fix mod from 755 to 644 bca1af1 [Mina Lee] Fix indentation (cherry picked from commit 1077921) Signed-off-by: Mina Lee <[email protected]>

### What is this PR for? This PR implement spark 2.0 support based on apache#747. This PR has approach from apache#980 which is reimplementing code in scala. You can try build this branch ``` mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests ``` ### What type of PR is it? Improvements ### Todos * [x] - Spark 2.0 support * [x] - Rebase after apache#747 merge * [x] - Update LICENSE file * [x] - Update related document (build) ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-759 ### How should this be tested? Build and try ``` mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0-preview -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests ``` ### Screenshots (if appropriate) ![spark2](https://cloud.githubusercontent.com/assets/1540981/16771611/fe804038-4805-11e6-8447-3fa4258bb51d.gif) ### Questions: * Does the licenses files need update? yes * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Lee moon soo <[email protected]> Closes apache#1195 from Leemoonsoo/spark-20 and squashes the following commits: d78b322 [Lee moon soo] trigger ci 8017e8b [Lee moon soo] Remove unnecessary spark.version property e3141bd [Lee moon soo] restart sparkcluster before sparkr test 1493b2c [Lee moon soo] print spark standalone cluster log when ci test fails a208cd0 [Lee moon soo] Debug sparkRTest 31369c6 [Lee moon soo] Update license 293896a [Lee moon soo] Update build instruction 862ff6c [Lee moon soo] Make ZeppelinSparkClusterTest.java work with spark 2 839912a [Lee moon soo] Update SPARK_HOME directory detection pattern for 2.0.0-preview in the test 3413707 [Lee moon soo] Update .travis.yml 02bcd5d [Lee moon soo] Update SparkSqlInterpreterTest f06a2fa [Lee moon soo] Spark 2.0 support

…ala 211 artifacts ### What is this PR for? - Update publish_release.sh to publish scala 2.10 and scala 2.11 artifacts to maven repository. - Set build spark version to 2.0 This PR supposed to work once apache#1186 and apache#1195 is merged ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-1180](https://issues.apache.org/jira/browse/ZEPPELIN-1180) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes apache#1209 from minahlee/ZEPPELIN-1180 and squashes the following commits: 40e5718 [Mina Lee] Use underscore as word separator in file name for consistency 0a91932 [Mina Lee] Include all module for scala 2.11 artifact build c1d6439 [Mina Lee] publish 2.11 artifact to maven a7918f0 [Mina Lee] Fix mod from 755 to 644 bca1af1 [Mina Lee] Fix indentation

Leemoonsoo force-pushed the spark-20 branch 4 times, most recently from 706b130 to 4d64187 Compare July 17, 2016 19:51

Leemoonsoo changed the title ~~[WIP] [ZEPPELIN-759] Spark 2.0 support~~ [ZEPPELIN-759] Spark 2.0 support Jul 18, 2016

minahlee mentioned this pull request Jul 20, 2016

[ZEPPELIN-1180] Update publish_release.sh to publish scala 210 and scala 211 artifacts #1209

Closed

echarles mentioned this pull request Jul 20, 2016

[ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.11 #980

Closed

4 tasks

Leemoonsoo force-pushed the spark-20 branch 6 times, most recently from 4414709 to 9321888 Compare July 22, 2016 02:41

Leemoonsoo force-pushed the spark-20 branch 4 times, most recently from bd118c1 to ec67164 Compare July 22, 2016 17:50

Leemoonsoo added 6 commits July 23, 2016 02:53

Spark 2.0 support

f06a2fa

Update SparkSqlInterpreterTest

02bcd5d

Update .travis.yml

3413707

Update SPARK_HOME directory detection pattern for 2.0.0-preview in th…

839912a

…e test

Make ZeppelinSparkClusterTest.java work with spark 2

862ff6c

Update build instruction

293896a

Leemoonsoo added 4 commits July 23, 2016 02:53

Update license

31369c6

Debug sparkRTest

a208cd0

print spark standalone cluster log when ci test fails

1493b2c

restart sparkcluster before sparkr test

e3141bd

Leemoonsoo force-pushed the spark-20 branch from ec67164 to 8017e8b Compare July 22, 2016 22:07

Leemoonsoo added 2 commits July 23, 2016 07:10

Remove unnecessary spark.version property

8017e8b

trigger ci

d78b322

asfgit closed this in 8546666 Jul 24, 2016

minahlee mentioned this pull request Jul 24, 2016

[HOTFIX] add scala version to spark-dependencies artifact suffix #1222

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-759] Spark 2.0 support#1195

[ZEPPELIN-759] Spark 2.0 support#1195
Leemoonsoo wants to merge 12 commits intoapache:masterfrom
Leemoonsoo:spark-20

Leemoonsoo commented Jul 16, 2016 •

edited

Loading

Uh oh!

Leemoonsoo commented Jul 18, 2016

Uh oh!

bzz commented Jul 19, 2016

Uh oh!

minahlee commented Jul 19, 2016

Uh oh!

Leemoonsoo commented Jul 20, 2016 •

edited

Loading

Uh oh!

minahlee commented Jul 20, 2016

Uh oh!

echarles commented Jul 20, 2016

Uh oh!

minahlee commented Jul 22, 2016

Uh oh!

Leemoonsoo commented Jul 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Leemoonsoo commented Jul 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

Leemoonsoo commented Jul 18, 2016

Uh oh!

bzz commented Jul 19, 2016

Uh oh!

minahlee commented Jul 19, 2016

Uh oh!

Leemoonsoo commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

minahlee commented Jul 20, 2016

Uh oh!

echarles commented Jul 20, 2016

Uh oh!

minahlee commented Jul 22, 2016

Uh oh!

Leemoonsoo commented Jul 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Leemoonsoo commented Jul 16, 2016 •

edited

Loading

Leemoonsoo commented Jul 20, 2016 •

edited

Loading