ZEPPELIN-55 Make tutorial notebook independent from filesystem.#140
Closed
Leemoonsoo wants to merge 1 commit intoapache:masterfrom
Closed
ZEPPELIN-55 Make tutorial notebook independent from filesystem.#140Leemoonsoo wants to merge 1 commit intoapache:masterfrom
Leemoonsoo wants to merge 1 commit intoapache:masterfrom
Conversation
Member
Author
|
Ready to merge |
Member
Author
|
Merging if there're no more discussions. |
asfgit
pushed a commit
that referenced
this pull request
Jul 5, 2015
Tutorial notebook is downloading data using `wget` and unzip and load the csv file. This works only in local-mode and not going to work with cluster deployments. Discussed solution in the issue ZEPPELIN-55 are * Upload data to HDFS * Upload data to S3 However, not all user will install HDFS, and accessing S3 via hdfs client needs accessKey and secretKey in configuration. this PR make tutorial notebook independent from any filesystem, by reading data from http(s) address and parallelize directly. Here's how this PR loads data ``` // load bank data val bankText = sc.parallelize( IOUtils.toString( new URL("https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv"), Charset.forName("utf8")).split("\n")) case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer) val bank = bankText.map(s => s.split(";")).filter(s => s(0) != "\"age\"").map( s => Bank(s(0).toInt, s(1).replaceAll("\"", ""), s(2).replaceAll("\"", ""), s(3).replaceAll("\"", ""), s(5).replaceAll("\"", "").toInt ) ).toDF() bank.registerTempTable("bank") ``` Author: Lee moon soo <[email protected]> Closes #140 from Leemoonsoo/ZEPPELIN-55 and squashes the following commits: 653b1bc [Lee moon soo] Load data directly from http without using filesystem (cherry picked from commit 4fa7019) Signed-off-by: Lee moon soo <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Jun 22, 2016
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tutorial notebook is downloading data using
wgetand unzip and load the csv file.This works only in local-mode and not going to work with cluster deployments.
Discussed solution in the issue ZEPPELIN-55 are
However, not all user will install HDFS, and accessing S3 via hdfs client needs accessKey and secretKey in configuration.
this PR make tutorial notebook independent from any filesystem, by reading data from http(s) address and parallelize directly.
Here's how this PR loads data