[ZEPPELIN-3994] Notebook serving by Leemoonsoo · Pull Request #3356 · apache/zeppelin

Leemoonsoo · 2019-04-25T18:25:49Z

What is this PR for?

This PR implements notebook serving explained here.

While this PR is already quite large, front-end implementation will have separate PR.
This PR is work in progress.

How to run

Prepare kubernetes cluster
Prepare persistentVolumeClaim with name 'task-context-volume-claim' in your kubernetes cluster
Build Zeppelin docker image (follow instructions here
Deploy api router by running kubectl apply -f k8s/serving-api-router/serving-api-router.yaml
Start Zeppelin on Kubernetes by running kubectl apply -f k8s/zeppelin-server.yaml
Port forward to the api router (e.g. kubectl port-forward <zeppelin-serving-api-router pod name> 8080:80) and browse localhost:8080

*Note: This branch does not include front-end implementation. To try with front-end, checkout this branch.

Summary of changes

To be updated

What type of PR is it?

Feature

Todos

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3994

How should this be tested?

First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
Strongly recommended: add automated unit tests for any new or changed behavior
Outline any manual steps to test the PR here.

Questions:

Does the licenses files need update? yes
Is there breaking changes for older versions? no
Does this needs documentation? yes

…er process

### What is this PR for? This PR adds a command-line option to run a single note. The code is pickled from #3356. Usage is ``` bin/zeppelin.sh --run <noteId> ``` The command will launch a Zeppelin server and run a given note, and terminate on after all paragraph's successful run (with exit 0) or terminate on any paragraph error (with exit 1). ### What type of PR is it? Feature ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-4619 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <[email protected]> This patch had conflicts when merged, resolved by Committer: Lee moon soo <[email protected]> Closes #3654 from Leemoonsoo/ZEPPELIN-4619 and squashes the following commits: 6a83163 [Lee moon soo] remove useless test a7531a4 [Lee moon soo] address comment 7743b43 [Lee moon soo] add log 99974f8 [Lee moon soo] overload loading runAllParagraphs() 101d155 [Lee moon soo] add --run option

amit-shahi · 2021-01-12T06:27:30Z

Which version of Zeppelin do I run on Kubernetes to work with Flink 1.11.3? Is there a working Zeppelin Kubernetes YAML that works with Flink?

zjffdu · 2021-01-12T06:34:40Z

@amit-shahi You can download latest zeppelin 0.9 which support flink 1.11, but we haven't verified it on k8s

amit-shahi · 2021-01-12T06:42:25Z

@zjffdu Can I use the Docker Image apache/zeppelin:0.9.0 (https://hub.docker.com/r/apache/zeppelin/tags?page=1&ordering=last_updated)? Also, I have flink running on another POD on the same K8 cluster. Where do I configure Zeppelin to connect to my existing Flink Job Manager POD so Zeppelin can submit JOBS through it's UI?

zjffdu · 2021-01-12T06:46:22Z

I am afraid the docker image of 0.9 is still using 0.9.0-preview2, we are working on it to upgrade to 0.9.0
But it still support flink 1.11 IIRC.

amit-shahi · 2021-01-12T06:54:46Z

@zjffdu Okay. Which Zeppelin version I should from Docker? I have flink running on another POD on the K8s cluster. Where do I configure Zeppelin to connect to my existing Flink Job Manager POD so Zeppelin can submit JOBS through it's UI?

zjffdu · 2021-01-12T07:01:36Z

@amit-shahi You can refer the document here, remote mode is the way you should use I believe, https://app.gitbook.com/@jeffzhang/s/flink-on-zeppelin/execution-mode/remote-mode

amit-shahi · 2021-01-12T07:09:08Z

@zjffdu Thanks. I'm accessing Kubernetes on the Windows system. As per the above documentation, Do I need to set FLINK_HOME to my local Flink directory (not on Kubernetes)? Can I bypass this so it can use Kubernetes Flink's FLINK_HOME reference instead? Where do I configure zeppelin so it can use K8's Flink settings for FLINK_HOME? What's the right way to do that?

zjffdu · 2021-01-12T07:16:38Z

flink should be put in the same host of zeppelin. FLINK_HOME point to that location.

amit-shahi · 2021-01-12T07:22:25Z

@zjffdu I'm not sure if I got that. Let's say on Kubernetes, Flink's Job Manager is installed on POD1 and Task Manager is installed on POD2. When we install Zeppelin through YAML, it gets installed on POD 3, How do I use POD1's (Job Manager) FLINK_HOME location/reference in POD 3 so Zeppelin server can connect to existing Flink installation?

zjffdu · 2021-01-12T07:26:53Z

You need to put flink in pod3 and specify FLINK_HOME to this path. Then specify flink.execution.remote.host and flink.execution.remote.port as the JobManager rest api address

amit-shahi · 2021-01-12T12:58:33Z

@zjffdu I made the changes as suggested.

When I run default Flink Job in Zeppelin, this is what I see-

or

This also causes the Flink Pod Restart which gets created by zeppelin-server.

I'm using zeppelin 0.9.0 from Docker Hub, and Flink 1.11.3. I'm not sure what else I need to change/fix, so I can submit Flink SQL job from zeppelin to main POD1 Flink cluster.

zjffdu · 2021-01-13T07:01:37Z

You need to set zeppelin.run.mode as local in zeppelin-site.xml, otherwise zeppelin would launch flink interpreter in another pod which may cause this issue.

amit-shahi · 2021-01-13T13:43:44Z

@zjffdu Thanks. I'm using this k8 YAML https://github.com/apache/zeppelin/blob/master/k8s/zeppelin-server.yaml

Is there a different image for interpreter apache/zeppelin-interpreter:0.9.0-SNAPSHOT (as mentioned in above YAML)
or can I use apache/zeppelin:0.9.0 from https://hub.docker.com/r/apache/zeppelin/tags as an image for ZEPPELIN_K8S_CONTAINER_IMAGE in zeppelin yaml?

zjffdu · 2021-01-13T13:46:26Z

@amit-shahi Let's discuss that in user mail list.

amit-shahi · 2021-01-13T15:02:39Z

@zjffdu Ok, I will create a thread on the mailing list.

amit-shahi · 2021-01-13T16:20:00Z

@zjffdu When I try to execute this YAML https://github.com/apache/zeppelin/blob/master/k8s/zeppelin-server.yaml thru
kubectl apply -f zeppelin-server.yaml, I see this error-

Is the yaml file correct?

amit-shahi · 2021-01-13T20:41:26Z

@zjffdu How do I set zeppelin.run.mode as local in zeppelin-site.xml?

github-actions · 2025-07-19T02:38:07Z

This pull request has been inactive for over a year. If no further activity occurs within the next 30 days, it will be automatically closed. If you believe this PR is still relevant, please feel free to leave a comment or make an update. Thank you!

github-actions · 2025-08-20T02:34:27Z

This pull request has been automatically closed due to prolonged inactivity (over one year without updates). If you feel this was done in error or would like to continue the discussion, feel free to reopen it. Thank you for your contributions!

Leemoonsoo mentioned this pull request Apr 25, 2019

Notebook serving front-end Leemoonsoo/zeppelin#9

Open

Leemoonsoo mentioned this pull request May 3, 2019

[ZEPPELIN-3840] Zeppelin on Kubernetes #3240

Closed

4 tasks

Leemoonsoo added 21 commits May 16, 2019 12:57

Make InterpreterContext create rest api endpoint running in Interpret…

c0c027d

…er process

implement Serving task

e02b1aa

add serving kubernetes manaifest

6642a40

start stop monitor serving task

10b529b

serving rest api

06c44d1

addRestApi implementation in ZeppelinContext

c982f42

populate informations needed for setting routing table

8b79fb0

wiring router, route table generator

86e63ff

py4j java->python call error workaround

8002a21

refactor PythonRestApiHandler

80974cb

NoteTestTask

9d362ad

test task don't need to create api route

733b6f2

Kubernetes Job for note test task

d397a56

list serving task based on actually running deployment

a84ebfd

task getInfo

1f84e74

serving metric

352108d

task lifecycle listener

985e805

update test task get info rest api

c31a516

custom metrics

679236f

handle disconnect to redis

f839fc3

custom metrics handle num/avg value

bfa38c4

Leemoonsoo force-pushed the note_serving branch from 012c921 to bfa38c4 Compare May 16, 2019 19:59

Leemoonsoo mentioned this pull request Feb 17, 2020

[ZEPPELIN-4619] Run a note from the commandline #3654

Closed

github-actions bot added the Stale label Jul 19, 2025

github-actions bot closed this Aug 20, 2025

Conversation

Leemoonsoo commented Apr 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

How to run

Summary of changes

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Questions:

Uh oh!

amit-shahi commented Jan 12, 2021

Uh oh!

zjffdu commented Jan 12, 2021

Uh oh!

amit-shahi commented Jan 12, 2021

Uh oh!

zjffdu commented Jan 12, 2021

Uh oh!

amit-shahi commented Jan 12, 2021

Uh oh!

zjffdu commented Jan 12, 2021

Uh oh!

amit-shahi commented Jan 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjffdu commented Jan 12, 2021

Uh oh!

amit-shahi commented Jan 12, 2021

Uh oh!

zjffdu commented Jan 12, 2021

Uh oh!

amit-shahi commented Jan 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjffdu commented Jan 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amit-shahi commented Jan 13, 2021

Uh oh!

zjffdu commented Jan 13, 2021

Uh oh!

amit-shahi commented Jan 13, 2021

Uh oh!

amit-shahi commented Jan 13, 2021

Uh oh!

amit-shahi commented Jan 13, 2021

Uh oh!

github-actions bot commented Jul 19, 2025

Uh oh!

github-actions bot commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Leemoonsoo commented Apr 25, 2019 •

edited

Loading

amit-shahi commented Jan 12, 2021 •

edited

Loading

amit-shahi commented Jan 12, 2021 •

edited

Loading

zjffdu commented Jan 13, 2021 •

edited

Loading