Skip to content

[ ZEPPELIN-1559 ] Code Editor slow performance resolve of the many Notebooks #1533

Closed
cloverhearts wants to merge 2 commits intoapache:masterfrom
cloverhearts:ZEPPELIN-1559-Notebook-broadcast-opti
Closed

[ ZEPPELIN-1559 ] Code Editor slow performance resolve of the many Notebooks #1533
cloverhearts wants to merge 2 commits intoapache:masterfrom
cloverhearts:ZEPPELIN-1559-Notebook-broadcast-opti

Conversation

@cloverhearts
Copy link
Copy Markdown
Member

@cloverhearts cloverhearts commented Oct 17, 2016 •
edited
Loading

What is this PR for?

Currently, if you have a Notebook large number of,
There is a sharp decrease in performance of the Code editor.
The number and Paragraph creation of the Notebook does not have a relationship.
We are should always use the Code editor of the same performance.

I had to print a Notebook list only if there is a request.
There was quite a lot of performance improvements.

What type of PR is it?

Improvement

Todos

  • - change Notebook list render logic

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1559
https://issues.apache.org/jira/browse/ZEPPELIN-1513

How should this be tested?

  1. Create a notebook over 1000. (tested 10,000)
    I've used script on this. (important! please, backup for notebooks directory.)
#/bin/bash

idx="0"

while [ $idx -lt 1000 ] # notebooks count 1000
do
  mkdir $idx
  echo """
{
    \"paragraphs\": [
    {
        \"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\",
            \"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\",
            \"config\": {
                \"colWidth\": 12.0,
            \"graph\": {
                \"mode\": \"table\",
            \"height\": 300.0,
            \"optionOpen\": false,
            \"keys\": [],
            \"values\": [],
            \"groups\": [],
            \"scatter\": {},
            \"map\": {
                \"baseMapType\": \"Streets\",
            \"isOnline\": true,
            \"pinCols\": []
    }
    },
        \"enabled\": true,
        \"editorMode\": \"ace/mode/scala\"
    },
        \"settings\": {
            \"params\": {},
        \"forms\": {}
    },
        \"apps\": [],
        \"jobName\": \"paragraph_1476345687682_-1459653828\",
        \"id\": \"20161013-170127_68316618\",
        \"dateCreated\": \"Oct 13, 2016 5:01:27 PM\",
        \"status\": \"READY\",
        \"progressUpdateIntervalMs\": 500
    }
    ],
        \"name\": \""${idx}"\",
        \"id\": \""${idx}"\",
        \"angularObjects\": {
            \"2BZSC9D3G:shared_process\": [],
        \"2BXGWF5TF:shared_process\": [],
        \"2BYKTADMA:shared_process\": [],
        \"2BYFT9HTZ:shared_process\": [],
        \"2BXDTJMED:shared_process\": [],
        \"2BZHPZ6NS:shared_process\": [],
        \"2BYKRWYB3:shared_process\": [],
        \"2BZH2UAT4:shared_process\": [],
        \"2BZ38J35G:shared_process\": [],
        \"2BXJ2X464:shared_process\": [],
        \"2BYQ57ED9:shared_process\": [],
        \"2BXDSA2SN:shared_process\": [],
        \"2BWZCTD2B:shared_process\": [],
        \"2BWT81MNU:shared_process\": [],
        \"2BWA7ZTRD:shared_process\": [],
        \"2BXW6X5KN:shared_process\": [],
        \"2BVWDQPXH:shared_process\": [],
        \"2BZGAF8KG:shared_process\": []
},
    \"config\": {},
    \"info\": {}
    }
  """ > $idx/note.json

  idx=$[$idx+1]
done
  1. zeppelin restart or refresh notebook.
  2. Try coding in any notebook. (fast)

Screenshots (if appropriate)

before ( 5000 notebooks)

optimization_before_codeditor

after ( 5000 notebooks)

optimization_for_codeeidtor

Questions:

  • Does the licenses files need update? no
  • Is there breaking changes for older versions? no
  • Does this needs documentation? no

@cloverhearts
Copy link
Copy Markdown
Member Author

cloverhearts commented Oct 17, 2016

added jira issue https://issues.apache.org/jira/browse/ZEPPELIN-1513

@cloverhearts cloverhearts reopened this Oct 17, 2016
@Leemoonsoo
Copy link
Copy Markdown
Member

Leemoonsoo commented Oct 17, 2016

Tested and confirmed angular watcher reduced from 5xxx -> 3xx in case of 1k notes. Great work!

corneadoug
corneadoug reviewed Oct 18, 2016
View reviewed changes
zeppelin-web/src/components/navbar/navbar.html Outdated
<ul class="nav navbar-nav" ng-if="ticket">
<li class="dropdown" dropdown>
<a href="#" class="dropdown-toggle" dropdown-toggle>Notebook <span class="caret"></span></a>
<li id="notebook-list-button" class="dropdown notebook-list-dropdown" dropdown>
Copy link
Copy Markdown
Contributor

@corneadoug corneadoug Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like that 'id' is being used

Copy link
Copy Markdown
Member Author

@cloverhearts cloverhearts Oct 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corneadoug
thank you!
Modification was completed. :)

@Leemoonsoo
Copy link
Copy Markdown
Member

Leemoonsoo commented Oct 18, 2016

LGTM and merge if there're no more discussions.

@asfgit asfgit closed this in a3a2e4d Oct 19, 2016
darionyaphet pushed a commit to darionyaphet/zeppelin that referenced this pull request Oct 27, 2016
@cloverhearts @darionyaphet
[ ZEPPELIN-1559 ] Code Editor slow performance resolve of the many No…
89467f2 
…tebooks

### What is this PR for?
Currently, if you have a Notebook large number of,
There is a sharp decrease in performance of the Code editor.
The number and Paragraph creation of the Notebook does not have a relationship.
We are should always use the Code editor of the same performance.

I had to print a Notebook list only if there is a request.
There was quite a lot of performance improvements.

### What type of PR is it?
Improvement

### Todos
- [x] - change Notebook list render logic

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1559
https://issues.apache.org/jira/browse/ZEPPELIN-1513

### How should this be tested?
1. Create a notebook over 1000. (tested 10,000)
   I've used script on this.  (important! please, backup for notebooks directory.)
```
#/bin/bash

idx="0"

while [ $idx -lt 1000 ] # notebooks count 1000
do
  mkdir $idx
  echo """
{
	\"paragraphs\": [
	{
		\"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\",
			\"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\",
			\"config\": {
				\"colWidth\": 12.0,
			\"graph\": {
				\"mode\": \"table\",
			\"height\": 300.0,
			\"optionOpen\": false,
			\"keys\": [],
			\"values\": [],
			\"groups\": [],
			\"scatter\": {},
			\"map\": {
				\"baseMapType\": \"Streets\",
			\"isOnline\": true,
			\"pinCols\": []
	}
	},
		\"enabled\": true,
		\"editorMode\": \"ace/mode/scala\"
	},
		\"settings\": {
			\"params\": {},
		\"forms\": {}
	},
		\"apps\": [],
		\"jobName\": \"paragraph_1476345687682_-1459653828\",
		\"id\": \"20161013-170127_68316618\",
		\"dateCreated\": \"Oct 13, 2016 5:01:27 PM\",
		\"status\": \"READY\",
		\"progressUpdateIntervalMs\": 500
	}
	],
		\"name\": \""${idx}"\",
		\"id\": \""${idx}"\",
		\"angularObjects\": {
			\"2BZSC9D3G:shared_process\": [],
		\"2BXGWF5TF:shared_process\": [],
		\"2BYKTADMA:shared_process\": [],
		\"2BYFT9HTZ:shared_process\": [],
		\"2BXDTJMED:shared_process\": [],
		\"2BZHPZ6NS:shared_process\": [],
		\"2BYKRWYB3:shared_process\": [],
		\"2BZH2UAT4:shared_process\": [],
		\"2BZ38J35G:shared_process\": [],
		\"2BXJ2X464:shared_process\": [],
		\"2BYQ57ED9:shared_process\": [],
		\"2BXDSA2SN:shared_process\": [],
		\"2BWZCTD2B:shared_process\": [],
		\"2BWT81MNU:shared_process\": [],
		\"2BWA7ZTRD:shared_process\": [],
		\"2BXW6X5KN:shared_process\": [],
		\"2BVWDQPXH:shared_process\": [],
		\"2BZGAF8KG:shared_process\": []
},
	\"config\": {},
	\"info\": {}
	}
  """ > $idx/note.json

  idx=$[$idx+1]
done

```

2. zeppelin restart or refresh notebook.
3. Try coding in any notebook. (fast)

### Screenshots (if appropriate)
#### before ( 5000 notebooks)
![optimization_before_codeditor](https://cloud.githubusercontent.com/assets/10525473/19457150/f18845a0-94ff-11e6-876d-4386dbf1e7de.gif)

#### after ( 5000 notebooks)
![optimization_for_codeeidtor](https://cloud.githubusercontent.com/assets/10525473/19457184/1f7d5036-9500-11e6-8b0c-91b301641c73.gif)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no

Author: CloverHearts <[email protected]>

Closes apache#1533 from cloverhearts/ZEPPELIN-1559-Notebook-broadcast-opti and squashes the following commits:

bd58102 [CloverHearts] id Delete unused for navbar
14e88bb [CloverHearts] selectively modified to render the Notebook list.
pedrozatta pushed a commit to pedrozatta/zeppelin that referenced this pull request Oct 27, 2016
@cloverhearts
[ ZEPPELIN-1559 ] Code Editor slow performance resolve of the many No…
564b16b 
…tebooks

### What is this PR for?
Currently, if you have a Notebook large number of,
There is a sharp decrease in performance of the Code editor.
The number and Paragraph creation of the Notebook does not have a relationship.
We are should always use the Code editor of the same performance.

I had to print a Notebook list only if there is a request.
There was quite a lot of performance improvements.

### What type of PR is it?
Improvement

### Todos
- [x] - change Notebook list render logic

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1559
https://issues.apache.org/jira/browse/ZEPPELIN-1513

### How should this be tested?
1. Create a notebook over 1000. (tested 10,000)
   I've used script on this.  (important! please, backup for notebooks directory.)
```
#/bin/bash

idx="0"

while [ $idx -lt 1000 ] # notebooks count 1000
do
  mkdir $idx
  echo """
{
	\"paragraphs\": [
	{
		\"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\",
			\"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\",
			\"config\": {
				\"colWidth\": 12.0,
			\"graph\": {
				\"mode\": \"table\",
			\"height\": 300.0,
			\"optionOpen\": false,
			\"keys\": [],
			\"values\": [],
			\"groups\": [],
			\"scatter\": {},
			\"map\": {
				\"baseMapType\": \"Streets\",
			\"isOnline\": true,
			\"pinCols\": []
	}
	},
		\"enabled\": true,
		\"editorMode\": \"ace/mode/scala\"
	},
		\"settings\": {
			\"params\": {},
		\"forms\": {}
	},
		\"apps\": [],
		\"jobName\": \"paragraph_1476345687682_-1459653828\",
		\"id\": \"20161013-170127_68316618\",
		\"dateCreated\": \"Oct 13, 2016 5:01:27 PM\",
		\"status\": \"READY\",
		\"progressUpdateIntervalMs\": 500
	}
	],
		\"name\": \""${idx}"\",
		\"id\": \""${idx}"\",
		\"angularObjects\": {
			\"2BZSC9D3G:shared_process\": [],
		\"2BXGWF5TF:shared_process\": [],
		\"2BYKTADMA:shared_process\": [],
		\"2BYFT9HTZ:shared_process\": [],
		\"2BXDTJMED:shared_process\": [],
		\"2BZHPZ6NS:shared_process\": [],
		\"2BYKRWYB3:shared_process\": [],
		\"2BZH2UAT4:shared_process\": [],
		\"2BZ38J35G:shared_process\": [],
		\"2BXJ2X464:shared_process\": [],
		\"2BYQ57ED9:shared_process\": [],
		\"2BXDSA2SN:shared_process\": [],
		\"2BWZCTD2B:shared_process\": [],
		\"2BWT81MNU:shared_process\": [],
		\"2BWA7ZTRD:shared_process\": [],
		\"2BXW6X5KN:shared_process\": [],
		\"2BVWDQPXH:shared_process\": [],
		\"2BZGAF8KG:shared_process\": []
},
	\"config\": {},
	\"info\": {}
	}
  """ > $idx/note.json

  idx=$[$idx+1]
done

```

2. zeppelin restart or refresh notebook.
3. Try coding in any notebook. (fast)

### Screenshots (if appropriate)
#### before ( 5000 notebooks)
![optimization_before_codeditor](https://cloud.githubusercontent.com/assets/10525473/19457150/f18845a0-94ff-11e6-876d-4386dbf1e7de.gif)

#### after ( 5000 notebooks)
![optimization_for_codeeidtor](https://cloud.githubusercontent.com/assets/10525473/19457184/1f7d5036-9500-11e6-8b0c-91b301641c73.gif)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no

Author: CloverHearts <[email protected]>

Closes apache#1533 from cloverhearts/ZEPPELIN-1559-Notebook-broadcast-opti and squashes the following commits:

bd58102 [CloverHearts] id Delete unused for navbar
14e88bb [CloverHearts] selectively modified to render the Notebook list.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@corneadoug corneadoug corneadoug left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cloverhearts @Leemoonsoo @corneadoug