SPARK-1408 Modify Spark on Yarn to point to the history server when app ... #362
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
docs/configuration.md
Outdated
There was a problem hiding this comment.
It seems from your other changes that the default is "" instead of localhost:18080. Also, nit: spark.historyServer.address
There was a problem hiding this comment.
Also, from this description alone it's not clear to me when I should set this. If I'm trying to start a history server independently, I might get confused about this parameter and try to use it to change the port. On the other hand, if I'm using YARN and I want to point my AM to the history server, I won't know what this parameter actually does until I grep for it in the code.
There was a problem hiding this comment.
Maybe I should make this a yarn specific config for now. I put it here thinking perhaps down the road it might be useful for spark core. thoughts? I'll fix the description on it.
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
@andrewor14 any additional comments? |
docs/running-on-yarn.md
Outdated
There was a problem hiding this comment.
ResourceManager* (last sentence)
|
One small typo, but other than that this LGTM |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14210/ |
|
thanks, I committed this to master and branch-1.0 |
…pp ... ...finishes Note this is dependent on #204 to have a working history server, but there are no code dependencies. This also fixes SPARK-1288 yarn stable finishApplicationMaster incomplete. Since I was in there I made the diagnostic message be passed properly. Author: Thomas Graves <[email protected]> Closes #362 from tgravescs/SPARK-1408 and squashes the following commits: ec89705 [Thomas Graves] Fix typo. 446122d [Thomas Graves] Make config yarn specific f5d5373 [Thomas Graves] SPARK-1408 Modify Spark on Yarn to point to the history server when app finishes (cherry picked from commit 0058b5d) Signed-off-by: Thomas Graves <[email protected]>
…pp ... ...finishes Note this is dependent on apache#204 to have a working history server, but there are no code dependencies. This also fixes SPARK-1288 yarn stable finishApplicationMaster incomplete. Since I was in there I made the diagnostic message be passed properly. Author: Thomas Graves <[email protected]> Closes apache#362 from tgravescs/SPARK-1408 and squashes the following commits: ec89705 [Thomas Graves] Fix typo. 446122d [Thomas Graves] Make config yarn specific f5d5373 [Thomas Graves] SPARK-1408 Modify Spark on Yarn to point to the history server when app finishes
|
Test Message |
Update README.md
4 participants
...finishes
Note this is dependent on #204 to have a working history server, but there are no code dependencies.
This also fixes SPARK-1288 yarn stable finishApplicationMaster incomplete. Since I was in there I made the diagnostic message be passed properly.