Skip to content

use MinionEventObserver to track finer grained task progress status on worker #9432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
npawar merged 3 commits into from
Sep 21, 2022
Merged

use MinionEventObserver to track finer grained task progress status on worker #9432

npawar merged 3 commits into from
Sep 21, 2022

Conversation

@klsince
Copy link
Contributor

@klsince klsince commented Sep 19, 2022

This PR reuses MinionEventObserver interface to track and report fine-grained task progress status.

Today, helix task status like RUNNING is used to indicate task progress status, but that's very coarse grained. e.g. for some long running data ingestion task, one can see 'RUNNING' status lasts for many minutes, which provides little info for the users about what's really going on in the tasks.

With the new MinionEventObserver.notifyProgress() method, the task implementor can use it to report finer grained progress status, like downloading files (n out of N), generating, compressing, and uploading segments (n out of N) etc.

The full stack trace upon task failure is also tracked and kept around for a configured period to help debug the task failure.

Release note

  1. pinot.minion.event.observer.cleanupDelayInSec to tune when to clean up the observer object after the subtask ends.

@klsince
Copy link
Contributor Author

klsince commented Sep 19, 2022

for e.g.

curl -X GET "http://localhost:9000/tasks/subtask/Task_SegmentGenerationAndPushTask_fc3e3d56-d035-4fe8-a27b-80fd945dac04_1663628861391/progress" -H "accept: application/json" | jq .
{
  "Task_SegmentGenerationAndPushTask_fc3e3d56-d035-4fe8-a27b-80fd945dac04_1663628861391_1": "Task failed in 76ms, with error:\njava.lang.RuntimeException: Failed to execute SegmentGenerationAndPushTask\n\tat org.apache.pinot.plugin.minion.tasks.segmentg...",
  "Task_SegmentGenerationAndPushTask_fc3e3d56-d035-4fe8-a27b-80fd945dac04_1663628861391_0": "Task failed in 76ms, with error:\njava.lang.RuntimeException: Failed to execute SegmentGenerationAndPushTask\n\tat org.apache.pinot.plugin.minion.tasks.segmentgenerationandpush..."
}

@klsince klsince changed the title use MinionEventObserver to track finer grained task progress status on worker [WIP] use MinionEventObserver to track finer grained task progress status on worker Sep 19, 2022
@codecov-commenter
Copy link

codecov-commenter commented Sep 20, 2022 •
edited
Loading

Codecov Report

Merging #9432 (ccaf80f) into master (985d0b5) will decrease coverage by 0.06%.
The diff coverage is 42.93%.

@@             Coverage Diff              @@
##             master    #9432      +/-   ##
============================================
- Coverage     69.82%   69.76%   -0.07%     
+ Complexity     5095     4711     -384     
============================================
  Files          1889     1894       +5     
  Lines        100851   101041     +190     
  Branches      15328    15345      +17     
============================================
+ Hits          70421    70487      +66     
- Misses        25453    25575     +122     
- Partials       4977     4979       +2
Flag Coverage Δ
integration1 25.97% <30.36%> (+0.01%) ⬆️
integration2 24.69% <19.89%> (-0.06%) ⬇️
unittests1 67.08% <0.00%> (+<0.01%) ⬆️
unittests2 15.38% <29.84%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...va/org/apache/pinot/controller/ControllerConf.java 56.97% <0.00%> (-0.68%) ⬇️
...roller/api/resources/PinotTaskRestletResource.java 3.00% <0.00%> (-0.34%) ⬇️
...ntroller/helix/core/PinotHelixResourceManager.java 71.60% <0.00%> (-0.21%) ⬇️
...lix/core/minion/PinotHelixTaskResourceManager.java 32.80% <0.00%> (-5.08%) ⬇️
...inion/api/resources/PinotTaskProgressResource.java 0.00% <0.00%> (ø)
...apache/pinot/minion/event/MinionEventObserver.java 0.00% <0.00%> (ø)
...va/org/apache/pinot/spi/utils/CommonConstants.java 27.69% <ø> (ø)
...che/pinot/minion/event/MinionProgressObserver.java 40.90% <40.90%> (ø)
...pache/pinot/minion/event/MinionEventObservers.java 91.66% <91.66%> (ø)
.../org/apache/pinot/core/minion/PinotTaskConfig.java 54.54% <100.00%> (ø)
... and 25 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@klsince klsince force-pushed the track_fine_grained_task_progress_on_worker branch from 2f60c85 to ccaf80f Compare September 20, 2022 04:58
@klsince klsince changed the title [WIP] use MinionEventObserver to track finer grained task progress status on worker use MinionEventObserver to track finer grained task progress status on worker Sep 20, 2022
jackjlli
jackjlli reviewed Sep 20, 2022
View reviewed changes
zhtaoxiang
zhtaoxiang reviewed Sep 20, 2022
View reviewed changes
@npawar npawar merged commit 7af0d20 into apache:master Sep 21, 2022
@klsince klsince deleted the track_fine_grained_task_progress_on_worker branch September 21, 2022 23:47
@klsince klsince mentioned this pull request Sep 22, 2022
Merged
61yao pushed a commit to 61yao/pinot that referenced this pull request Oct 3, 2022
@klsince
use MinionEventObserver to track finer grained task progress status o…
796933a 
…n worker (apache#9432)

* use MinionEventObserver to track finer grained task progress status in memory

* UTs and refine

* cr and more UTs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jackjlli jackjlli jackjlli approved these changes

+1 more reviewer

@zhtaoxiang zhtaoxiang zhtaoxiang approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@klsince @codecov-commenter @zhtaoxiang @jackjlli @npawar