Skip to content

Conversation

@tibrewalpratik17
Copy link
Contributor

label:
API
feature

This patch accepts a minionInstanceTag string in /tasks/schedule API to allow users to schedule the tasks on that particular instance-tag nodes. This overwrites the values provided in the table configs.
We can use /tasks/execute too as that honors this tag in the config but using an API param seems more intuitive and uses pretty much all the configs of the already-defined table-tasks anyways.

Usecase:

  • If users want to schedule their backlog table tasks on adhoc nodes, this param will be useful.

cc @vvivekiyer

@codecov-commenter
Copy link

codecov-commenter commented Apr 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.64%. Comparing base (59551e4) to head (2b26373).
Report is 314 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12786       +/-   ##
=============================================
- Coverage     61.75%   46.64%   -15.11%     
- Complexity      207      953      +746     
=============================================
  Files          2436     1903      -533     
  Lines        133233    99964    -33269     
  Branches      20636    16126     -4510     
=============================================
- Hits          82274    46629    -35645     
- Misses        44911    49940     +5029     
+ Partials       6048     3395     -2653     
Flag Coverage Δ
custom-integration1 ?
integration ?
integration1 ?
integration2 ?
java-11 46.64% <ø> (-15.07%) ⬇️
java-21 ?
skip-bytebuffers-false 46.64% <ø> (-15.11%) ⬇️
skip-bytebuffers-true ?
temurin 46.64% <ø> (-15.11%) ⬇️
unittests 46.64% <ø> (-15.11%) ⬇️
unittests1 46.64% <ø> (-0.25%) ⬇️
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@vvivekiyer vvivekiyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Minor comments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add @Nullable annotation here and in other places.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not add minionInstanceTag to the public method above? It's only invoked from PinotTaskRestletResource and a bunch of tests.

IMO, there are already too many scheduleTask methods making this code difficult to read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I started with that but I see there were 35+ places in tests where we were calling that method so I ended up with adding a new function for simplicity. Lmk if we want to remove this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer that to make the code readable. Feel free to leave a todo if you want to pick it up later.

@siddharthteotia
Copy link
Contributor

@tibrewalpratik17 - can you please rebase ?

@tibrewalpratik17 tibrewalpratik17 force-pushed the overwrite_instance_tag_schedule_job branch from f2ea514 to 2b26373 Compare April 16, 2024 18:39
@tibrewalpratik17
Copy link
Contributor Author

@siddharthteotia @vvivekiyer rebased with master. Sorry for the delay as I was on PTO.

@vvivekiyer vvivekiyer merged commit d206f12 into apache:master Apr 17, 2024
@shounakmk219
Copy link
Collaborator

Hey @tibrewalpratik17 the changes that went into this PR may create unexpected damage as the behaviour for the method
PinotTaskManager#scheduleTask(String, String) has been changed from scheduleTask(taskType, tableNameWithType) to scheduleTask(taskType, minionInstanceTag).
This may break the systems that consume this method, for instance https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/CronJobScheduleJob.java#L68

@tibrewalpratik17
Copy link
Contributor Author

Hey @shounakmk219 thanks for pointing this out! Let me raise a fix shortly.

@tibrewalpratik17
Copy link
Contributor Author

@shounakmk219 please refer #12964

@Jackie-Jiang Jackie-Jiang added the incompatible Indicate PR that introduces backward incompatibility label Apr 18, 2024
@tibrewalpratik17 tibrewalpratik17 deleted the overwrite_instance_tag_schedule_job branch June 14, 2024 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature incompatible Indicate PR that introduces backward incompatibility rest-api

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants