Problem: Unable to track status of table rebalance after it is initiated which leads to

• Not knowing whether a rebalance is taking time or is indefinitely stuck.
• Operational burden (productivity impact) to track status and debug.
  Field reported Issues - 1, 2

Solution: Provide stats that helps users track rebalance progress -

• Difference between initial and target states (what rebalance starts with)
• Difference between ideal state and external view (tells us pending work)
• Difference between ideal state and recomputed target state (if rebalance takes multiple iterations when ideal state changes, during the rebalance)

Stats are shown in the test section.
API for stats (rebalanceStatus/jobId)
A rebalance operation will return a rebalanceId which will be used to look at rebalanceStatus.
http://localhost:9000/rebalanceStatus/f83093b1-6b1e-47f3-8721-7984d442815d

Code changes:

• Introduced a uniqueId for rebalance operation (rebalanceJobId). This is a required parameter passed along in the rebalanceConfig.
• Introduced rebalanceStatus API to get rebalance status + stats, given the rebalanceJobId
• Introduced Observer/callback for TableRebalancer that encapsulates the logging of rebalance status + stats to ZooKeeper. rebalanceJobId is the jobId to lookup the Zookeeper state, used by rebalanceStatus API. rebalanceId is part of rebalanceResult that's returned to the user when a rebalance is initiated.
• The triggers to the observer in TableRebalancer to collect status + stats are the following
  1) At the start of rebalance when we have initial and target states. This will tell us amount of work that rebalance starts
  with.
  2) Rebalance phase that waits for external view to converge to ideal state. This will tell us the pending work for
  rebalance.
  3) Phase if/when ideal state changes due to other events and a new target is computed. This will tell us the work
  involved when ideal state changes multiple times during the rebalance.
• Created new znode path for rebalance jobType.
• Enhanced /table/tableName/jobs api to retrieve rebalance jobs as well. This is to discover rebalanceJobs, so as to get the jobId to look at status.
• Applied pinotStyle codeStyle to files which made formatting changes to unrelated areas of code as well.

Testing

• Unit test
• Integration test after a rebalance is initiated - initialToTargetStateConvergence tells us total work when rebalance was initiated, externalViewToIdealStateConvergence tells us pending work, currentToTargetConvergence tells us if ideal state changed multiple time since rebalance started and how far is target from ideal state.
  {
  "tableRebalanceProgressStats": {
  "status": "DONE",
  "completionStatusMsg": "Finished rebalancing table: billing_OFFLINE with minAvailableReplicas: 1, enableStrictReplicaGroup: false, bestEfforts: false in 32 ms.",
  "initialToTargetStateConvergence": {
  "_segmentsMissing": 0,
  "_segmentsToRebalance": 1,
  "_percentSegmentsToRebalance": 100,
  "_replicasToRebalance": 9
  },
  "externalViewToIdealStateConvergence": {
  "_segmentsMissing": 0,
  "_segmentsToRebalance": 0,
  "_percentSegmentsToRebalance": 0,
  "_replicasToRebalance": 0
  },
  "currentToTargetConvergence": {
  "_segmentsMissing": 0,
  "_segmentsToRebalance": 0,
  "_percentSegmentsToRebalance": 0,
  "_replicasToRebalance": 0
  },
  "startTimeInMilliseconds": 1678214633564,
  "timeToFinishInSeconds": 0,
  },
  "timeElapsedSinceStartInSeconds": 9
  }

Output of http://localhost:9000/table/airlineStats_OFFLINE/jobs?type=OFFLINE

{
"178b0d79-92d8-43c7-9aa0-0448da01fa94": {
"jobId": "178b0d79-92d8-43c7-9aa0-0448da01fa94",
"messageCount": "10",
"submissionTimeMs": "1678302700785",
"jobType": "RELOAD_ALL_SEGMENTS",
"tableName": "airlineStats_OFFLINE"
},
"095855b6-5b9b-48d5-ad97-c7a2db2d1b7b": {
"jobId": "095855b6-5b9b-48d5-ad97-c7a2db2d1b7b",
"submissionTimeMs": "1678302711030",
"REBALANCE_PROGRESS_STATS": "{"status":"DONE","timeToFinishInSeconds":0,"completionStatusMsg":"Finished rebalancing table: airlineStats_OFFLINE with minAvailableReplicas: 1, enableStrictReplicaGroup: false, bestEfforts: false in 41 ms.","startTimeInMilliseconds":1.678302711006E12,"initialToTargetStateConvergence":{"_segmentsMissing":0,"_segmentsToRebalance":31,"_percentSegmentsToRebalance":100.0,"_replicasToRebalance":279},"externalViewToIdealStateConvergence":{"_segmentsMissing":0,"_segmentsToRebalance":0,"_percentSegmentsToRebalance":0.0,"_replicasToRebalance":0},"currentToTargetConvergence":{"_segmentsMissing":0,"_segmentsToRebalance":0,"_percentSegmentsToRebalance":0.0,"_replicasToRebalance":0}}",
"tableName": "airlineStats_OFFLINE"
}
}