Skip to content

Add segment level debug API #9609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jackie-Jiang merged 3 commits into from
Oct 20, 2022
Merged

Add segment level debug API #9609

Jackie-Jiang merged 3 commits into from
Oct 20, 2022

Conversation

@saurabhd336
Copy link
Contributor

@saurabhd336 saurabhd336 commented Oct 17, 2022 •
edited
Loading

This PR adds a segment level debug API that aims to make the debugging of segment state mismatches easier. The API will return the same information as the /debug/table/{tableName} API, but at the segment name level instead. A similar debug API has been added at the server level too. The controller API calls the server level APIs for the all the servers hosting the segment.

Sample responses:

curl -X GET "http://localhost:9000/debug/segments/transcript_REALTIME/transcript__0__0__20221017T0733Z" -H "accept: application/json"

{
  "segmentName": "transcript__0__0__20221017T0733Z",
  "serverState": {
    "Server_192.168.18.216_8098": {
      "idealState": "CONSUMING",
      "externalView": "CONSUMING",
      "segmentSize": "0 bytes",
      "consumerInfo": {
        "segmentName": "transcript__0__0__20221017T0733Z",
        "consumerState": "CONSUMING",
        "lastConsumedTimestamp": 1666249965431,
        "partitionToOffsetMap": {
          "0": "0"
        },
        "partitionOffsetInfo": null
      },
      "errorInfo": null
    }
  }
}

@saurabhd336
Copy link
Contributor Author

saurabhd336 commented Oct 17, 2022

@npawar @jayeshchoudhary

@agavra
Copy link
Contributor

agavra commented Oct 17, 2022

Thanks @saurabhd336! As a Pinot newbie trying to understand how things work better this will be really helpful, can you paste a sample output in the PR description?

@Jackie-Jiang Jackie-Jiang added feature release-notes Referenced by PRs that need attention when compiling the next release notes labels Oct 19, 2022
Jackie-Jiang
Jackie-Jiang reviewed Oct 19, 2022
View reviewed changes
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/DebugResource.java Outdated
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang Oct 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor)

Suggested change
@ApiParam(value = "Name of the table (with type)", required = true) @PathParam("tableName") String tableName,
@ApiParam(value = "Name of the table (with type)", required = true) @PathParam("tableName") String tableNameWithType,

Copy link
Contributor Author

@saurabhd336 saurabhd336 Oct 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

pinot-server/src/main/java/org/apache/pinot/server/api/resources/DebugResource.java Outdated
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang Oct 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for the table level one

Suggested change
@ApiParam(value = "Name of the table", required = true) @PathParam("tableName") String tableNameWithType,
@ApiParam(value = "Name of the table (with type)", required = true) @PathParam("tableName") String tableNameWithType,

Copy link
Contributor Author

@saurabhd336 saurabhd336 Oct 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

@saurabhd336
Copy link
Contributor Author

saurabhd336 commented Oct 20, 2022

@agavra Added a sample response!

@codecov-commenter
Copy link

codecov-commenter commented Oct 20, 2022 •
edited
Loading

Codecov Report

Merging #9609 (2f103db) into master (3e1aba3) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #9609      +/-   ##
============================================
- Coverage     69.92%   69.87%   -0.05%     
+ Complexity     4930     4853      -77     
============================================
  Files          1943     1943              
  Lines        103907   103960      +53     
  Branches      15745    15753       +8     
============================================
- Hits          72653    72640      -13     
- Misses        26153    26212      +59     
- Partials       5101     5108       +7
Flag Coverage Δ
integration1 25.79% <0.00%> (+<0.01%) ⬆️
integration2 24.45% <0.00%> (-0.11%) ⬇️
unittests1 67.30% <ø> (ø)
unittests2 15.68% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../pinot/controller/api/resources/DebugResource.java 0.00% <0.00%> (ø)
...ache/pinot/server/api/resources/DebugResource.java 0.00% <0.00%> (ø)
...ntroller/helix/core/minion/TaskMetricsEmitter.java 34.88% <0.00%> (-55.82%) ⬇️
...n/java/org/apache/pinot/common/utils/URIUtils.java 66.66% <0.00%> (-7.41%) ⬇️
...elix/core/periodictask/ControllerPeriodicTask.java 66.07% <0.00%> (-5.36%) ⬇️
...pache/pinot/core/query/optimizer/filter/Range.java 87.75% <0.00%> (-4.09%) ⬇️
...core/operator/docvalsets/TransformBlockValSet.java 59.43% <0.00%> (-2.84%) ⬇️
...lix/core/minion/PinotHelixTaskResourceManager.java 40.70% <0.00%> (-2.27%) ⬇️
...e/pinot/segment/local/io/util/PinotDataBitSet.java 95.62% <0.00%> (-1.46%) ⬇️
...ery/optimizer/filter/MergeEqInFilterOptimizer.java 92.59% <0.00%> (-1.24%) ⬇️
... and 22 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Jackie-Jiang Jackie-Jiang merged commit 7de438e into apache:master Oct 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jackie-Jiang Jackie-Jiang Jackie-Jiang approved these changes

Assignees

No one assigned

Labels

feature release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@saurabhd336 @agavra @codecov-commenter @Jackie-Jiang