Skip to content

Conversation

@xiangfu0
Copy link
Contributor

@xiangfu0 xiangfu0 commented Aug 21, 2022

Adding endpoint to download local log files for each component for controller/broker/server/minion.
The purpose here is to simplify users debugging from the remote environment without access to containers.
E.g. users may have access only to controller LoadBalancer without the entire cluster, but want to debug a pinot server logger for why my real-time table is not ingesting, any issue with Kafka consumer or message decoder.

  1. Adding configs for logger root directory: pinot.controller.logger.root.dir, pinot.broker.logger.root.dir, pinot.server.logger.root.dir, pinot.minion.logger.root.dir
  2. Add new endpoints in all controller/broker/server/minion to fetch local logs:
  • /loggers/files: List all the log files under the logger root directory.
  • /loggers/download?filePath={filePath}: Download the log file based on the relative path.
  1. Add new endpoints in the controller to fetch remote logs:
  • /loggers/instances: Collect log files from all the instances in the cluster.
  • /loggers/instances/{instanceName}: List all the log files for a given instance.
  • /loggers/instances/{instanceName}/download?filePath={filePath}: Download the log file from a given instance on a given path.

@xiangfu0 xiangfu0 force-pushed the adding_logger_file_entrypoint branch 2 times, most recently from 18b2a51 to f1079d3 Compare August 21, 2022 11:35
@codecov-commenter
Copy link

codecov-commenter commented Aug 21, 2022

Codecov Report

Merging #9259 (ac19d48) into master (022fd37) will decrease coverage by 45.06%.
The diff coverage is 20.06%.

❗ Current head ac19d48 differs from pull request most recent head fa36a8d. Consider uploading reports for the commit fa36a8d to get more accurate results

@@              Coverage Diff              @@
##             master    #9259       +/-   ##
=============================================
- Coverage     69.93%   24.87%   -45.07%     
+ Complexity     5009       53     -4956     
=============================================
  Files          1860     1853        -7     
  Lines         99185    99234       +49     
  Branches      15082    15106       +24     
=============================================
- Hits          69367    24683    -44684     
- Misses        24905    72069    +47164     
+ Partials       4913     2482     -2431     
Flag Coverage Δ
integration1 ?
integration2 24.87% <20.06%> (+0.03%) ⬆️
unittests1 ?
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../pinot/broker/api/resources/PinotBrokerLogger.java 0.00% <0.00%> (ø)
...requesthandler/MultiStageBrokerRequestHandler.java 0.00% <0.00%> (-74.08%) ⬇️
...rg/apache/pinot/common/utils/LoggerFileServer.java 0.00% <0.00%> (ø)
...pache/pinot/common/utils/config/InstanceUtils.java 15.85% <0.00%> (-78.52%) ⬇️
...org/apache/pinot/common/utils/http/HttpClient.java 74.17% <ø> (-1.10%) ⬇️
...er/api/access/ZkBasicAuthAccessControlFactory.java 0.00% <ø> (ø)
...ache/pinot/controller/api/resources/Constants.java 21.05% <ø> (-21.06%) ⬇️
...sources/PinotAccessControlUserRestletResource.java 0.00% <ø> (-60.61%) ⬇️
...ler/api/resources/PinotControllerAuthResource.java 0.00% <ø> (ø)
...ontroller/api/resources/PinotControllerLogger.java 0.00% <0.00%> (ø)
... and 1411 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@xiangfu0 xiangfu0 force-pushed the adding_logger_file_entrypoint branch 8 times, most recently from 6fcf0cc to 1c213da Compare August 23, 2022 05:51
@xiangfu0 xiangfu0 force-pushed the adding_logger_file_entrypoint branch 2 times, most recently from 6cbc9cb to 083aa21 Compare August 23, 2022 06:42
@xiangfu0 xiangfu0 force-pushed the adding_logger_file_entrypoint branch 4 times, most recently from f7c608f to 9ba8d31 Compare August 24, 2022 00:21
@xiangfu0 xiangfu0 requested a review from KKcorps August 24, 2022 03:10
@Jackie-Jiang Jackie-Jiang added feature release-notes Referenced by PRs that need attention when compiling the next release notes labels Aug 25, 2022
@xiangfu0 xiangfu0 force-pushed the adding_logger_file_entrypoint branch 3 times, most recently from 020b701 to 8778666 Compare August 26, 2022 23:55
@xiangfu0 xiangfu0 force-pushed the adding_logger_file_entrypoint branch from 8778666 to fa36a8d Compare August 26, 2022 23:56
@xiangfu0 xiangfu0 merged commit 70ecda4 into apache:master Aug 27, 2022
@xiangfu0 xiangfu0 deleted the adding_logger_file_entrypoint branch August 27, 2022 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants