Skip to content

Conversation

@zhtaoxiang
Copy link
Contributor

@zhtaoxiang zhtaoxiang commented Jun 21, 2022

What Does This PR Do

Add /tasks/{taskType}/{tableNameWithType}/debug API so that we can get a task debug info for the given table. This makes debugging a specific table's problem easier.

Test

Test by setting up a Pinot cluster locally.

@codecov-commenter
Copy link

codecov-commenter commented Jun 21, 2022

Codecov Report

Merging #8949 (af5e437) into master (2644f48) will decrease coverage by 3.42%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #8949      +/-   ##
============================================
- Coverage     69.69%   66.26%   -3.43%     
+ Complexity     4715     4695      -20     
============================================
  Files          1813     1361     -452     
  Lines         94503    68700   -25803     
  Branches      14117    10746    -3371     
============================================
- Hits          65860    45523   -20337     
+ Misses        24039    19899    -4140     
+ Partials       4604     3278    -1326     
Flag Coverage Δ
integration1 ?
integration2 ?
unittests1 66.26% <ø> (-0.04%) ⬇️
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...va/org/apache/pinot/core/routing/RoutingTable.java 0.00% <0.00%> (-100.00%) ⬇️
...va/org/apache/pinot/common/config/NettyConfig.java 0.00% <0.00%> (-100.00%) ⬇️
...a/org/apache/pinot/common/metrics/MinionMeter.java 0.00% <0.00%> (-100.00%) ⬇️
...g/apache/pinot/common/metrics/ControllerMeter.java 0.00% <0.00%> (-100.00%) ⬇️
.../apache/pinot/common/metrics/BrokerQueryPhase.java 0.00% <0.00%> (-100.00%) ⬇️
.../apache/pinot/common/metrics/MinionQueryPhase.java 0.00% <0.00%> (-100.00%) ⬇️
...apache/pinot/common/helix/ExtraInstanceConfig.java 0.00% <0.00%> (-100.00%) ⬇️
...ache/pinot/server/access/AccessControlFactory.java 0.00% <0.00%> (-100.00%) ⬇️
...he/pinot/common/messages/SegmentReloadMessage.java 0.00% <0.00%> (-100.00%) ⬇️
...he/pinot/common/messages/TableDeletionMessage.java 0.00% <0.00%> (-100.00%) ⬇️
... and 696 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2644f48...af5e437. Read the comment docs.

String tableNameWithType,
@ApiParam(value = "verbosity (By default, prints for running and error tasks. Value of >0 prints for all tasks)")
@DefaultValue("0") @QueryParam("verbosity") int verbosity) {
return _pinotHelixTaskResourceManager.getTasksDebugInfoByTable(taskType, tableNameWithType, verbosity);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this out with RealtimeQuickstartWithMinion, and saw it return 3 COMPLETED tasks, for both 0 and 1 verbosity. Only difference was level of detail. Is this By default, prints for running and error tasks. Value of >0 prints for all tasks) incorrect or does the impl need some change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simply copied and pasted the existing docs 😄 . fixed this one and existing ones.

String tableNameConfig = pinotConfigs.get(TABLE_NAME);
if (tableNameConfig != null && tableNameConfig.equals(tableNameWithType)) {
// Found a match. Add task debug info to the result
// TODO: we may want to filter out debug info that does not belong to the given table.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't follow this TODO. why would we get info from getTaskDebugInfo that's not for this table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated the TODO.

@npawar
Copy link
Contributor

npawar commented Jun 21, 2022

What Does This PR Do

Add /tasks/{taskType}/{tableNameWithType}/debug API so that we can get a task debug info for the given table. This makes debugging a specific table's problem easier.

Test

Test by setting up a Pinot cluster locally.

Mind adding a sample response here?

@npawar npawar added enhancement release-notes Referenced by PRs that need attention when compiling the next release notes labels Jun 21, 2022
@zhtaoxiang
Copy link
Contributor Author

zhtaoxiang commented Jun 22, 2022

Sample response when verbosity = 0

curl -X GET "http://localhost:18998/tasks/FileIngestionTask/fileIngestionTaskTestTable_AdhocExecute_OFFLINE/debug?verbosity=0" -H "accept: application/json"

{
  "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_1a36671a-c451-41b1-94a0-86cfdf6dd67e": {
    "taskState": "COMPLETED",
    "subtaskCount": {
      "total": 2,
      "completed": 2,
      "running": 0,
      "waiting": 0,
      "error": 0,
      "unknown": 0
    },
    "startTime": "2022-06-21 19:54:35 PDT",
    "executionStartTime": "2022-06-21 19:54:35 PDT"
  },
  "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_2c9e0820-1a0b-4bec-a17d-1d06403727f5": {
    "taskState": "COMPLETED",
    "subtaskCount": {
      "total": 2,
      "completed": 2,
      "running": 0,
      "waiting": 0,
      "error": 0,
      "unknown": 0
    },
    "startTime": "2022-06-21 19:55:03 PDT",
    "executionStartTime": "2022-06-21 19:55:03 PDT"
  },
  "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_b7294c35-45aa-4450-ae5a-82eee4a4ef49": {
    "taskState": "COMPLETED",
    "subtaskCount": {
      "total": 2,
      "completed": 2,
      "running": 0,
      "waiting": 0,
      "error": 0,
      "unknown": 0
    },
    "startTime": "2022-06-21 19:54:50 PDT",
    "executionStartTime": "2022-06-21 19:54:50 PDT"
  }
}

@zhtaoxiang
Copy link
Contributor Author

Sample response when verbosity = 1

curl -X GET "http://localhost:18998/tasks/FileIngestionTask/fileIngestionTaskTestTable_AdhocExecute_OFFLINE/debug?verbosity=1" -H "accept: application/json"

{
  "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_1a36671a-c451-41b1-94a0-86cfdf6dd67e": {
    "taskState": "COMPLETED",
    "subtaskCount": {
      "total": 2,
      "completed": 2,
      "running": 0,
      "waiting": 0,
      "error": 0,
      "unknown": 0
    },
    "startTime": "2022-06-21 19:54:35 PDT",
    "executionStartTime": "2022-06-21 19:54:35 PDT",
    "subtaskInfos": [
      {
        "taskId": "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_1a36671a-c451-41b1-94a0-86cfdf6dd67e_1",
        "state": "COMPLETED",
        "startTime": "2022-06-21 19:54:35 PDT",
        "finishTime": "2022-06-21 19:54:39 PDT",
        "participant": "Minion_127.0.0.1_9514",
        "info": "Succeeded",
        "taskConfig": {
          "taskType": "FileIngestionTask",
          "configs": {
            "recordReader.configClassName": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig",
            "maxNumRecordsPerSegment": "1000",
            "inputDirURI": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/",
            "segmentNamePrefix": "fileIngestionTaskTestTable_AdhocExecute",
            "tableMaxNumTasks": "2",
            "inputFormat": "csv",
            "sequenceId": "FileIngestionTask_1655866466031_1",
            "segmentNamePostfix": "FileIngestionTask_1655866466031_1",
            "tableName": "fileIngestionTaskTestTable_AdhocExecute_OFFLINE",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReader",
            "checkpointTTLSec": "0",
            "input.data.file.uri": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00003.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00004.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00005.csv",
            "taskMaxNumFiles": "3",
            "uploadURL": "http://localhost:18998/segments",
            "push.controllerUri": "http://localhost:18998"
          }
        }
      },
      {
        "taskId": "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_1a36671a-c451-41b1-94a0-86cfdf6dd67e_0",
        "state": "COMPLETED",
        "startTime": "2022-06-21 19:54:35 PDT",
        "finishTime": "2022-06-21 19:54:38 PDT",
        "participant": "Minion_127.0.0.1_9514",
        "info": "Succeeded",
        "taskConfig": {
          "taskType": "FileIngestionTask",
          "configs": {
            "recordReader.configClassName": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig",
            "maxNumRecordsPerSegment": "1000",
            "inputDirURI": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/",
            "segmentNamePrefix": "fileIngestionTaskTestTable_AdhocExecute",
            "tableMaxNumTasks": "2",
            "inputFormat": "csv",
            "sequenceId": "FileIngestionTask_1655866466031_0",
            "segmentNamePostfix": "FileIngestionTask_1655866466031_0",
            "tableName": "fileIngestionTaskTestTable_AdhocExecute_OFFLINE",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReader",
            "checkpointTTLSec": "0",
            "input.data.file.uri": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00000.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00001.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00002.csv",
            "taskMaxNumFiles": "3",
            "uploadURL": "http://localhost:18998/segments",
            "push.controllerUri": "http://localhost:18998"
          }
        }
      }
    ]
  },
  "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_2c9e0820-1a0b-4bec-a17d-1d06403727f5": {
    "taskState": "COMPLETED",
    "subtaskCount": {
      "total": 2,
      "completed": 2,
      "running": 0,
      "waiting": 0,
      "error": 0,
      "unknown": 0
    },
    "startTime": "2022-06-21 19:55:03 PDT",
    "executionStartTime": "2022-06-21 19:55:03 PDT",
    "subtaskInfos": [
      {
        "taskId": "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_2c9e0820-1a0b-4bec-a17d-1d06403727f5_1",
        "state": "COMPLETED",
        "startTime": "2022-06-21 19:55:03 PDT",
        "finishTime": "2022-06-21 19:55:03 PDT",
        "participant": "Minion_127.0.0.1_9514",
        "info": "Succeeded",
        "taskConfig": {
          "taskType": "FileIngestionTask",
          "configs": {
            "recordReader.configClassName": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig",
            "maxNumRecordsPerSegment": "1000",
            "inputDirURI": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/",
            "segmentNamePrefix": "fileIngestionTaskTestTable_AdhocExecute",
            "tableMaxNumTasks": "2",
            "inputFormat": "csv",
            "sequenceId": "FileIngestionTask_1655866492891_1",
            "segmentNamePostfix": "FileIngestionTask_1655866492891_1",
            "tableName": "fileIngestionTaskTestTable_AdhocExecute_OFFLINE",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReader",
            "checkpointTTLSec": "0",
            "input.data.file.uri": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_Dir1/emptyFile_00002.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_Dir1/emptyFile_00003.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_Dir1/emptyFile_00004.csv",
            "taskMaxNumFiles": "3",
            "uploadURL": "http://localhost:18998/segments",
            "push.controllerUri": "http://localhost:18998"
          }
        }
      },
      {
        "taskId": "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_2c9e0820-1a0b-4bec-a17d-1d06403727f5_0",
        "state": "COMPLETED",
        "startTime": "2022-06-21 19:55:03 PDT",
        "finishTime": "2022-06-21 19:55:04 PDT",
        "participant": "Minion_127.0.0.1_9514",
        "info": "Succeeded",
        "taskConfig": {
          "taskType": "FileIngestionTask",
          "configs": {
            "recordReader.configClassName": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig",
            "maxNumRecordsPerSegment": "1000",
            "inputDirURI": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/",
            "segmentNamePrefix": "fileIngestionTaskTestTable_AdhocExecute",
            "tableMaxNumTasks": "2",
            "inputFormat": "csv",
            "sequenceId": "FileIngestionTask_1655866492891_0",
            "segmentNamePostfix": "FileIngestionTask_1655866492891_0",
            "tableName": "fileIngestionTaskTestTable_AdhocExecute_OFFLINE",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReader",
            "checkpointTTLSec": "0",
            "input.data.file.uri": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00012.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_Dir1/emptyFile_00000.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_Dir1/emptyFile_00001.csv",
            "taskMaxNumFiles": "3",
            "uploadURL": "http://localhost:18998/segments",
            "push.controllerUri": "http://localhost:18998"
          }
        }
      }
    ]
  },
  "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_b7294c35-45aa-4450-ae5a-82eee4a4ef49": {
    "taskState": "COMPLETED",
    "subtaskCount": {
      "total": 2,
      "completed": 2,
      "running": 0,
      "waiting": 0,
      "error": 0,
      "unknown": 0
    },
    "startTime": "2022-06-21 19:54:50 PDT",
    "executionStartTime": "2022-06-21 19:54:50 PDT",
    "subtaskInfos": [
      {
        "taskId": "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_b7294c35-45aa-4450-ae5a-82eee4a4ef49_0",
        "state": "COMPLETED",
        "startTime": "2022-06-21 19:54:50 PDT",
        "finishTime": "2022-06-21 19:54:51 PDT",
        "participant": "Minion_127.0.0.1_9514",
        "info": "Succeeded",
        "taskConfig": {
          "taskType": "FileIngestionTask",
          "configs": {
            "recordReader.configClassName": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig",
            "maxNumRecordsPerSegment": "1000",
            "inputDirURI": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/",
            "segmentNamePrefix": "fileIngestionTaskTestTable_AdhocExecute",
            "tableMaxNumTasks": "2",
            "inputFormat": "csv",
            "sequenceId": "FileIngestionTask_1655866480047_0",
            "segmentNamePostfix": "FileIngestionTask_1655866480047_0",
            "tableName": "fileIngestionTaskTestTable_AdhocExecute_OFFLINE",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReader",
            "checkpointTTLSec": "0",
            "input.data.file.uri": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00006.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00007.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00008.csv",
            "taskMaxNumFiles": "3",
            "uploadURL": "http://localhost:18998/segments",
            "push.controllerUri": "http://localhost:18998"
          }
        }
      },
      {
        "taskId": "Task_FileIngestionTask_fileIngestionTaskTestTable_AdhocExecute_OFFLINE_b7294c35-45aa-4450-ae5a-82eee4a4ef49_1",
        "state": "COMPLETED",
        "startTime": "2022-06-21 19:54:50 PDT",
        "finishTime": "2022-06-21 19:54:52 PDT",
        "participant": "Minion_127.0.0.1_9514",
        "info": "Succeeded",
        "taskConfig": {
          "taskType": "FileIngestionTask",
          "configs": {
            "recordReader.configClassName": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig",
            "maxNumRecordsPerSegment": "1000",
            "inputDirURI": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/",
            "segmentNamePrefix": "fileIngestionTaskTestTable_AdhocExecute",
            "tableMaxNumTasks": "2",
            "inputFormat": "csv",
            "sequenceId": "FileIngestionTask_1655866480047_1",
            "segmentNamePostfix": "FileIngestionTask_1655866480047_1",
            "tableName": "fileIngestionTaskTestTable_AdhocExecute_OFFLINE",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.csv.CSVRecordReader",
            "checkpointTTLSec": "0",
            "input.data.file.uri": "file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00009.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00010.csv,file:/var/folders/kd/4c8chyjn6kdcy7bmv69x6bc40000gn/T/FileIngestionTaskIntegrationTest/fileIngestionTaskTestTable_AdhocExecute/tempFile_00011.csv",
            "taskMaxNumFiles": "3",
            "uploadURL": "http://localhost:18998/segments",
            "push.controllerUri": "http://localhost:18998"
          }
        }
      }
    ]
  }
}

@npawar npawar merged commit 5f86b5e into apache:master Jun 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants