Description

Adds a metric task/waiting/time which measures the time it takes for a task to be placed onto the task runner for scheduling+running. This time encompasses initial persistent storage insertion, lock acquisition time, any task queue-related latencies, noisy-neighbor-related slowness, and any other delays between when a task is first added and when it is scheduled for running on the task runner. Useful for debugging issues related to task startup.

task/run/time ~= task/waiting/time + task/pending/time + <task run time> .

This codepath is called in a hot-loop but emission is non-blocking so this should be alright. Manual testing indicates no material increase in overhead.

Release note

Add task/waiting/time metric.

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.