Repository navigation
compute: instrument logging dataflow volume and step duration - #39178
Merged
Merged
Conversation
antiguru
changed the title
antiguru
marked this pull request as ready for review
September 27, 2026 18:58
Add two per-worker metrics for the logging dataflow. The counter `mz_compute_logging_records_total` counts the records each logger hands to the logging dataflow, labeled by log. The histogram `mz_compute_logging_step_duration_seconds` observes how long each scheduling of the logging dataflow takes as a whole. The worker logs the start and stop of every dataflow scheduling to the logger passed at dataflow creation. The logging dataflow now receives a dedicated logger that turns its own schedule events into histogram observations. With `log_logging` set, it keeps the worker's timely logger, and the histogram stays empty because `mz_scheduling_elapsed` covers the dataflow. Co-Authored-By: Claude Opus 5.5 <[email protected]>
antiguru
force-pushed
the
logging-metrics
branch
from
September 29, 2026 16:02
fcc4c6b to
853952d
Compare
antiguru
added a commit
that referenced
this pull request
Sep 29, 2026
The logging replay operators only woke up once per introspection interval, so each worker processed a whole second of timely, reachability, differential and compute events in one uninterruptible call. That burst kept all workers busy for about 100 ms each second and stalled dataflows that need every worker to make progress, such as subscribes. The burstiness hurt, not the volume: a 100 ms interval used the same total logging CPU and did not show the tail. This counts every published log batch towards the replay's existing activation threshold of 128, so the logging dataflow drains its input in bounded chunks. The activator stays on the worker thread and never unparks it, so idle workers keep sleeping. On a 4-worker replica with 1k subscribe timestamps/s, p99 latency of results waiting on the subscribe frontier dropped from 78-86 ms to 9-14 ms, close to running with logging disabled. The per-second worst index `SELECT` latency halved. This was measured on a noisy shared machine; #39178 adds metrics to confirm the effect elsewhere. Posted by Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two per-worker metrics for the compute logging dataflow, to make its volume and burstiness visible.
mz_compute_logging_records_total{log}counts the records each logger (timely, reachability, differential, compute) hands to the logging dataflow.mz_compute_logging_step_duration_secondsobserves how long each scheduling of the logging dataflow takes as a whole.The step histogram comes from the schedule events timely logs for the dataflow root, which the logging dataflow now receives on a dedicated logger. With
log_loggingset the dataflow keeps the worker's timely logger and the histogram stays empty, becausemz_scheduling_elapsedalready covers it in that mode.Posted by Claude Code.
🤖 Generated with Claude Code