Repository navigation
compute: activate logging replay on data, not only per interval - #39179
Merged
Merged
Conversation
antiguru
changed the title
antiguru
marked this pull request as ready for review
September 27, 2026 18:59
The loggers only activated the replay operators when their capability advanced, and the replay's own periodic activation fires once per introspection interval. Each worker therefore handed its logging dataflow a full interval of timely, reachability, differential, and compute events in one go. The demux and batchers processed them in a burst that kept all workers busy for about 100 ms each second, and dataflows that need all workers to make progress, such as a subscribe, stalled for its duration. Count every published batch towards the replay's `RcActivator`, including batches published while flushing. Its threshold of 128 then wakes the replay once enough batches accumulated, so the logging dataflow works in bounded chunks. The activator is worker-local and never unparks a worker, so a parked worker still sleeps until its next wakeup. Co-Authored-By: Claude Opus 5.5 <[email protected]>
antiguru
force-pushed
the
logging-replay-activate
branch
from
September 29, 2026 16:02
1fecd0d to
94c6f88
Compare
This was referenced Oct 2, 2026
val-materialize
added a commit
that referenced
this pull request
Oct 9, 2026
Release notes for the v26.45 train, published as **v26.45.1**. v26.45.0 did not ship, so (as with v26.44) its notes are folded in here and [#39477](#39477) is closed unmerged. - **v26.45.0 content**: 8 Improvements, 2 Agent Skills, 8 Bug Fixes, copied verbatim from #39477 with anchors renamed to `{#v26.45.1-…}`. - **v26.45.1 content**: Alerting for Self-Managed and the external dependency monitoring Improvement (editor-added). The release's only PR, [#39659](#39659), is omitted. - **Dates**: Cloud 2026-10-08, Self-Managed 2026-10-09. The operator compatibility row is `v26.45.1`. `v26.45.md` (`patch: 1`, no `rc`) is already on `main`.⚠️ **Open questions** - **Group-sync bullet**: `oidc_group_role_sync_enabled` defaults to `false` at the tag. Is it on in Cloud? If not, cut the "Quieter logins" bullet. - **Certificate SAN requirements** ([#39316](#39316), [#39415](https://github.com/MaterializeInc/materialize/pull/39415)/[#39459](https://github.com/MaterializeInc/materialize/pull/39459)) need action at upgrade time but sit in Improvements. Should they be called out more loudly? Also, does [#39351](#39351) (S3 on rustls) impose the same SAN rule on custom S3 endpoints? - **[#39659](#39659) (omitted)**: is `enable_upsert_paged_spill` on in production? A draft bullet is in [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.45.1/final/omitted.md#borderline). - **`EXPLAIN ANALYZE ... CPU`**: [agent-skills#90](MaterializeInc/agent-skills#90) says it overstates joins, reductions, and TopK by 1.5x–3x before v26.46. Not published. Is that a known issue? - **Metric sinks**: [#39332](#39332), [#38910](#38910), and [#39337](#39337) are withheld because `enable_metric_sink` is `false`. #39337 fixes a panic any superuser can reach, so consider publishing it regardless. - **Possible cuts**: [#39362](#39362) was observed only in CI. The latency figures from [#39179](#39179) were measured on one noisy machine. - **Possible adds**: [#38894](https://github.com/MaterializeInc/materialize/pull/38894)–[#38896](https://github.com/MaterializeInc/materialize/pull/38896) (MV correction buffer) need an end-to-end number. [#39149](#39149) fixes the kind install guide, which crash-loops today. [#39549](#39549) (jemalloc page) is omitted because no released version had the regression. Snapshot drafts in mz-skills: [v26.45.0](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.0) (rc.1–rc.4, final), [v26.45.1](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.1/final). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The logging replay operators only woke up once per introspection interval, so each worker processed a whole second of timely, reachability, differential and compute events in one uninterruptible call. That burst kept all workers busy for about 100 ms each second and stalled dataflows that need every worker to make progress, such as subscribes. The burstiness hurt, not the volume: a 100 ms interval used the same total logging CPU and did not show the tail.
This counts every published log batch towards the replay's existing activation threshold of 128, so the logging dataflow drains its input in bounded chunks. The activator stays on the worker thread and never unparks it, so idle workers keep sleeping.
On a 4-worker replica with 1k subscribe timestamps/s, p99 latency of results waiting on the subscribe frontier dropped from 78-86 ms to 9-14 ms, close to running with logging disabled. The per-second worst index
SELECTlatency halved. This was measured on a noisy shared machine; #39178 adds metrics to confirm the effect elsewhere.Posted by Claude Code.
🤖 Generated with Claude Code