Repository navigation
SS-488 - storage: read the MySQL replication stream during a snapshot - #40206
Conversation
4daba3a to
3f07889
Compare
QA LLM Review1. MEDIUM -- Avoid rescanning snapshot history for every CDC batch
Starting replication while the snapshot still holds its minimum capabilities introduces a growing CPU cost on busy, long-running MySQL snapshots. Each batch of recent CDC updates scans timestamp bindings accumulated since the snapshot began, slowing both replication and the snapshot on their shared timely worker. DetailsSnapshot and replication are concatenated before reclocking ( An isolated optimized probe of the current reclock implementation processing 1,000 separate one-row CDC batches took 143 ms with 3,600 retained bindings and 553 ms with 14,400, versus approximately 2 ms after advancing the source frontier. These represent one and four hours of advancing one-second bindings. Keep CDC reclocking independent of the snapshot frontier, or change binding lookup to avoid repeatedly scanning the pinned history. |
Confirmed, and this is an issue in reclock that affects the existing PG implementation too, so I'm going to create a different PR for it (tracking: SS-563). This isn't the first time it's come up (see key frames), but at the time a more surgical fix was implemented. |
3f07889 to
5800bfa
Compare
QA LLM Review1. MEDIUM -- Keep microsecond lookaheads consistent with persist grouping
With DetailsThe SQL duration parser accepts microseconds ( Without grouping, Check the original duration for zero before converting and applying the interval floor, so the snapshot and sink agree on whether grouping is enabled. |
5800bfa to
b370ca4
Compare
This is a discrepancy in how I implemented the check in the persist client, fixed in #40525 |
With `storage_persist_sink_description_lookahead` nonzero, the MySQL snapshot operator emits its rewind requests as soon as the snapshot GTID frontier is known, so the replication operator reads the binlog while tables are copied. Exports still advance together, so replication data is staged in the dataflow until the snapshot completes. With the lookahead at zero, rewinds are emitted after the copy, as before. The persist sink now groups a snapshotting MySQL export's updates into one batch, as it does for Postgres. Co-Authored-By: Claude <[email protected]>
b370ca4 to
18c7915
Compare
QA LLM Review1. MEDIUM -- Group CDC for resumed exports while another table snapshots
With a normal lookahead such as DetailsThe snapshot operator creates capabilities for every export ( Their CDC therefore enters Release resumed exports' snapshot capabilities before starting replication, or extend grouping to every export held back by the snapshot. |
This address in #40207, and the entire stack is getting merged together. |
…napshot (#40207) ### Motivation Every export's frontier stays at the minimum until the last table's snapshot read finishes. Adding one large table to a MySQL source therefore still stalls ingestion for every table already hydrated in that source. #39479 fixed the same problem for PostgreSQL. ### Description With `storage_persist_sink_description_lookahead` nonzero, each export's port holds the minimum capability only while that export still has snapshot data to emit or a pending rewind. - Exports that are not being snapshotted drop their snapshot capabilities up front. They advance with the binlog. - After a worker plans its reads, it drops the ports of exports it reads no part of. This also covers exports that failed schema verification. - A snapshotted export releases its snapshot capability on each worker as soon as that worker finishes reading its table. - The replication operator downgrades every port without a pending rewind at each yield point. With the lookahead at zero, all ports advance together. Behavior then matches #40206. The replication operator now reads the setting at render time as well. Both operators read it in the same dataflow build and agree with the persist sink. ### Verification The mzcompose default sets the lookahead, so the existing mzcompose-based MySQL source tests exercise per-export progress. #40208 adds targeted coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <[email protected]>
…40528) ### Motivation #40206 reads the MySQL replication stream while a snapshot runs. Nightly [18689](https://buildkite.com/materialize/nightly/builds/18689) is the first main nightly with it, and it flags two scenarios against v26.49.1. The wallclock delay is expected. CI sets `storage_persist_sink_description_lookahead` to 2s, and a MySQL snapshot now becomes visible up to that much later. ### Description This adds an ancestor override for the three MySqlInitialLoad scenarios, keyed on the #40206 squash commit, with v26.50.0 as the minimum baseline. MySqlInitialLoad passed in 18689 but takes the same delay, so it is included to keep it off the threshold. ### Verification Checked that the override resolves for the three scenarios and leaves the others unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <[email protected]>
Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. **The `final` snapshot has been applied: the dates below are the real ones** (Cloud 2026-11-12, Self-Managed 2026-11-13). The `final` snapshot adds no entries — `v26.50.0-rc.1...v26.50.0` is one commit, the version bump, and no agent-skills PR merged in the window. Its substantive work was the dates, a rebase onto `main`, and the three removals below. **Rebased onto `main`.** The branch was 107 commits behind and conflicted: `main`'s `## v26.49.1` section replaced the stale `## v26.49.0` stub the branch was cut from. Resolved by keeping `main`'s section verbatim and re-inserting `## v26.50.0` above it; same for the compatibility YAML (`v26.50`, then `v26.49.1`, then `v26.48.2`). **Three entries removed as already published on `main`.** The `v26.49.1` patch train cherry-picked four Console PRs that also merged to `main` inside this train's `rc.1` window, so both trains classified them independently and `main` already carries them under `## v26.49.1`: | PR(s) | Published on `main` in `## v26.49.1` as | Removed from `## v26.50.0` | |---|---|---| | [#40044](#40044), [#40074](#40074) | the feature section *"Right size your clusters, using the new Hydration tab on the console"*, with a screenshot | *"**Diagnose hydration from the cluster details page**"* | | [#40511](#40511) | *"**Freshness and cluster details pages on Self-Managed**"*, covering both pages | *"**The Freshness page on Self-Managed**"* | | [#40492](#40492) | the Bug Fix about the Console's clusters list showing a dash in the Disk column | *"**Swap usage in the Console's Disk column**"* | `main`'s wording is the later and fuller treatment in all three cases. That leaves `## v26.50.0` with **5 Improvements and 7 Bug Fixes**. Decisions required: - **MCP protocol default flip** ([#39762](#39762)): published as an Improvement because `enable_mcp_protocol_2026_07_28` is `true` at the tag and the flip *is* the change. The PR was held in draft "until the 2026-07-28 code ships in a release and passes a test on a deployed environment" — is it confirmed on? - **Columnar accumulators** ([#40364](#40364)): omitted as Borderline — the default is `true` but the PR states no measured user-facing effect. Is there a memory or CPU figure worth publishing? - **MySQL snapshot/replication concurrency** ([#40206](#40206), [#40207](#40207)): omitted as Borderline — gated on `storage_persist_sink_description_lookahead`, which is `Duration::ZERO` at the tag, same as the PostgreSQL counterparts held back in v26.47.0. If the flag is served on, both halves want announcing together. - **MCP servers GA** ([#40448](#40448)): omitted as Borderline — the preview banners are gone, but the GA call belongs to the release owner. Publish it? - **Balancer connection metrics** ([#40192](#40192)): omitted as Borderline — `mz_balancer_connection_status{status="error"}` changes meaning, so Self-Managed dashboards and alerts on that series will see a step change. Worth a note? - **Console query comments** ([#40075](#40075)): omitted as Borderline — every Console statement's `sql_hash` changes once when this deploys. Worth calling out? Every flag above was re-read in source at the `v26.50.0` ship tag; not one default moved between `rc.1` and ship, so all six holds still read correctly. Snapshot drafts (with PR links) in [mz-skills](https://github.com/MaterializeInc/mz-skills): - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 7 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md#borderline), final: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md#borderline) (no PRs in scope; `rc.1`'s six holds re-examined at the ship tag and unchanged) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>
Motivation
Today a MySQL source snapshots all of its tables before it reads any of the binlog. This PR lets the replication operator read the binlog while the snapshot runs, as #39419 did for PostgreSQL. #40207 then lets each export's frontier advance on its own.
NOTE: this needs an exception added for the 2s increase in snapshot performance, but I can't add it until I have a merge hash from main. Will do in a follow up PR.
Closes: SS-488
Description
All new behavior is keyed off
storage_persist_sink_description_lookaheadbeing nonzero. This is the same switch PostgreSQL uses.With a nonzero lookahead, the snapshot operator emits its rewind requests as soon as the snapshot GTID frontier is known. That happens before any table is read. The replication operator then reads the binlog while the tables are read. Every export still advances together. Replication data is staged in the dataflow until the snapshot completes.
With the lookahead at zero, rewinds are emitted after the snapshot completes, as before.
The snapshot operator reads the setting once at render time. That matches the value the persist sink reads in the same dataflow build.
MySQL sources now count as OLTP sources in
render.rs. A snapshotting MySQL export therefore gets the persist sink's snapshot ceiling and lookahead grouping, like a PostgreSQL export. The mzcompose default sets the lookahead to 2s. The MySQL initial load feature benchmarks should see their first rows about that much later, as the PgCdc benchmarks did after #39602.#39419 also added a leader check that the replication slot exists before the snapshot starts. MySQL needs no counterpart. The replication operator already checks
@@gtid_purgedagainst the resume upper before it waits for rewind requests.Verification
The mzcompose default sets the lookahead, so the existing mzcompose-based MySQL source tests exercise the early rewind path. #40208 adds targeted coverage.
🤖 Generated with Claude Code