Repository navigation
SS-489 - storage: let MySQL exports progress independently during a snapshot - #40207
Conversation
patrickwwbutler
left a comment
There was a problem hiding this comment.
Overall looks good, with a question about capability handling mostly for my own knowledge
| trace!(%id, "timely-{worker_id} snapshotted {} records from \ | ||
| table '{table}'", snapshot_staged * u64::cast_from(outputs.len())); | ||
| // A transient error above returns with the capabilities still held. That | ||
| // keeps a partially copied export from advancing. |
There was a problem hiding this comment.
I'm a bit confused by this comment, though I think it's just a reviewer error? - It seems accurate, and the behavior it describes seems correct to me (if a snapshot has a transient error, the export shouldn't advance as the snapshot isn't complete), but the code following it seems unrelated, sort of. In the case of a transient error, the operator would return without dropping capabilities, and this code wouldn't run, or change anything about that, right?
What's the point of dropping caps here? How would the caps have been downgraded before this?
There was a problem hiding this comment.
Correct. I can reword, this is mainly referring to the ? in while let Some(row) = results.try_next().await? - which would return early with Err.
Dropping caps for the concurrent case allows the frontier of the exports to be driven by the replication operator (this is within the loop (table, outputs) in &reader_snapshot_table_info).
The non-concurrent behavior is that all exports must be done snapshotting before replication can start. The capabilities for all ports are dropped when this operator returns with Ok ln:1177. Returning an error does not have the same effect (that's why we're safe above):
materialize/src/timely-util/src/builder_async.rs
Lines 705 to 711 in 616f9c3
78e0250 to
f7117e7
Compare
…napshot With `storage_persist_sink_description_lookahead` nonzero, each MySQL export's port holds the minimum capability only while that export still has snapshot data to emit or a pending rewind. Exports that are not being snapshotted advance with the replication stream, and a snapshotted export advances once its own copy finishes and its rewind resolves. With the lookahead at zero, all ports still advance together. Co-Authored-By: Claude <[email protected]>
f7117e7 to
c95f468
Compare
…#40206) ### Motivation Today a MySQL source snapshots all of its tables before it reads any of the binlog. This PR lets the replication operator read the binlog while the snapshot runs, as #39419 did for PostgreSQL. #40207 then lets each export's frontier advance on its own. NOTE: this needs an exception added for the 2s increase in snapshot performance, but I can't add it until I have a merge hash from main. Will do in a follow up PR. Closes: [SS-488](https://linear.app/materializeinc/issue/SS-488) ### Description All new behavior is keyed off `storage_persist_sink_description_lookahead` being nonzero. This is the same switch PostgreSQL uses. With a nonzero lookahead, the snapshot operator emits its rewind requests as soon as the snapshot GTID frontier is known. That happens before any table is read. The replication operator then reads the binlog while the tables are read. Every export still advances together. Replication data is staged in the dataflow until the snapshot completes. With the lookahead at zero, rewinds are emitted after the snapshot completes, as before. The snapshot operator reads the setting once at render time. That matches the value the persist sink reads in the same dataflow build. MySQL sources now count as OLTP sources in `render.rs`. A snapshotting MySQL export therefore gets the persist sink's snapshot ceiling and lookahead grouping, like a PostgreSQL export. The mzcompose default sets the lookahead to 2s. The MySQL initial load feature benchmarks should see their first rows about that much later, as the PgCdc benchmarks did after #39602. #39419 also added a leader check that the replication slot exists before the snapshot starts. MySQL needs no counterpart. The replication operator already checks `@@gtid_purged` against the resume upper before it waits for rewind requests. ### Verification The mzcompose default sets the lookahead, so the existing mzcompose-based MySQL source tests exercise the early rewind path. #40208 adds targeted coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]>
Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. **The `final` snapshot has been applied: the dates below are the real ones** (Cloud 2026-11-12, Self-Managed 2026-11-13). The `final` snapshot adds no entries — `v26.50.0-rc.1...v26.50.0` is one commit, the version bump, and no agent-skills PR merged in the window. Its substantive work was the dates, a rebase onto `main`, and the three removals below. **Rebased onto `main`.** The branch was 107 commits behind and conflicted: `main`'s `## v26.49.1` section replaced the stale `## v26.49.0` stub the branch was cut from. Resolved by keeping `main`'s section verbatim and re-inserting `## v26.50.0` above it; same for the compatibility YAML (`v26.50`, then `v26.49.1`, then `v26.48.2`). **Three entries removed as already published on `main`.** The `v26.49.1` patch train cherry-picked four Console PRs that also merged to `main` inside this train's `rc.1` window, so both trains classified them independently and `main` already carries them under `## v26.49.1`: | PR(s) | Published on `main` in `## v26.49.1` as | Removed from `## v26.50.0` | |---|---|---| | [#40044](#40044), [#40074](#40074) | the feature section *"Right size your clusters, using the new Hydration tab on the console"*, with a screenshot | *"**Diagnose hydration from the cluster details page**"* | | [#40511](#40511) | *"**Freshness and cluster details pages on Self-Managed**"*, covering both pages | *"**The Freshness page on Self-Managed**"* | | [#40492](#40492) | the Bug Fix about the Console's clusters list showing a dash in the Disk column | *"**Swap usage in the Console's Disk column**"* | `main`'s wording is the later and fuller treatment in all three cases. That leaves `## v26.50.0` with **5 Improvements and 7 Bug Fixes**. Decisions required: - **MCP protocol default flip** ([#39762](#39762)): published as an Improvement because `enable_mcp_protocol_2026_07_28` is `true` at the tag and the flip *is* the change. The PR was held in draft "until the 2026-07-28 code ships in a release and passes a test on a deployed environment" — is it confirmed on? - **Columnar accumulators** ([#40364](#40364)): omitted as Borderline — the default is `true` but the PR states no measured user-facing effect. Is there a memory or CPU figure worth publishing? - **MySQL snapshot/replication concurrency** ([#40206](#40206), [#40207](#40207)): omitted as Borderline — gated on `storage_persist_sink_description_lookahead`, which is `Duration::ZERO` at the tag, same as the PostgreSQL counterparts held back in v26.47.0. If the flag is served on, both halves want announcing together. - **MCP servers GA** ([#40448](#40448)): omitted as Borderline — the preview banners are gone, but the GA call belongs to the release owner. Publish it? - **Balancer connection metrics** ([#40192](#40192)): omitted as Borderline — `mz_balancer_connection_status{status="error"}` changes meaning, so Self-Managed dashboards and alerts on that series will see a step change. Worth a note? - **Console query comments** ([#40075](#40075)): omitted as Borderline — every Console statement's `sql_hash` changes once when this deploys. Worth calling out? Every flag above was re-read in source at the `v26.50.0` ship tag; not one default moved between `rc.1` and ship, so all six holds still read correctly. Snapshot drafts (with PR links) in [mz-skills](https://github.com/MaterializeInc/mz-skills): - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 7 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md#borderline), final: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md#borderline) (no PRs in scope; `rc.1`'s six holds re-examined at the ship tag and unchanged) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>
Motivation
Every export's frontier stays at the minimum until the last table's snapshot read finishes. Adding one large table to a MySQL source therefore still stalls ingestion for every table already hydrated in that source. #39479 fixed the same problem for PostgreSQL.
Description
With
storage_persist_sink_description_lookaheadnonzero, each export's port holds the minimum capability only while that export still has snapshot data to emit or a pending rewind.With the lookahead at zero, all ports advance together. Behavior then matches #40206.
The replication operator now reads the setting at render time as well. Both operators read it in the same dataflow build and agree with the persist sink.
Verification
The mzcompose default sets the lookahead, so the existing mzcompose-based MySQL source tests exercise per-export progress. #40208 adds targeted coverage.
🤖 Generated with Claude Code