Skip to content

SS-489 - storage: let MySQL exports progress independently during a snapshot - #40207

Merged
martykulma merged 1 commit into
maz-mysql-concurrent-replicationfrom
maz-mysql-per-export-progress
Nov 5, 2026
Merged

martykulma merged 1 commit into
maz-mysql-concurrent-replicationfrom
maz-mysql-per-export-progress

Conversation

@martykulma

@martykulma martykulma commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Every export's frontier stays at the minimum until the last table's snapshot read finishes. Adding one large table to a MySQL source therefore still stalls ingestion for every table already hydrated in that source. #39479 fixed the same problem for PostgreSQL.

Description

With storage_persist_sink_description_lookahead nonzero, each export's port holds the minimum capability only while that export still has snapshot data to emit or a pending rewind.

  • Exports that are not being snapshotted drop their snapshot capabilities up front. They advance with the binlog.
  • After a worker plans its reads, it drops the ports of exports it reads no part of. This also covers exports that failed schema verification.
  • A snapshotted export releases its snapshot capability on each worker as soon as that worker finishes reading its table.
  • The replication operator downgrades every port without a pending rewind at each yield point.

With the lookahead at zero, all ports advance together. Behavior then matches #40206.

The replication operator now reads the setting at render time as well. Both operators read it in the same dataflow build and agree with the persist sink.

Verification

The mzcompose default sets the lookahead, so the existing mzcompose-based MySQL source tests exercise per-export progress. #40208 adds targeted coverage.

🤖 Generated with Claude Code

@martykulma
martykulma added this pull request to stack #40209 October 27, 2026 01:55
@linear-code

linear-code Bot commented Oct 27, 2026

Copy link
Copy Markdown

SS-489

@patrickwwbutler patrickwwbutler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, with a question about capability handling mostly for my own knowledge

trace!(%id, "timely-{worker_id} snapshotted {} records from \
table '{table}'", snapshot_staged * u64::cast_from(outputs.len()));
// A transient error above returns with the capabilities still held. That
// keeps a partially copied export from advancing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused by this comment, though I think it's just a reviewer error? - It seems accurate, and the behavior it describes seems correct to me (if a snapshot has a transient error, the export shouldn't advance as the snapshot isn't complete), but the code following it seems unrelated, sort of. In the case of a transient error, the operator would return without dropping capabilities, and this code wouldn't run, or change anything about that, right?

What's the point of dropping caps here? How would the caps have been downgraded before this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. I can reword, this is mainly referring to the ? in while let Some(row) = results.try_next().await? - which would return early with Err.

Dropping caps for the concurrent case allows the frontier of the exports to be driven by the replication operator (this is within the loop (table, outputs) in &reader_snapshot_table_info).

The non-concurrent behavior is that all exports must be done snapshotting before replication can start. The capabilities for all ports are dropped when this operator returns with Ok ln:1177. Returning an error does not have the same effect (that's why we're safe above):

if let Err(err) = constructor(&mut *caps).await {
error_output.give(&error_cap, Rc::new(err));
drop(error_cap);
// IMPORTANT: wedge this operator until the button is pressed. Returning would drop
// the capabilities and could produce incorrect progress statements.
std::future::pending().await
}

@martykulma
martykulma force-pushed the maz-mysql-per-export-progress branch 4 times, most recently from 78e0250 to f7117e7 Compare November 5, 2026 18:21
…napshot

With `storage_persist_sink_description_lookahead` nonzero, each MySQL export's
port holds the minimum capability only while that export still has snapshot
data to emit or a pending rewind. Exports that are not being snapshotted
advance with the replication stream, and a snapshotted export advances once
its own copy finishes and its rewind resolves. With the lookahead at zero, all
ports still advance together.

Co-Authored-By: Claude <[email protected]>
@martykulma
martykulma force-pushed the maz-mysql-per-export-progress branch from f7117e7 to c95f468 Compare November 5, 2026 21:56
@martykulma
martykulma merged commit 1dfbe05 into main Nov 5, 2026
85 checks passed
@martykulma
martykulma deleted the maz-mysql-per-export-progress branch November 5, 2026 22:31
martykulma added a commit that referenced this pull request Nov 5, 2026
…#40206)

### Motivation

Today a MySQL source snapshots all of its tables before it reads any of
the binlog. This PR lets the replication operator read the binlog while
the snapshot runs, as #39419 did for PostgreSQL. #40207 then lets each
export's frontier advance on its own.

NOTE: this needs an exception added for the 2s increase in snapshot
performance, but I can't add it until I have a merge hash from main.
Will do in a follow up PR.

Closes: [SS-488](https://linear.app/materializeinc/issue/SS-488)

### Description

All new behavior is keyed off
`storage_persist_sink_description_lookahead` being nonzero. This is the
same switch PostgreSQL uses.

With a nonzero lookahead, the snapshot operator emits its rewind
requests as soon as the snapshot GTID frontier is known. That happens
before any table is read. The replication operator then reads the binlog
while the tables are read. Every export still advances together.
Replication data is staged in the dataflow until the snapshot completes.

With the lookahead at zero, rewinds are emitted after the snapshot
completes, as before.

The snapshot operator reads the setting once at render time. That
matches the value the persist sink reads in the same dataflow build.

MySQL sources now count as OLTP sources in `render.rs`. A snapshotting
MySQL export therefore gets the persist sink's snapshot ceiling and
lookahead grouping, like a PostgreSQL export. The mzcompose default sets
the lookahead to 2s. The MySQL initial load feature benchmarks should
see their first rows about that much later, as the PgCdc benchmarks did
after #39602.

#39419 also added a leader check that the replication slot exists before
the snapshot starts. MySQL needs no counterpart. The replication
operator already checks `@@gtid_purged` against the resume upper before
it waits for rewind requests.

### Verification

The mzcompose default sets the lookahead, so the existing
mzcompose-based MySQL source tests exercise the early rewind path.
#40208 adds targeted coverage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
@bosconi bosconi mentioned this pull request Nov 6, 2026
val-materialize added a commit that referenced this pull request Nov 13, 2026
Dates in RC snapshots are provisional until the release ships. One
commit is added per snapshot; the final snapshot sets the real dates.
**The `final` snapshot has been applied: the dates below are the real
ones** (Cloud 2026-11-12, Self-Managed 2026-11-13).

The `final` snapshot adds no entries — `v26.50.0-rc.1...v26.50.0` is one
commit, the version bump, and no agent-skills PR merged in the window.
Its substantive work was the dates, a rebase onto `main`, and the three
removals below.

**Rebased onto `main`.** The branch was 107 commits behind and
conflicted: `main`'s `## v26.49.1` section replaced the stale `##
v26.49.0` stub the branch was cut from. Resolved by keeping `main`'s
section verbatim and re-inserting `## v26.50.0` above it; same for the
compatibility YAML (`v26.50`, then `v26.49.1`, then `v26.48.2`).

**Three entries removed as already published on `main`.** The `v26.49.1`
patch train cherry-picked four Console PRs that also merged to `main`
inside this train's `rc.1` window, so both trains classified them
independently and `main` already carries them under `## v26.49.1`:

| PR(s) | Published on `main` in `## v26.49.1` as | Removed from `##
v26.50.0` |
|---|---|---|
| [#40044](#40044),
[#40074](#40074) | the
feature section *"Right size your clusters, using the new Hydration tab
on the console"*, with a screenshot | *"**Diagnose hydration from the
cluster details page**"* |
| [#40511](#40511) |
*"**Freshness and cluster details pages on Self-Managed**"*, covering
both pages | *"**The Freshness page on Self-Managed**"* |
| [#40492](#40492) |
the Bug Fix about the Console's clusters list showing a dash in the Disk
column | *"**Swap usage in the Console's Disk column**"* |

`main`'s wording is the later and fuller treatment in all three cases.
That leaves `## v26.50.0` with **5 Improvements and 7 Bug Fixes**.

Decisions required:

- **MCP protocol default flip**
([#39762](#39762)):
published as an Improvement because `enable_mcp_protocol_2026_07_28` is
`true` at the tag and the flip *is* the change. The PR was held in draft
"until the 2026-07-28 code ships in a release and passes a test on a
deployed environment" — is it confirmed on?
- **Columnar accumulators**
([#40364](#40364)):
omitted as Borderline — the default is `true` but the PR states no
measured user-facing effect. Is there a memory or CPU figure worth
publishing?
- **MySQL snapshot/replication concurrency**
([#40206](#40206),
[#40207](#40207)):
omitted as Borderline — gated on
`storage_persist_sink_description_lookahead`, which is `Duration::ZERO`
at the tag, same as the PostgreSQL counterparts held back in v26.47.0.
If the flag is served on, both halves want announcing together.
- **MCP servers GA**
([#40448](#40448)):
omitted as Borderline — the preview banners are gone, but the GA call
belongs to the release owner. Publish it?
- **Balancer connection metrics**
([#40192](#40192)):
omitted as Borderline — `mz_balancer_connection_status{status="error"}`
changes meaning, so Self-Managed dashboards and alerts on that series
will see a step change. Worth a note?
- **Console query comments**
([#40075](#40075)):
omitted as Borderline — every Console statement's `sql_hash` changes
once when this deploys. Worth calling out?

Every flag above was re-read in source at the `v26.50.0` ship tag; not
one default moved between `rc.1` and ship, so all six holds still read
correctly.

Snapshot drafts (with PR links) in
[mz-skills](https://github.com/MaterializeInc/mz-skills):

- rc.1:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/release_notes.md),
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md)
- final:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/release_notes.md),
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md)

**Borderline PRs omitted:** rc.1: 7 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/rc.1/omitted.md#borderline),
final: 0 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.50.0/final/omitted.md#borderline)
(no PRs in scope; `rc.1`'s six holds re-examined at the ship tag and
unchanged)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>
Co-authored-by: Valerie Chiang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants