Skip to content

SS-489 - test: cover MySQL exports progressing during a held snapshot - #40208

Merged
martykulma merged 1 commit into
maz-mysql-per-export-progressfrom
maz-mysql-snapshot-freshness-tests
Nov 5, 2026
Merged

martykulma merged 1 commit into
maz-mysql-per-export-progressfrom
maz-mysql-snapshot-freshness-tests

Conversation

@martykulma

@martykulma martykulma commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Adds targeted coverage for concurrent replication in the MySQL source, mirroring #39420 for PostgreSQL.

NOTE: this needs an exception added for the 2s increase in snapshot performance, but I can't add it until I have a merge hash from main. Will do in a follow up PR.

Closes: SS-489

Description

Adds a mysql_snapshot_pause failpoint to the MySQL snapshot operator. return(<table>) holds that table's read until the failpoint is deactivated. return(<table>:<millis>) holds it for a fixed time. The snapshot then completes in the same dataflow incarnation. Each worker reads its tables in order of their qualified names. Tables ordered before the held one finish their snapshots even when the same worker reads them.

Two new test/cluster workflows share setup and add-tables phases in test/cluster/mysql-snapshot-freshness/.

mysql-snapshot-freshness holds t_paused indefinitely. It checks that an already hydrated table keeps ingesting CDC. It checks that t_alongside, added in the same transaction, completes its own snapshot. It also checks that CDC for the held table is staged without committing. It then restarts clusterd without the failpoint and checks that every table converges. The row inserted during the hold appears exactly once.

mysql-snapshot-freshness-release holds t_paused for 30 seconds while CDC for it arrives across several timestamps. When the hold lifts, it checks that the snapshot and all staged CDC commit exactly once without a restart. It also checks that later CDC lands as for any hydrated table.

The cluster composition now includes a MySQL service. The cluster-tests and slow-cluster CI steps gain the mysql topic accordingly.

Verification

Both workflows are new in this PR. They rely on the mzcompose default that sets storage_persist_sink_description_lookahead, which turns concurrent replication on. With the lookahead set to zero, mysql-snapshot-freshness fails waiting for t_alongside.

🤖 Generated with Claude Code

@martykulma
martykulma added this pull request to stack #40209 October 27, 2026 01:55
@linear-code

linear-code Bot commented Oct 27, 2026

Copy link
Copy Markdown

SS-489

@def- def- left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to see more tests here!

@martykulma
martykulma force-pushed the maz-mysql-snapshot-freshness-tests branch 4 times, most recently from fe7ade6 to 4da8093 Compare November 5, 2026 18:21
@martykulma
martykulma force-pushed the maz-mysql-snapshot-freshness-tests branch from 4da8093 to 73d91d3 Compare November 5, 2026 21:56
@martykulma
martykulma merged commit 27e5dac into main Nov 5, 2026
91 checks passed
@martykulma
martykulma deleted the maz-mysql-snapshot-freshness-tests branch November 5, 2026 22:31
martykulma added a commit that referenced this pull request Nov 5, 2026
…#40206)

### Motivation

Today a MySQL source snapshots all of its tables before it reads any of
the binlog. This PR lets the replication operator read the binlog while
the snapshot runs, as #39419 did for PostgreSQL. #40207 then lets each
export's frontier advance on its own.

NOTE: this needs an exception added for the 2s increase in snapshot
performance, but I can't add it until I have a merge hash from main.
Will do in a follow up PR.

Closes: [SS-488](https://linear.app/materializeinc/issue/SS-488)

### Description

All new behavior is keyed off
`storage_persist_sink_description_lookahead` being nonzero. This is the
same switch PostgreSQL uses.

With a nonzero lookahead, the snapshot operator emits its rewind
requests as soon as the snapshot GTID frontier is known. That happens
before any table is read. The replication operator then reads the binlog
while the tables are read. Every export still advances together.
Replication data is staged in the dataflow until the snapshot completes.

With the lookahead at zero, rewinds are emitted after the snapshot
completes, as before.

The snapshot operator reads the setting once at render time. That
matches the value the persist sink reads in the same dataflow build.

MySQL sources now count as OLTP sources in `render.rs`. A snapshotting
MySQL export therefore gets the persist sink's snapshot ceiling and
lookahead grouping, like a PostgreSQL export. The mzcompose default sets
the lookahead to 2s. The MySQL initial load feature benchmarks should
see their first rows about that much later, as the PgCdc benchmarks did
after #39602.

#39419 also added a leader check that the replication slot exists before
the snapshot starts. MySQL needs no counterpart. The replication
operator already checks `@@gtid_purged` against the resume upper before
it waits for rewind requests.

### Verification

The mzcompose default sets the lookahead, so the existing
mzcompose-based MySQL source tests exercise the early rewind path.
#40208 adds targeted coverage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
martykulma added a commit that referenced this pull request Nov 5, 2026
…napshot (#40207)

### Motivation

Every export's frontier stays at the minimum until the last table's
snapshot read finishes. Adding one large table to a MySQL source
therefore still stalls ingestion for every table already hydrated in
that source. #39479 fixed the same problem for PostgreSQL.

### Description

With `storage_persist_sink_description_lookahead` nonzero, each export's
port holds the minimum capability only while that export still has
snapshot data to emit or a pending rewind.

- Exports that are not being snapshotted drop their snapshot
capabilities up front. They advance with the binlog.
- After a worker plans its reads, it drops the ports of exports it reads
no part of. This also covers exports that failed schema verification.
- A snapshotted export releases its snapshot capability on each worker
as soon as that worker finishes reading its table.
- The replication operator downgrades every port without a pending
rewind at each yield point.

With the lookahead at zero, all ports advance together. Behavior then
matches #40206.

The replication operator now reads the setting at render time as well.
Both operators read it in the same dataflow build and agree with the
persist sink.

### Verification

The mzcompose default sets the lookahead, so the existing
mzcompose-based MySQL source tests exercise per-export progress. #40208
adds targeted coverage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <[email protected]>
@martykulma

Copy link
Copy Markdown
Contributor Author

thanks for the reviews!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants