SS-457 - storage: read the PostgreSQL replication stream during a snapshot by martykulma · Pull Request #39419 · MaterializeInc/materialize

martykulma · 2026-09-30T19:59:56Z

Motivation

Today a PostgreSQL source snapshots all of its tables before it reads any of the replication stream. This PR lets the replication operator read the stream while the snapshot runs. #39479 then lets each export's frontier advance on its own.

Description

All new behavior is keyed off storage_persist_sink_description_lookahead being nonzero. That dyncfg already controls whether the persist sink groups a snapshotting export's updates into one batch, and it defaults to zero in production and to 5s in mzcompose. Concurrent replication is what makes a snapshot accumulate many timestamps, and reading the stream concurrently without the grouping keeps one batch builder per timestamp resident until each reaches the blob target size. The two are therefore one setting rather than a separate flag.

With a nonzero lookahead, the snapshot operator emits its rewind requests as soon as the snapshot LSN is known, before any table is copied. The replication operator then reads the stream while tables are copied. Every export still advances together, so replication data is staged in the dataflow until the snapshot completes. This trades cluster memory during hydration for progress on the replication stream.

With the lookahead at zero, rewinds are emitted after the copy completes, as before.

The snapshot operator reads the setting once, since the early and the late emission must not both happen or both be skipped. Only the snapshot operator reads it in this PR, so a change takes effect for snapshots that start afterwards. parallel-workload already rolls the lookahead through zero and nonzero values.

The persist sink logs the resolved lookahead per export at dataflow build time, so a hydration's logs show whether grouping and concurrent replication are on.

Verification

The mzcompose default sets the lookahead to 5s, so the existing mzcompose-based PostgreSQL source tests exercise the early rewind path. #39420 adds targeted coverage.

🤖 Generated with Claude Code

linear-code · 2026-10-02T13:01:41Z

SS-457

def- · 2026-10-07T15:34:26Z

QA LLM Review

1. MEDIUM -- Hydrated exports pile up one unflushed batch builder per tick while a sibling table snapshots

src/storage/src/source/postgres/snapshot.rs:703

With a nonzero lookahead, adding a table to a Postgres source that already has hydrated tables now feeds those hydrated exports CDC for the whole copy. Their frontier stays pinned and the persist sink's grouping is off for them. Each tick's rows sit in a separate in-memory BatchBuilder per worker until the copy finishes, so replica memory grows with upstream write volume times snapshot duration. That is the shape the PR description says coupling to the lookahead prevents.

Details

The snapshot operator holds every export's data capability at the minimum until it returns (snapshot.rs:417), so a hydrated export's reclocked frontier stays at the as_of for the whole copy. Before this change the replication operator did not read during that window, so those sinks received nothing.
The sink enables the lookahead only for exports whose resume upper is at or below the as_of (src/storage/src/render.rs:394, persist_sink.rs:593). A hydrated export gets snapshot_time = None, so next_mint never commits a ceiling for it.
Its rows therefore go to uncovered_builders.entry(ts) (persist_sink.rs:1036). Each builder stays resident until it reaches persist_blob_target_size (128 MiB) or a description covers it, and no description comes until the snapshot ends.
Remap now ticks during the copy because raw_stream's probe task runs (replication.rs:816). A multi-hour copy of a new table next to a busy existing one therefore means thousands of builders per worker. An OOM restarts the snapshot from scratch, which hits the same pile-up again.

Trigger: CREATE TABLE ... FROM SOURCE or ALTER SOURCE ... ADD SUBSOURCE on a source whose existing tables take writes, with the lookahead nonzero (the mzcompose default).

#39479 releases the hydrated exports' capabilities, which removes this. If this PR can land or be enabled without it, emit early only when every data export is snapshotting in this incarnation, and keep the mixed case serial.

martykulma · 2026-10-07T17:21:01Z

MEDIUM -- Hydrated exports pile up one unflushed batch builder per tick while a sibling table snapshots

As noted, the next PR in the stack addresses this. The PRs in the stack will be merged together, they're just broken apart to make reviewing easier.

peterdukelarsen · 2026-10-08T21:43:03Z

@@ -361,6 +361,13 @@ pub(crate) fn render<'scope>(
        dyncfgs::STORAGE_PERSIST_SINK_DESCRIPTION_LOOKAHEAD.get(config_set),
        timestamp_interval,


Thoughts on pulling this value once during render and wiring it down to avoid the risk of races across operators?

This is currently being captured before the persist sink operators are created (happens further down in this function).

The value is pulled from the storage configuration, which is only updated via a sequenced command (UpdateConfiguration). It's safe from races, barring any bugs in the command sequencing.

oh - maybe you mean the one in snapshot.rs? That one I will definitely move out of the operator!

peterdukelarsen

Logic and change look like they do what they say they do. As we discussed, recommend that you merge this in lockstep with the next change to avoid any risk of accidentally accruing a ton of little batches.

def- · 2026-10-09T04:19:05Z

QA LLM Review

1. MEDIUM -- A definite replication error during a snapshot turns into a restart loop instead of being committed

src/storage/src/source/postgres/snapshot.rs:706

With a nonzero lookahead, the replication operator hits its definite errors (invalidated slot, timeline change, replica promotion) before the copy. The snapshot operator still pins every export at the minimum, so the error cannot commit, and the halting health update restarts the dataflow 5 s later. A source whose copy and append take longer than that window re-copies its tables every few seconds indefinitely, and reads of its tables block instead of returning the error.

Details

Every replication error, definite or transient, becomes HealthStatusUpdate::halting (src/storage/src/source/postgres.rs:236), which sends SuspendAndRestart after storage_suspend_and_restart_delay (5 s).
The replication operator gives the error at u64::MAX and returns. The snapshot operator holds its data capabilities until it returns, so like the snapshot rows, the error can only commit after the copy.
The snapshot operator doesn't check the persistent errors: InvalidReplicationSlot (replication.rs:796, for example a slot invalidated by max_slot_wal_keep_size), timeline mismatch (replication.rs:729), InvalidPhysicalReplica (replication.rs:722), SlotCompactedPastResumePoint (replication.rs:360). With rewinds after the copy, the copy finishes first and the error commits once.
SS-492 - storage: let PostgreSQL exports progress independently during a snapshot #39479 still holds a snapshotted export's capability until its own copy finishes, so it doesn't change this.

Repro with plain docker, A/B against the base PR image, lookahead 2s. Postgres 16 runs with max_slot_wal_keep_size=64MB. Hydrate one table, set replication factor 0, write WAL until the slot is lost, restore replication factor 1, then CREATE TABLE big FROM SOURCE:

5 GB table. Base commits the error at t+125s after 1 restart. This PR showed 48 restarts in 16 minutes with big's upper stuck at [0], each cycle creating a temp slot and starting up to 16 parallel COPYs upstream. SELECT count(*) FROM big blocked for over 10 minutes despite statement_timeout = '10s', while on the base it returns the slot-invalidated error.
1 GB table. Base commits the error after 2 restarts (about 40 s). This PR committed it only after 42 restarts (about 6 min), on a cycle where copy and append happened to fit in the window.

One direction is to run the slot, timeline and replica checks in the snapshot operator before the early rewinds. They would then fail through its existing early-return path, which drops the capabilities, as PublicationMissing already does.

martykulma · 2026-10-09T14:55:32Z

MEDIUM -- A definite replication error during a snapshot turns into a restart loop instead of being committed

confirmed and fixed

peterdukelarsen

The new commit (4e06932) LGTM

Behind `storage_persist_sink_description_lookahead`, the snapshot operator emits its rewind requests as soon as the snapshot LSN is known instead of after the copy completes, so the replication operator reads the stream while tables are copied. Replication data is staged in the dataflow until the snapshot completes, and every export still advances together. The flag defaults off, which keeps the serial snapshot-then-replication behavior. It defaults on in mzcompose. Co-Authored-By: Claude <[email protected]>

… out of the async operator

…g a snapshot (#39479) ### Motivation Every export's frontier stays at the minimum until the last table's COPY finishes. Adding one large table to a source therefore still stalls ingestion for every table already hydrated in that source. ### Description With `storage_persist_sink_description_lookahead` nonzero, each export's port holds the minimum capability only while that export still has snapshot data to emit or a pending rewind. - Exports that are not being snapshotted drop their snapshot capabilities up front, so they advance with the replication stream. - A snapshotted export releases its snapshot capability as soon as its own copy finishes on each worker. - The replication operator downgrades every port without a pending rewind at each yield point. With the lookahead at zero, all ports advance together, so behavior matches #39419. The replication operator now reads the setting as well. The two operators read it independently at startup, and parallel-workload rolls the lookahead through zero and nonzero values, so a change can land between those two reads and leave the operators in different modes within one dataflow. That is benign in both directions. With only the snapshot operator concurrent, the replication operator holds every port until the rewinds resolve and then releases them all. With only the replication operator concurrent, it still waits for the rewind input to close, which the serial snapshot operator does after the copy. ### Verification The mzcompose default sets the lookahead, so the existing mzcompose-based PostgreSQL source tests exercise per-export progress. #39420 adds targeted coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]>

Release notes for the **v26.47.0** train. One commit was added per snapshot; the `final` snapshot sets the real dates. **All six snapshots (`rc.1`–`rc.5`, `final`) are now applied — this branch is complete and ready for review.** - **Content**: 2 Feature sections (`TRY_CAST`, `EXCLUDE ALL NOT NULL CONSTRAINTS`), 7 Improvements, 13 Bug Fixes — `rc.1` contributed all of it except the top four Bug Fixes: `rc.2` added two, and `rc.3` and `rc.4` one each. `rc.5` added no new entries; it amended `rc.4`'s probe-port Bug Fix instead (see below). `final` adds nothing at all — its tracking issue holds only the unassociated `release: bump to version v26.47.0` commit ([`bed23494d`](bed2349)), and the `v26.47.0-rc.5` and `v26.47.0` tags are 14h28m apart with that bump the only thing between them. Its commit is empty and exists to record on the branch that the snapshot was applied. - **Dates (final, no longer provisional)**: Cloud **2026-10-22**, Self-Managed **2026-10-23**. Every RC estimated this same pair from the `v26.47.0-rc.1` tag, and the real dates confirmed it — the five RCs compressed into one week rather than pushing the train out — so no published date text changed in the `final` commit. ⚠️ One thing worth a last look before merge: **2026-10-22 is a Wednesday**, one day off the usual Thursday Cloud date (2026-10-23 is the normal Friday Self-Managed day). That is the supplied value and it stands, but if the release owner expected Thursday 2026-10-23 / Friday 2026-10-24, this branch is where to correct it. - **Operator compatibility**: row added for `v26.47`, release date `2026-10-23`. - **Branch base**: `main`, whose newest section is `## v26.45.1` — v26.46.0's section is still on its own unmerged branch (#39690). Expect a top-of-list conflict when both merge; `v26.47.0` goes above `v26.46.0`. ✅ **Corrections already on this branch** — a `final` verification pass over the whole assembled section confirmed these and deliberately left them alone: - **`rc.1` named a syntax that does not exist.** It published "`CREATE TABLE ... FROM SOURCE` now accepts an `ALL COLUMNS NULLABLE` option", taking the name from [#39680](#39680 title and body. At the `v26.47.0` tag the option is `EXCLUDE ALL NOT NULL CONSTRAINTS` (`TableFromSourceOptionName::ExcludeAllNotNullConstraints`), and `NULLABLE` is not a lexer keyword at all, so the published syntax would have been a parse error for anyone who copied it. A human editor caught this and replaced the bullet with the `{#v26.47-exclude-not-null-constraints}` feature section now on the branch. The `rc.1` snapshot directory in mz-skills still carries the wrong wording — snapshot directories are the record of what was reviewed at the time and are left untouched. - **`rc.5`'s cross-snapshot rewrite is applied once**, including the "Deployments using password, SASL, or OIDC authentication have no separate internal SQL port and continue to be probed on the external one" clause. - **PR citations are stripped from all entries** as a deliberate house-style edit, matching how published sections read. PR links remain available in the mz-skills snapshot drafts linked below. ⚠️ **Open questions for the release owner** — none of these were resolved by `final`; this branch is the last place to act on them - **Was any `rc.4` artifact handed to a customer or used for an upgrade rehearsal?** This decides a published entry. [#40036](#40036) (published as a Bug Fix by `rc.4`) made the operator probe `environmentd`'s internal SQL port unconditionally, which crashloops deployments using Password, SASL, or OIDC authentication — they have no internal SQL port. [#40046](#40046) fixed that roughly seven hours later, inside this same unreleased train, so no shipped build (`v26.46.1` probes the external port unconditionally) could have hit it. It is therefore recorded as **Borderline** rather than published as a Bug Fix of its own, and its substance folded into `rc.4`'s bullet as a scoping clause. If an `rc.4` build did reach anyone, the crashloop *was* reachable and [#40046](#40046) should be published in its own right. Note the author wrote an explicit positive release-note line for it, which is why this is flagged rather than quietly dropped. - **GA announcements — the largest judgment call in this train.** Three docs PRs remove preview markers: [#39831](#39831) (source versioning, `CREATE TABLE ... FROM SOURCE`), [#39836](#39836) (replacement materialized views), [#39830](#39830) (Kafka sink headers, noted as "GA in v26.47"). None changes code, so all three were held; drafted wording for each is in `omitted.md`. Which are actually GA in v26.47.0 and should be announced? Note v26.46.0 published Kafka sink message attribution as *private preview* — if the header goes GA here, this release should say so rather than repeat the preview framing. - **GCS blob read throughput** ([#39906](#39906), held in `rc.2`): restores the read throughput GCS deployments lost when [#39351](#39351) — already shipped — forced HTTP/2 and collapsed per-request connections into one. It is selected by `persist_blob_http_pool`, verified to default to `off` at the `v26.47.0-rc.2` tag, and the author notes "the default should eventually be 'GCS-only'". **Is `persist_blob_http_pool` being set to `gcs` for v26.47.0?** If so this is a clear win to publish for GCP users, who carry the regression today; drafted wording is in rc.2's `omitted.md`. - **`SUBSCRIBE` soft assertions** ([#37611](#37611), [#39764](#39764), both held): drafted as Bug Fixes, then pulled back. Both failures come from `soft_assert_or_log!`, which panics only where soft assertions are enabled and otherwise logs; neither PR says what a release build does. Are soft assertions on in the environments customers run? If yes, both are publishable as drafted in `omitted.md`. - **Console Freshness page** ([#39340](#39340), held): a complete new Monitoring > Freshness page, gated behind `freshness-page-CNS164`, off in both Cloud and self-managed, and the LaunchDarkly flag is not yet created. If it is turned on for v26.47.0, this likely deserves a Feature heading. - **PostgreSQL sources ingesting during a snapshot** ([#39419](#39419), [#39479](#39479), held): adding a large table to an existing source would no longer stall the tables already hydrated — but `storage_persist_sink_description_lookahead` is `Duration::ZERO` at the tag. Publish if the lookahead is being enabled. - **Coalesced materialized-view sink batches** ([#37591](#37591), held): measures ~67% fewer blob writes and ~54% smaller state diffs, at ~10 ms added p50 latency on small writes, but `compute_mv_sink_shared_batch_max_bytes` is `0` at the tag. Publish (with the latency cost stated) if the staged rollout reaches customers in this release. - **MCP 2026-07-28 conformance checks** ([#39694](#39694), [#39695](#39695), held): `enable_mcp_protocol_2026_07_28` is `false` at the tag. If the revision is being served, these belong together as one Improvement. - **Refresh strategy columns** ([#39597](#39597), published as an Improvement): this renames `aligned_to` → `aligned_to_expression` and `at` → `at_expression` and changes their type. It is a breaking change for anyone selecting them, in a schema documented as unstable — it may warrant more prominence than a bullet. - **Retained input after `ALTER SINK ... SET FROM`** (from `rc.3`'s [#37653](#37653)): the PR records a pre-existing gap it does not fix — after `SET FROM`, the *old* upstream relation stays held, and therefore un-compactable, until the sink is dropped or `environmentd` restarts. It is not something this release changed, so it is out of these notes, but it is a real footgun for the blue/green pattern that `sql/alter-sink.md` recommends `SET FROM` for. Should the `ALTER SINK` page or the blue/green guide call it out? - **Operator upgrade shows pending changes** (from `rc.4`'s [#40036](#40036)): the probe-port change alters the generated `environmentd` StatefulSet, so after upgrading the operator, existing environments report pending changes and pick up the new port on their next rollout — they are **not** restarted on their own. That is an upgrade consequence rather than something the release changed, so it is out of the notes, but a Self-Managed operator who cannot account for the pending-changes state may read it as a problem. The v26.47 upgrade guide looks like the right venue. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/omitted.md) - rc.4: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/omitted.md) - rc.5: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 14 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/omitted.md#borderline), rc.2: 2 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/omitted.md#borderline), rc.3: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/omitted.md#borderline), rc.4: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/omitted.md#borderline), rc.5: 1 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/omitted.md#borderline), final: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/omitted.md#borderline) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>

…#40206) ### Motivation Today a MySQL source snapshots all of its tables before it reads any of the binlog. This PR lets the replication operator read the binlog while the snapshot runs, as #39419 did for PostgreSQL. #40207 then lets each export's frontier advance on its own. NOTE: this needs an exception added for the 2s increase in snapshot performance, but I can't add it until I have a merge hash from main. Will do in a follow up PR. Closes: [SS-488](https://linear.app/materializeinc/issue/SS-488) ### Description All new behavior is keyed off `storage_persist_sink_description_lookahead` being nonzero. This is the same switch PostgreSQL uses. With a nonzero lookahead, the snapshot operator emits its rewind requests as soon as the snapshot GTID frontier is known. That happens before any table is read. The replication operator then reads the binlog while the tables are read. Every export still advances together. Replication data is staged in the dataflow until the snapshot completes. With the lookahead at zero, rewinds are emitted after the snapshot completes, as before. The snapshot operator reads the setting once at render time. That matches the value the persist sink reads in the same dataflow build. MySQL sources now count as OLTP sources in `render.rs`. A snapshotting MySQL export therefore gets the persist sink's snapshot ceiling and lookahead grouping, like a PostgreSQL export. The mzcompose default sets the lookahead to 2s. The MySQL initial load feature benchmarks should see their first rows about that much later, as the PgCdc benchmarks did after #39602. #39419 also added a leader check that the replication slot exists before the snapshot starts. MySQL needs no counterpart. The replication operator already checks `@@gtid_purged` against the resume upper before it waits for rewind requests. ### Verification The mzcompose default sets the lookahead, so the existing mzcompose-based MySQL source tests exercise the early rewind path. #40208 adds targeted coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]>

martykulma added this pull request to stack #39421 September 30, 2026 20:00

martykulma mentioned this pull request Sep 30, 2026

SS-492 - test: cover PostgreSQL exports progressing during a held snapshot #39420

Merged

martykulma force-pushed the maz-pg-concurrent-replication branch from 21e4634 to 5fcda51 Compare October 1, 2026 15:52

martykulma force-pushed the maz-pg-concurrent-replication branch from 5fcda51 to d1c6d07 Compare October 2, 2026 11:36

martykulma removed this pull request from stack #39421 October 2, 2026 11:39

martykulma added this pull request to stack #39480 October 2, 2026 11:39

martykulma changed the title ~~storage: let PostgreSQL exports progress independently during a snapshot~~ storage: read the PostgreSQL replication stream during a snapshot Oct 2, 2026

martykulma mentioned this pull request Oct 2, 2026

SS-492 - storage: let PostgreSQL exports progress independently during a snapshot #39479

Merged

martykulma changed the title ~~storage: read the PostgreSQL replication stream during a snapshot~~ SS-457 - storage: read the PostgreSQL replication stream during a snapshot Oct 2, 2026

martykulma force-pushed the maz-pg-concurrent-replication branch 2 times, most recently from 09f2b94 to b6dec28 Compare October 7, 2026 13:45

martykulma marked this pull request as ready for review October 7, 2026 14:20

martykulma requested a review from a team as a code owner October 7, 2026 14:20

martykulma requested a review from peterdukelarsen October 7, 2026 14:26

martykulma force-pushed the maz-pg-concurrent-replication branch 2 times, most recently from 33e31e5 to 8ea4c9f Compare October 7, 2026 17:15

peterdukelarsen reviewed Oct 8, 2026

View reviewed changes

peterdukelarsen approved these changes Oct 8, 2026

View reviewed changes

martykulma force-pushed the maz-pg-concurrent-replication branch from 8ea4c9f to 0357f9e Compare October 9, 2026 02:37

peterdukelarsen approved these changes Oct 13, 2026

View reviewed changes

martykulma and others added 3 commits October 14, 2026 11:06

snapshot.rs: hoist reading STORAGE_PERSIST_SINK_DESCRIPTION_LOOKAHEAD…

6b3e537

… out of the async operator

PR feedback: detect errors before snapshot

d09db3b

martykulma force-pushed the maz-pg-concurrent-replication branch from 4e06932 to d09db3b Compare October 14, 2026 15:06

martykulma merged commit f1791b5 into main Oct 14, 2026
83 checks passed

martykulma deleted the maz-pg-concurrent-replication branch October 14, 2026 16:40

bosconi mentioned this pull request Oct 16, 2026

v26.47.0 release notes #39891

Merged

val-materialize mentioned this pull request Oct 23, 2026

docs: document sizing for PostgreSQL non-blocking add-table snapshots #40124

Merged

martykulma mentioned this pull request Oct 27, 2026

SS-488 - storage: read the MySQL replication stream during a snapshot #40206

Merged

Navigation Menu

Repository navigation

SS-457 - storage: read the PostgreSQL replication stream during a snapshot - #39419

SS-457 - storage: read the PostgreSQL replication stream during a snapshot#39419
martykulma merged 3 commits into
maz-pg-demux-exportsfrom
maz-pg-concurrent-replication

martykulma commented Sep 30, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Oct 2, 2026

Uh oh!

def- commented Oct 7, 2026

Uh oh!

martykulma commented Oct 7, 2026

Uh oh!

peterdukelarsen Oct 8, 2026

Uh oh!

martykulma Oct 9, 2026

Uh oh!

martykulma Oct 9, 2026

Uh oh!

peterdukelarsen left a comment

Uh oh!

def- commented Oct 9, 2026

Uh oh!

martykulma commented Oct 9, 2026

Uh oh!

peterdukelarsen left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -361,6 +361,13 @@ pub(crate) fn render<'scope>(
		dyncfgs::STORAGE_PERSIST_SINK_DESCRIPTION_LOOKAHEAD.get(config_set),
		timestamp_interval,

Conversation

martykulma commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Verification

Uh oh!

linear-code Bot commented Oct 2, 2026

Uh oh!

def- commented Oct 7, 2026

QA LLM Review

1. MEDIUM -- Hydrated exports pile up one unflushed batch builder per tick while a sibling table snapshots

Uh oh!

martykulma commented Oct 7, 2026

Uh oh!

peterdukelarsen Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

martykulma Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

martykulma Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen left a comment

Choose a reason for hiding this comment

Uh oh!

def- commented Oct 9, 2026

QA LLM Review

1. MEDIUM -- A definite replication error during a snapshot turns into a restart loop instead of being committed

Uh oh!

martykulma commented Oct 9, 2026

Uh oh!

peterdukelarsen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martykulma commented Sep 30, 2026 •

edited

Loading

peterdukelarsen left a comment •

edited

Loading