compute: allow SUBSCRIBE AS OF MAX UP TO MAX by antiguru · Pull Request #37611 · MaterializeInc/materialize

antiguru · 2026-07-13T14:56:53Z

SUBSCRIBE ... AS OF MAX UP TO MAX describes the empty range [as_of, up_to): as_of is inclusive and equals the exclusive up_to.
The adapter accepts it and reports the EqualSubscribeBounds notice.
It resolves to as_of = {MAX}, until = {MAX}.

Previously the empty subscribe still lowered MIR to LIR, shipped a dataflow to the cluster, installed a sink, and tore it all down to produce nothing.
The as_of = MAX, until = MAX dataflow also tripped a soft assertion in is_single_time that wrongly expected an empty until whenever as_of was MAX, which catch_unwind_optimize surfaced as internal error in optimizer.

This short-circuits the empty subscribe in implement_subscribe, where the coordinator sequencer and frontend subscribe paths converge.
When as_of == up_to it emits the initial progress row from ActiveSubscribe::initialize (WITH PROGRESS only), then closes the channel to complete the subscribe, skipping dataflow shipping and sink registration.

The wrong is_single_time assertion is removed.
is_single_time runs during MIR to LIR lowering, before the short-circuit, so the empty dataflow still reaches it.
try_step_forward(MAX) returns None, so until = {MAX} is correctly classified as not single time and until = {} (unbounded) as single time.

Fixes CPU-169

aljoscha

looks good, but I think we need to find a better way to do the "per type" matching in there

aljoscha · 2026-10-07T08:43:36Z

+                ctx_extra: std::mem::take(ctx_extra),
+                instance_id: cluster_id,
+            };
+            let resp = match plan.copy_to {


feels a bit smelly that we're matching on a specific thing here, and this would break if we had more like this. I think we need a more future-proof solution

Agreed. The empty case no longer duplicates the response construction. It only skips sink registration and dataflow shipping, then falls through to the shared Subscribing/CopyTo construction, so new response shapes only need handling in one place.

Posted by Claude Code.

ggevay

LGTM, minor comments. Needs a rebase: it conflicts with main in dataflows.rs, and the shortcut no longer compiles on main (inline).

ggevay · 2026-10-10T17:23:55Z

+            drop(active_subscribe);
+            drop(read_holds);
+            let resp = ExecuteResponse::Subscribing {
+                rx,


Building on @aljoscha's point: this copy has already drifted. On main, RowBatchStream is a boxed stream and the regular path wraps rx for the backlog accounting (#37905), so this no longer compiles after a rebase. Suggest skipping only the sink registration and try_ship_dataflow for the empty case, then falling through to the shared response construction with a ready notify (that shape compiles on main). It also lets the empty case call active_subscribe.retire(ActiveComputeSinkRetireReason::Finished) instead of dropping it, as retire's doc asks.

Done as suggested in d9b42d5: the empty case skips add_active_compute_sink and try_ship_dataflow, returns a ready notify, and falls through to the shared response path. It calls active_subscribe.retire(ActiveComputeSinkRetireReason::Finished) instead of dropping it.

Posted by Claude Code.

ggevay · 2026-10-10T17:23:55Z

        };
        active_subscribe.initialize();

+        // A subscribe whose `as_of` equals its `up_to` covers the empty range


Worth saying why this is needed, not only what it skips: the sink only finishes once its input frontier reaches up_to, which a live input never does at MAX.

Added: the comment now states that the sink only finishes once its input frontier reaches up_to, which a live input never does at MAX.

Posted by Claude Code.

ggevay · 2026-10-10T17:23:55Z

+> DECLARE c CURSOR FOR SUBSCRIBE t1 AS OF 18446744073709551615 UP TO 18446744073709551615;
+
+# No rows expected.
+> FETCH ALL c WITH (timeout = '5s');


This passes without the shortcut: on v26.45.1, where this subscribe never completes, the FETCH just returns no rows after 5 s. Dropping the timeout makes testdrive hang instead of fail. What fails cleanly is checking, before each COMMIT, that nothing is left running:

> SELECT count(*) FROM mz_internal.mz_subscriptions 0

That returns 1 on v26.45.1 in both blocks and 0 with this PR.

Thanks, added the mz_internal.mz_subscriptions count check before each COMMIT. It passes with this PR.

Posted by Claude Code.

ggevay · 2026-10-10T17:23:55Z

+
+        // `as_of = MAX` with `until = {MAX}` is an empty range, produced by
+        // `SUBSCRIBE ... AS OF MAX UP TO MAX`. It must not be classified as single
+        // time, and it must not trip an invariant assertion (see CLU-169).


nit: CPU-169 (also in fetch-tail-as-of.td:65).

Fixed in both places.

Posted by Claude Code.

`DataflowDescription::is_single_time` asserted that an `as_of` of `Timestamp::MAX` implied an empty `until`. That is not an invariant. `SUBSCRIBE ... AS OF MAX UP TO MAX` describes an empty range (`as_of` inclusive equals the exclusive `up_to`), which the adapter accepts and reports with the `EqualSubscribeBounds` notice. It resolves to `as_of = {MAX}, until = {MAX}`, tripping the soft assertion. Because soft assertions panic in test and CI builds, `catch_unwind_optimize` surfaced it as `internal error in optimizer: internal transform error`. `MAX` is a legitimate `AS OF` / `UP TO` value. It is only unusable as the `as_of` of a bounded single-time subscribe, where `until = as_of + 1` would overflow. The final frontier comparison in `is_single_time` already classifies both valid `as_of = MAX` frontiers correctly: `until = {}` (unbounded) is single time, and `until = {MAX}` (empty range) is not. So the assertion was both wrong and redundant, and is removed. Adds a `mz-compute-types` unit test covering the three `is_single_time` frontier cases, and a testdrive regression in `fetch-tail-as-of.td`. Fixes CLU-169 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

A subscribe whose `as_of` equals its `up_to` covers the empty range `[as_of, up_to)` and is guaranteed to produce no data. Previously it still lowered MIR to LIR, shipped a dataflow to the cluster, installed a sink, and tore it all down to yield nothing. Short-circuit it in `implement_subscribe`, the point where both the coordinator sequencer and frontend subscribe paths converge. When `as_of == up_to`, emit the initial progress row that `ActiveSubscribe::initialize` produces (WITH PROGRESS only), then close the channel to complete the subscribe, skipping dataflow shipping and sink registration. The `is_single_time` assertion removal stays: `is_single_time` runs during MIR to LIR lowering, before `implement_subscribe`, so the empty subscribe would still trip it. The assertion was wrong and redundant regardless. Extends the `fetch-tail-as-of.td` regression with a WITH PROGRESS case that locks the preserved progress-row contract for the short-circuited path. Fixes CLU-169 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The `as_of <= until` soft assertion message read "expected empty `as_of ≤ until`". The "empty" is a leftover with no meaning here. Drop it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The empty-subscribe shortcut duplicated the response construction, which had drifted from the regular path. Skip only the sink registration and dataflow shipping, then fall through to the shared response construction with an already-satisfied notify. Retire the subscribe with `Finished` instead of dropping it, as `retire` requires. Treat every `up_to` that is not beyond `as_of` as empty, and explain why the shortcut is required: the sink only finishes once its input frontier reaches `up_to`, which a live input never does at `MAX`. The testdrive regression now checks that no subscription is left running. The FETCH timeout alone passes even when the subscribe never completes. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Release notes for the **v26.47.0** train. One commit was added per snapshot; the `final` snapshot sets the real dates. **All six snapshots (`rc.1`–`rc.5`, `final`) are now applied — this branch is complete and ready for review.** - **Content**: 2 Feature sections (`TRY_CAST`, `EXCLUDE ALL NOT NULL CONSTRAINTS`), 7 Improvements, 13 Bug Fixes — `rc.1` contributed all of it except the top four Bug Fixes: `rc.2` added two, and `rc.3` and `rc.4` one each. `rc.5` added no new entries; it amended `rc.4`'s probe-port Bug Fix instead (see below). `final` adds nothing at all — its tracking issue holds only the unassociated `release: bump to version v26.47.0` commit ([`bed23494d`](bed2349)), and the `v26.47.0-rc.5` and `v26.47.0` tags are 14h28m apart with that bump the only thing between them. Its commit is empty and exists to record on the branch that the snapshot was applied. - **Dates (final, no longer provisional)**: Cloud **2026-10-22**, Self-Managed **2026-10-23**. Every RC estimated this same pair from the `v26.47.0-rc.1` tag, and the real dates confirmed it — the five RCs compressed into one week rather than pushing the train out — so no published date text changed in the `final` commit. ⚠️ One thing worth a last look before merge: **2026-10-22 is a Wednesday**, one day off the usual Thursday Cloud date (2026-10-23 is the normal Friday Self-Managed day). That is the supplied value and it stands, but if the release owner expected Thursday 2026-10-23 / Friday 2026-10-24, this branch is where to correct it. - **Operator compatibility**: row added for `v26.47`, release date `2026-10-23`. - **Branch base**: `main`, whose newest section is `## v26.45.1` — v26.46.0's section is still on its own unmerged branch (#39690). Expect a top-of-list conflict when both merge; `v26.47.0` goes above `v26.46.0`. ✅ **Corrections already on this branch** — a `final` verification pass over the whole assembled section confirmed these and deliberately left them alone: - **`rc.1` named a syntax that does not exist.** It published "`CREATE TABLE ... FROM SOURCE` now accepts an `ALL COLUMNS NULLABLE` option", taking the name from [#39680](#39680 title and body. At the `v26.47.0` tag the option is `EXCLUDE ALL NOT NULL CONSTRAINTS` (`TableFromSourceOptionName::ExcludeAllNotNullConstraints`), and `NULLABLE` is not a lexer keyword at all, so the published syntax would have been a parse error for anyone who copied it. A human editor caught this and replaced the bullet with the `{#v26.47-exclude-not-null-constraints}` feature section now on the branch. The `rc.1` snapshot directory in mz-skills still carries the wrong wording — snapshot directories are the record of what was reviewed at the time and are left untouched. - **`rc.5`'s cross-snapshot rewrite is applied once**, including the "Deployments using password, SASL, or OIDC authentication have no separate internal SQL port and continue to be probed on the external one" clause. - **PR citations are stripped from all entries** as a deliberate house-style edit, matching how published sections read. PR links remain available in the mz-skills snapshot drafts linked below. ⚠️ **Open questions for the release owner** — none of these were resolved by `final`; this branch is the last place to act on them - **Was any `rc.4` artifact handed to a customer or used for an upgrade rehearsal?** This decides a published entry. [#40036](#40036) (published as a Bug Fix by `rc.4`) made the operator probe `environmentd`'s internal SQL port unconditionally, which crashloops deployments using Password, SASL, or OIDC authentication — they have no internal SQL port. [#40046](#40046) fixed that roughly seven hours later, inside this same unreleased train, so no shipped build (`v26.46.1` probes the external port unconditionally) could have hit it. It is therefore recorded as **Borderline** rather than published as a Bug Fix of its own, and its substance folded into `rc.4`'s bullet as a scoping clause. If an `rc.4` build did reach anyone, the crashloop *was* reachable and [#40046](#40046) should be published in its own right. Note the author wrote an explicit positive release-note line for it, which is why this is flagged rather than quietly dropped. - **GA announcements — the largest judgment call in this train.** Three docs PRs remove preview markers: [#39831](#39831) (source versioning, `CREATE TABLE ... FROM SOURCE`), [#39836](#39836) (replacement materialized views), [#39830](#39830) (Kafka sink headers, noted as "GA in v26.47"). None changes code, so all three were held; drafted wording for each is in `omitted.md`. Which are actually GA in v26.47.0 and should be announced? Note v26.46.0 published Kafka sink message attribution as *private preview* — if the header goes GA here, this release should say so rather than repeat the preview framing. - **GCS blob read throughput** ([#39906](#39906), held in `rc.2`): restores the read throughput GCS deployments lost when [#39351](#39351) — already shipped — forced HTTP/2 and collapsed per-request connections into one. It is selected by `persist_blob_http_pool`, verified to default to `off` at the `v26.47.0-rc.2` tag, and the author notes "the default should eventually be 'GCS-only'". **Is `persist_blob_http_pool` being set to `gcs` for v26.47.0?** If so this is a clear win to publish for GCP users, who carry the regression today; drafted wording is in rc.2's `omitted.md`. - **`SUBSCRIBE` soft assertions** ([#37611](#37611), [#39764](#39764), both held): drafted as Bug Fixes, then pulled back. Both failures come from `soft_assert_or_log!`, which panics only where soft assertions are enabled and otherwise logs; neither PR says what a release build does. Are soft assertions on in the environments customers run? If yes, both are publishable as drafted in `omitted.md`. - **Console Freshness page** ([#39340](#39340), held): a complete new Monitoring > Freshness page, gated behind `freshness-page-CNS164`, off in both Cloud and self-managed, and the LaunchDarkly flag is not yet created. If it is turned on for v26.47.0, this likely deserves a Feature heading. - **PostgreSQL sources ingesting during a snapshot** ([#39419](#39419), [#39479](#39479), held): adding a large table to an existing source would no longer stall the tables already hydrated — but `storage_persist_sink_description_lookahead` is `Duration::ZERO` at the tag. Publish if the lookahead is being enabled. - **Coalesced materialized-view sink batches** ([#37591](#37591), held): measures ~67% fewer blob writes and ~54% smaller state diffs, at ~10 ms added p50 latency on small writes, but `compute_mv_sink_shared_batch_max_bytes` is `0` at the tag. Publish (with the latency cost stated) if the staged rollout reaches customers in this release. - **MCP 2026-07-28 conformance checks** ([#39694](#39694), [#39695](#39695), held): `enable_mcp_protocol_2026_07_28` is `false` at the tag. If the revision is being served, these belong together as one Improvement. - **Refresh strategy columns** ([#39597](#39597), published as an Improvement): this renames `aligned_to` → `aligned_to_expression` and `at` → `at_expression` and changes their type. It is a breaking change for anyone selecting them, in a schema documented as unstable — it may warrant more prominence than a bullet. - **Retained input after `ALTER SINK ... SET FROM`** (from `rc.3`'s [#37653](#37653)): the PR records a pre-existing gap it does not fix — after `SET FROM`, the *old* upstream relation stays held, and therefore un-compactable, until the sink is dropped or `environmentd` restarts. It is not something this release changed, so it is out of these notes, but it is a real footgun for the blue/green pattern that `sql/alter-sink.md` recommends `SET FROM` for. Should the `ALTER SINK` page or the blue/green guide call it out? - **Operator upgrade shows pending changes** (from `rc.4`'s [#40036](#40036)): the probe-port change alters the generated `environmentd` StatefulSet, so after upgrading the operator, existing environments report pending changes and pick up the new port on their next rollout — they are **not** restarted on their own. That is an upgrade consequence rather than something the release changed, so it is out of the notes, but a Self-Managed operator who cannot account for the pending-changes state may read it as a problem. The v26.47 upgrade guide looks like the right venue. Snapshot drafts (with PR links) in mz-skills: - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/omitted.md) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/omitted.md) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/omitted.md) - rc.4: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/omitted.md) - rc.5: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/omitted.md) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/release_notes.md) / [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 14 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/omitted.md#borderline), rc.2: 2 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/omitted.md#borderline), rc.3: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/omitted.md#borderline), rc.4: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/omitted.md#borderline), rc.5: 1 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/omitted.md#borderline), final: 0 — [omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/omitted.md#borderline) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>

antiguru marked this pull request as ready for review July 13, 2026 16:10

antiguru requested a review from a team as a code owner July 13, 2026 16:10

antiguru requested a review from frankmcsherry July 13, 2026 21:29

antiguru commented Jul 13, 2026

View reviewed changes

Comment thread test/testdrive/fetch-tail-as-of.td Resolved

antiguru requested a review from a team as a code owner July 15, 2026 02:04

aljoscha approved these changes Oct 7, 2026

View reviewed changes

ggevay approved these changes Oct 10, 2026

View reviewed changes

antiguru and others added 4 commits October 12, 2026 17:32

compute: fix stray "empty" in is_single_time assert message

ea5ba22

The `as_of <= until` soft assertion message read "expected empty `as_of ≤ until`". The "empty" is a leftover with no meaning here. Drop it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

antiguru enabled auto-merge (squash) October 12, 2026 15:45

antiguru force-pushed the moritz/clu-169-subscribe-as-of-max-up-to-max branch from 17e6f13 to d9b42d5 Compare October 12, 2026 15:56

antiguru merged commit 38f69f6 into MaterializeInc:main Oct 12, 2026
92 checks passed

antiguru deleted the moritz/clu-169-subscribe-as-of-max-up-to-max branch October 12, 2026 18:26

bosconi mentioned this pull request Oct 16, 2026

v26.47.0 release notes #39891

Merged

Conversation

antiguru commented Jul 13, 2026 • edited by SangJunBak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aljoscha left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggevay left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

antiguru commented Jul 13, 2026 •

edited by SangJunBak

Loading