console: Ory cloud auth — Talos app passwords + auth discovery - #12

…user_objects bypass (MaterializeInc#37206)

A `restrict_to_user_objects` read deferred for the session's startup
`mz_sessions` builtin-table append resumed `sequence_plan` with
`ResolvedIds::empty()`, silently skipping the system-catalog guard. The
fix carries the planner's resolved IDs through `DeferredPlan` so the
resumed `rbac::check_plan` sees the same dependencies as the
non-deferred path. Same bug class as the SHOW fix in MaterializeInc#36543. Fixes
[SQL-383](https://linear.app/materializeinc/issue/SQL-383/mcp-restrict-to-user-objects-rbac-bypass).

…zeInc#37271)

The JSON `EXPLAIN` output in
`test/sqllogictest/transform/case_literal.slt` embeds the global ID of
table `t`.
That ID depends on how many catalog objects precede it, so it churns
whenever unrelated tests or migrations shift the allocation, producing
spurious diffs like `"User": 22` vs `"User": 1`.

This registers a `replace` directive that substitutes `"User": <num>`
with a `<ID>` placeholder in the actual output before comparison, and
updates the expected output to match.
The plan structure is still fully asserted; only the unstable ID is
masked.

…alizeInc#37275)

https://linear.app/materializeinc/issue/SS-280/thread-main-panicked-at-srcadaptersrccatalogapplyrs127533

### Motivation

consistency with MaterializeInc/cloud#12768

### Verification

testing in my personal stack

…aterializeInc#37233)

A materialized-view sink in read-only mode held a `desired - persist`
correction buffer that grows by ~one record per write the old generation
makes during the read-only window, doubling the memory of a hydrating
replica during a 0dt upgrade (contributor to INC-1095).

A read-only sink mints no batches, so the `WriteBatch` path that drives
`consolidate_before` forward never runs. The one-shot consolidation
cancels the snapshot, but every forward write adds a `desired`/`persist`
pair that is never consolidated — `advance_since` only advances times
lazily, it does not cancel. The buffer therefore accumulates and stays
resident until promotion. Re-arm the forced consolidation in read-only
mode at the current persist frontier so it keeps sweeping forward as the
frontiers advance, mirroring the read-write path. Applied to both the v2
(sync) and original (async) sinks; v2 also gets a watch task to wake the
operator on the read-only -> read-write transition.

Adds the `materialized_view_read_only_correction_pruning` 0dt scenario:
`mz_new` hydrates read-only and is never promoted while `mz_old` writes
300k wide rows. It scrapes the read-only replica's own clusterd metrics
socket (environmentd's federated `/metrics` does not surface an
unorchestrated read-only replica) and asserts both that the correction
buffer stays bounded and that `mz_new`'s anonymous memory stays within a
few hundred MiB of baseline. Verified: fails on unfixed `main` (buffer
climbs to net ~400k, anon memory ~435–715 MiB over baseline), passes
with this fix (buffer drains per batch, memory near baseline). The
pre-existing single-INSERT `materialized_view_correction_pruning`
scenario only exercises the snapshot and is left unchanged.

Closes CLU-131.

Co-authored-by: Claude Opus 4.8 <[email protected]>

MaterializeInc#37267)

This tries to fight against Claude's wall-of-text style of commenting.

…ializeInc#37256)

Auth errors were sticky on the login page across refresh and back
navigation. Clear out the url param and make the error banner
dismissable
<img width="2690" height="1402" alt="image"
src="https://github.com/user-attachments/assets/c3c34caf-3736-4f7b-97fc-534a221ce51a"
/>

Fixes
[CNS-88](https://linear.app/materializeinc/issue/CNS-88/sso-error-persisting-on-page-reload)

…c#37232)

Correctness fixes for `mz-deploy` (the declarative SQL project
deployment tool),
plus the `stage --redeploy-schema`/`--redeploy-all` flags. Split out
from the
docs/init ergonomics changes so the substantive fixes can be reviewed on
their
own.

### How to review

**Each commit is a self-contained semantic change and can be reviewed on
its
own.** Every commit is either *trivially correct* (test-only
infrastructure or a
checked-in lockfile, no production behavior change) or *provably
correct* against
the test that ships in the same commit. Reviewing commit-by-commit is
the
intended path.

### Commits

1. **run a source-table's grants/comments outside the create
transaction**
`CREATE TABLE ... FROM SOURCE` and its companion `GRANT`/`COMMENT` were
bundled
into one transaction, which the server rejects — the deploy failed.
Splits the
grants and comments out of the create transaction. *Provably correct:*
   regression test exercises a source table with a grant.

2. **store object identifiers as Idents so reserved-word names work**
Object names that are SQL reserved words (or otherwise need quoting)
were
stored as raw strings and broke end-to-end. Switches the identity type
to a
typed `Ident`, quoting only at the edges. *Provably correct:* tests
cover
   reserved-word object names.

3. **check in regenerated types.lock files**
Ships the reserved-words test project's `types.lock` alongside the other
test
fixtures (used by commit 2's tests). *Trivially correct:* checked-in
generated
   lockfile, no code change.

4. **reject indexes on tables and sources**
An index on a table/source could silently be dropped at promote when its
cluster participated in a swap. Rejects indexes on tables and sources up
front
with a clear error hinting at creating a view and indexing that.
*Provably
   correct:* test asserts the rejection and message.

5. **add `stage --redeploy-schema` and `--redeploy-all`**
Adds flags to force one or more schemas (or the whole project) to
redeploy even
when unchanged, by seeding the dirty-propagation fixed point with forced
schemas. This commit also **fixes the replacement-MV (stable schema)
dirty
propagation**: the special "don't propagate" rule for stable MVs belongs
only
on downstream dependency propagation, not on schema atomicity. The
corrected
semantics: a dirty MV in a stable schema dirties all views in that
schema; a
dirty stable schema redeploys all its MVs atomically; a dirty stable
MV's
cluster still propagates; only *downstream* fan-out from a dirty stable
MV is
suppressed. *Provably correct:* unit tests for each rule (including
inverted
   tests for the old behavior and a new downstream-skip test), plus the
`redeploy-flags` and `replacement-mvs` mzcompose workflows. The rules
were
first validated as a runnable Soufflé datalog spec before porting to
Rust.

6. **test promote crash recovery via fault injection**
Adds a test-only crash hook and an mzcompose workflow that kills the
process
at chosen points during promote to exercise the resume paths. *Trivially
   correct:* test-only infrastructure; no production behavior change.

7. **resume promote after a crash between promotion and cleanup**
A crash after a deploy was promoted but before its marker schemas were
cleaned
up left the deploy un-resumable — validation rejected the
already-promoted
state. Gates that validation on the not-yet-started state so promote
resumes
cleanly. *Provably correct:* the fault-injection test from commit 6
drives a
   crash at exactly this point and asserts a clean resume.

### Release notes

This release will fix several `mz-deploy` bugs: source-table grants
failing to
deploy, reserved-word object names being unusable, indexes on
tables/sources
being silently dropped at promote, incorrect redeploy of materialized
views in
stable-API schemas, and promote not resuming after a crash between
promotion and
cleanup. It also adds `stage --redeploy-schema` and `--redeploy-all` for
forcing
redeploys.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

Remove these sections if your commit already has a good description!

### Motivation

Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand *why* after reading this section.

If this implements or addresses an existing issue, it's enough to link
to that:
Closes <issue>
Fixes <bug>
etc.

### Description

What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what
the
diff *can't* tell a reviewer.

### Verification

How do you know this change is correct? Describe new or existing
automated
tests, or manual steps you took.

Bump the `mz-deploy` crate version from `0.1.0` to `0.2.0`.

This updates:
- `version` in `src/mz-deploy/Cargo.toml` (and the matching `Cargo.lock`
entry)
- The version called out in the user-facing docs
(`manage/mz-deploy/_index.md` and `get-started.md`), which previously
read "v0.1 release"

No functional changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…`as_of` (MaterializeInc#37263)

Hydrating Iceberg sinks currently wait until after they've read the
entire source snapshot to mint a batch description and start writing.
This means they accumulate the entire source snapshot in memory.

This PR allows Iceberg sinks to immediately mint a batch description for
the source snapshot so they can flush to the writer instead of
accumulating everything in memory.

----
_Notes:_

- This always mints a batch description for the source snapshot. If
there's no data _and_ the input stream closes, we're now making an empty
batch instead of no batches.
- This needs to run through QA Canary to see if it fixes SS-319/SS-282.

Adds E2E testing for AWS Glue schema registry with a Kafka source.

AWS Glue supports multiple registries, including a default. Schemas
exist within registries. mzcompose.py takes care of registry setup and
teardown, testdrive takes care of schema creation (glue-schema-create).

- local/test pipeline use [Moto](https://github.com/getmoto/moto) to
emulate schema registry (default workflow)
- nightly pipeline uses AWS (aws workflow)


This is not an exhaustive list of tests, we are missing
availability/connectivity tests (MZ restart, glue schema registry
unreachable, etc.). I'll draft a separate PR for those.

…ns are excluded (MaterializeInc#37224)

Adds a new error type and check during postgres `CREATE TABLE`
purification to validate that none of the excluded columns are primary
keys.


https://linear.app/materializeinc/issue/SS-197/thread-tokiowork-3-panicked-at-srcsqlsrcpurepostgresrs29422-key-exists

Customers won't pick those up unless they use `--devel`

For each Histogram metric, the Prometheus library generates three
series:
* `_bucket`, the cumulative per-bucket counters, which carry an
additional `le` label
* `_sum`, the running sum of observed values
* `_count`, the running count of observations

We weren't expanding these metrics out in our docs. This fixes that. We
use the fact that histogram metrics have `buckets` associated with them.

Based on failures in
https://buildkite.com/materialize/nightly/builds/16879

Follow-up to MaterializeInc#36959

See
https://materializeinc.slack.com/archives/C029REFQ031/p1781862364295249
for Azure

Test run: https://buildkite.com/materialize/nightly/builds/16881

### Motivation

Resolve scoped feature flags for a freshly created cluster or replica at
create
time, rather than on the next sync tick. Stacked on MaterializeInc#36959 (resolution
and
gating); the catalog-implications foundation MaterializeInc#37151 has merged.

Without this, a new object gets its overrides on the next sync tick
(≤1s). For
render-frozen replica-scoped flags (e.g. the column-paged batcher,
chosen at
arrangement-build time) that is too late: the override must reach the
controller
before the replica's dataflows render.

### Description

Evaluate a freshly created cluster or replica from its plan through the
shared
`SystemParameterFrontend`, and fold the resulting overrides as an
`Op::UpdateScopedSystemParameters` into the *same* transaction that
creates the
objects. The committed diff then drives the replica-scoped controller
push (the
implication from MaterializeInc#37151) before `create_replica`, so the new replica's
first
configuration replays with its overrides.

The override map is keyed by replica id, so the create paths
pre-allocate
replica ids out-of-band (`allocate_replica_ids`, user or system per the
owning
cluster, as of MaterializeInc#37178) and carry them in `Op::CreateClusterReplica`'s
required
`replica_id` field. This replaces the post-transact
`resolve_scoped_for_new_objects`
from MaterializeInc#36959.

`ALTER CLUSTER` paths that recreate replicas (size or availability-zone
change,
scale-up, zero-downtime reconfig) fold their replica-scoped overrides
the same
way, so a size-family change that flips a render-frozen flag reaches the
recreated replica before it renders. Scale-down and no-op alters
recreate no
replicas and fold nothing.

### Verification

The create-time cases in the `test/launchdarkly` mzcompose workflow
(introduced
in MaterializeInc#36959) exercise this path, validated against a live LaunchDarkly
project.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01FUorxx5m6B8TT2VAek2STo

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…rializeInc#37078)

Problem:
`ALTER CONNECTION ... ROTATE KEYS` is a read-modify-write with a wide
window between the catalog read and the catalog write. The slow steps in
the middle are SSH key generation and secret-store I/O, all running in a
spawned task. If `ALTER CONNECTION SET (...)` commits inside that
window, the blind catalog write at the end of ROTATE overwrites the
SET's change with the pre-SET `create_sql`. The SET silently disappears.

Note:
ROTATE KEYS is one of the few DDLs exempt from the global
`serialized_ddl` lock (see `must_serialize_ddl` in
`command_handler.rs`), which is what opens the race window in the first
place. Other ALTER CONNECTION variants take the lock and can't race
against themselves.

Solution:
`PlanValidity` already runs at every off-thread to on-thread hop and
catches dropped dependencies. Extend it to also catch content changes:
capture a hash of each dependency's `create_sql` at plan time, compare
against the live hash in `check`, and return a new
`ConcurrentDependencyMutation` error (SQLSTATE 40001) on mismatch. The
user retries the statement.

`PlanValidity::new` and `extend_dependencies` now take `&Catalog` so
they can read entries to hash; all callsites are updated.

Testing:
Regression test that triggers the wait point

Closes SQL-272.

…ND/OR in reduce" (MaterializeInc#37293)

Reverts MaterializeInc#37049

In support of CLU-137

…aterializeInc#37291)

Caused by MaterializeInc#37107

Closes SQL-420

The reason we only noticed this so late is the auto dev version
migration I added to make it easier to bump versions in staging, I'll
disable that for CI in a separate PR.

…nput (MaterializeInc#36985)

Hardens decode paths in `persist`, `persist-client`, `persist-types`,
`pgrepr`, and `postgres-util` against malformed/untrusted input: reject
rollup-less/empty-frontier/hollow-only/invalid StateDiff state instead
of panicking or debug-asserting, guard UUID parsing, fix i16 overflow in
pgrepr numeric-scale binary decode, and propagate u16-conversion errors
in the postgres table-desc protos.

Found by the cargo-fuzz suite ([separate infra
PR](MaterializeInc#36982)). Each fix
has a regression test.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…zeInc#37295)

Inspired by SQL-420, which we could have caught in tests with this

Nightly run: https://buildkite.com/materialize/nightly/builds/16883

)

OIDC auth middleware attaches the cached id_token as a Bearer header on
every API request. Envd reuses the password session cookie only when the
request carries no credentials. When the OIDC token expired, next
request still contained the stale expired token and was rejected with
the "authentication credentials have expired" error. Fixed the
MzOidcUserManager to check the id_token's expiry so it doesn't break
when envd tries validating it

[Fixes
CNS-91](<https://linear.app/materializeinc/issue/CNS-91/fix-expired-oidc-logins-causing-password-auth-to-break>)

## To repro:

* Set the ID token to 60s in the IDP
* Login using SSO
* When SSO expires, user sees this <img
src="https://uploads.linear.app/974a4381-d068-46e0-9fcd-1a1c00168131/1232c1cf-8281-43f6-84f4-1fe7bea8b52b/9369ba86-a002-4fc8-8733-ad0ee5a403af?signature=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXRoIjoiLzk3NGE0MzgxLWQwNjgtNDZlMC05ZmNkLTFhMWMwMDE2ODEzMS8xMjMyYzFjZi04MjgxLTQzZjYtODRmNC0xZmU3YmVhOGI1MmIvOTM2OWJhODYtYTAwMi00ZmM4LTg3MzMtYWQwZWU1YTQwM2FmIiwiaWF0IjoxNzgyMzE5NTk2LCJleHAiOjE4MTM4OTAxNTZ9.VJokSTNuwum5bN2wCg1Ac4tNkdeQc62AgMp3aMw5Qcs
" alt="image" width="763" data-linear-height="858" />
* Try logging in using password auth, can't log back in using password
auth

### Verification


https://github.com/user-attachments/assets/cabab8f3-8243-4163-a521-e46caae24a29

Cheaper. Works together with
MaterializeInc/i2#3428

…eric precision (MaterializeInc#37276)

### Motivation

Dennis filed an issue flagging a test failure in the iceberg sink where
we failed to output the decimal value "62912915282.360725" to an iceberg
sink where we expected precision of 3. The issue appears to be upstream,
where the data being sinked doesn't match the schema.

For parallel_workload it appears we use `COPY`.

Issue
https://linear.app/materializeinc/issue/SS-193/iceberg-failed-to-convert-row-to-recordbatch-failed-to-add-insert-row

### Description

I added some tests for `COPY` behavior that failed initially and now
pass, covering default and CSV input via STDIN and Parquet. Generally
trying to mirror existing insert behavior that ultimately calls rescale.

### Verification

I've compared the results for the basic tab-separated `COPY` command to
postgres and confirmed that postgres rounds while materialize currently
doesn't.

I've added some testdrive tests and unit tests to prevent future
regressions and explore edge cases like numbers overflowing max
precision on rescale that wouldn't have before.

Note: There are a variety of other types that may have similar issues,
i.e. timestamp max precision, which I'm thinking I'll file a separate
follow-up issue on.

###  Potential Gotchas

The fivetran sink uses the constraints from `Type::Numeric`, so would
end up creating a `DECIMAL` with no options, see
https://github.com/MaterializeInc/materialize/blob/4229f003472695da7f0fb9c8c1f2aecc3fa8e39e/src/fivetran-destination/src/utils.rs#L94.
I think DECIMAL in the SDK maps to fivetran's BIGDECIMAL and so should
be more correct by default.

…Inc#37277)

### Motivation

The webhook source request body limit was a compile-time constant (5
MiB) applied to all HTTP routes, so operators could not adjust it
without a rebuild.

### Description

Adds the `webhook_max_request_size_bytes` dyncfg (default 5 MiB,
preserving prior behavior). It is read from the live shared persist
`ConfigSet`, so there is no coordinator round-trip on the webhook hot
path. The webhook route disables the global `DefaultBodyLimit` and the
handler enforces the cap per request on the decompressed body via
`to_bytes`: size-exceeded returns 413, other body-read errors return
500. Enforcement is scoped to the webhook route only; SQL and other HTTP
routes keep the static 5 MiB limit.

### Verification

New integration test `webhook_max_request_size` in
`src/environmentd/tests/server.rs`: the 5 MiB default rejects a 6 MiB
body, raising the dyncfg accepts it, and lowering it below the default
re-rejects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>

### Motivation

The size limits the in-memory `Row` encoding imposes are spread across
unrelated constants (array dimensions, char/varchar length, numeric
precision, regex size, the length-tiered string/bytes/list tags) with no
single reference.

### Description

Adds `doc/developer/row-encoding.md` collecting them, each pointing at
its enforcing constant and noting whether overflow is a graceful error,
a panic, or unbounded. Covers per-datum limits and whole-`Row` bounds:
arrangements store rows with `u64` offsets, while persist's Arrow
encoding uses `i32` offsets, capping an encoded row at ~2 GiB. Links the
doc from the developer docs index and adds a module doc on `row.rs`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…Inc#37294)

### Motivation

A replica creates its `ComputeState` when it handles `CreateInstance`,
seeding `worker_config` with dyncfg defaults. `handle_create_instance`
then immediately runs `apply_worker_config`, but the controller's synced
configuration only arrives one command later, in the first
`UpdateConfiguration`. Both the live command stream (`Instance::run`
sends `Hello` then `CreateInstance` before the first
`update_configuration`) and the reconciliation order
(`ComputeCommandHistory::reduce` places the single `UpdateConfiguration`
after `CreateInstance`) put configuration after creation. Reordering is
not viable either: there is no `ComputeState`/`worker_config` to apply
configuration to before `CreateInstance`.

As a result, every create-time read that depends on configuration
silently used defaults: lgalloc, the pager, the column-paged batcher,
the memory limiter, `ore_overflowing_behavior`, and the introspection
dataflows rendered during `CreateInstance`. This is also why
create-time-frozen scoped flags from MaterializeInc#37158 had to be worked around.

### Description

Carry the configuration with `CreateInstance`. `InstanceConfig` gains an
`initial_config` snapshot of the controller's dyncfg with the replica's
scoped overrides applied on top. The controller builds it in
`Instance::specialize_command_for_replica`, the same layer that already
injects scoped overrides into `UpdateConfiguration`, and it has the live
`dyncfg` and override map there. It is evaluated fresh on every send and
on every `add_replica` replay, so the snapshot always reflects current
values. The replica applies it to `worker_config` before
`apply_worker_config`.

The snapshot is excluded from `InstanceConfig::compatible_with`, like
dictionary compression, so reconnecting to a running replica does not
halt on configuration that has changed since creation. An empty snapshot
leaves the worker at its defaults until the first `UpdateConfiguration`,
preserving prior behavior.

Soundness:
* Fresh startup or replica restart: reconcile applies `CreateInstance`,
seeding `worker_config` from the snapshot before create-time setup.
* `environmentd` reconnect to a running replica: reconcile only
compatibility-checks `CreateInstance`; the existing already-synced
`worker_config` is retained and the replayed `UpdateConfiguration` keeps
it current.
* `UpdateConfiguration` still applies globally and remains hoistable;
the snapshot is a redundant create-time seed of the same values, not a
new ordering dependency.

### Tests

Adds unit tests in the compute controller covering the create-time
snapshot and the scoped-override merge, and confirming the existing
`UpdateConfiguration` override merge is unchanged. Adds a
`clusterd-test-driver` scenario and a parse test exercising the
create-time snapshot plumbing end to end.

### Checklist

* [ ] This PR has adequate test coverage / or no new functionality
requires testing.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

<!--
Describe the contents of the PR briefly but completely.

If you write detailed commit messages, it is acceptable to copy/paste
them
here, or write "see commit messages for details." If there is only one
commit
in the PR, GitHub will have already added its commit message above.
-->

### Motivation
In this PR we initially validated whether DNS caching improved
performance or reliability on systems where node local dns was enabled.
It did not. Rather than adding caching this PR simply fixes some issues
and does some code cleanup on the SNI path for pgwire.

<!--
Which of the following best describes the motivation behind this PR?

  * This PR fixes a recognized bug.

    [Ensure issue is linked somewhere.]

  * This PR adds a known-desirable feature.

    [Ensure issue is linked somewhere.]

  * This PR fixes a previously unreported bug.

    [Describe the bug in detail, as if you were filing a bug report.]

  * This PR adds a feature that has not yet been specified.

[Write a brief specification for the feature, including justification
for its inclusion in Materialize, as if you were writing the original
     feature specification.]

   * This PR refactors existing code.

[Describe what was wrong with the existing code, if it is not obvious.]
-->

### Tips for reviewer

<!--
Leave some tips for your reviewer, like:

    * The diff is much smaller if viewed with whitespace hidden.
    * [Some function/module/file] deserves extra attention.
* [Some function/module/file] is pure code movement and only needs a
skim.

Delete this section if no tips.
-->

### Checklist

- [ ] This PR has adequate test coverage / QA involvement has been duly
considered. ([trigger-ci for additional test/nightly
runs](https://trigger-ci.dev.materialize.com/))
- [ ] This PR has an associated up-to-date [design
doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md),
is a design doc
([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)),
or is sufficiently small to not require a design.
  <!-- Reference the design in the description. -->
- [ ] If this PR evolves [an existing `$T ⇔ Proto$T`
mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md)
(possibly in a backwards-incompatible way), then it is tagged with a
`T-proto` label.
- [ ] If this PR will require changes to cloud orchestration or tests,
there is a companion cloud PR to account for those changes that is
tagged with the release-blocker label
([example](MaterializeInc/cloud#5021)).
<!-- Ask in #team-cloud on Slack if you need help preparing the cloud
PR. -->
- [ ] If this PR includes major [user-facing behavior
changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note),
I have pinged the relevant PM to schedule a changelog post.

---------

Co-authored-by: Claude Fable 5 <[email protected]>

Based on https://buildkite.com/materialize/nightly/builds/17120

@sjwiesman Noticed that mz-deploy has become the slowest test in test
pipeline. Since I think most don't touch anything that breaks mz-deploy,
I've moved it to Nightly instead of parallelizing it.

@antiguru I assume the column paged batcher testing in Staging is
intentional.

Test run: https://buildkite.com/materialize/nightly/builds/17131

…izeInc#37551)

Replay: commit MySQL initial-data and continuous-ingestion writes
(autocommit=False was silently rolling them back), actually start the
webhook continuous-ingestion thread, cancel SUBSCRIBE via a watchdog
instead of the no-op SET LOCAL statement_timeout, and anchor the webhook
CHECK-strip regex to end-of-string so it also matches anonymized
captures. Stop injecting the captured default SQL expression into COPY
data and Avro/webhook values, and stop SQL-quoting Kafka text. Reset all
state between files in workflow_default, and tear down polaris/minio/
azurite in the benchmark comparison.

Capture: sample in a subquery for --avg-column-size, always pin the
captured-workloads commit (fresh clones included) and strip the token
from .git/config, aggregate source statistics across replicas and drop
the ~60s-per-idle-source wait, parse search_path as a proper list, guard
the cancel-timer race, and remove stray time.time() calls.

Test runs:
https://buildkite.com/materialize/release-qualification/builds/1304 &
https://buildkite.com/materialize/nightly/builds/17119

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…rializeInc#37563)

Closes: SQL-506

…ead_privileges/item_dependents (MaterializeInc#37556)

By not recursing.

Motivated by #incident-1124

Closes: https://linear.app/materializeinc/issue/SQL-507

…QL (MaterializeInc#37553)

Adds Azure SQL tests to the nightly pipeline. Skipped by default because
the infra is not in place yet.

These are a basic set of tests that look to exercise the areas where SQL
Server and Azure SQL differ.  It is possible to run it manually as a local
test by setting up an Azure SQL instance and setting the appropriate
env variables.

Replacement stack for MaterializeInc#36982. This is part 3 of 10.

Summary:
- Adds sql-parser cargo-fuzz crate.
- Adds the grammar target and the SQL parse, print, and reparse
round-trip target.
- Adds the grammar source used by the structured generator.

Review notes:
- Focus review on the SQL round-trip oracle and the generated statement
families.
- This PR intentionally contains only sql-parser fuzzing code.

Verification:
- Split from original commit 082e197 and reapplied onto current
upstream/main.
- Top of stack has the same touched-file set as MaterializeInc#36982 and passes git
diff --check.

Stack:
1. MaterializeInc#37381 tests: Add shared cargo-fuzz runner
2. MaterializeInc#37382 tests: Expose fuzz-only internals
3. MaterializeInc#37383 sql-parser: Add cargo-fuzz targets
4. MaterializeInc#37384 expr, transform: Add cargo-fuzz targets
5. MaterializeInc#37385 repr, pgwire: Add cargo-fuzz targets
6. MaterializeInc#37386 avro, interchange: Add cargo-fuzz targets
7. MaterializeInc#37387 storage-types: Add proto cargo-fuzz targets
8. MaterializeInc#37388 persist-client: Add cargo-fuzz targets
9. MaterializeInc#37389 storage: Add upsert cargo-fuzz targets
10. MaterializeInc#37390 ci: Run cargo-fuzz in release qualification

The original monolithic PR is MaterializeInc#36982.

…terializeInc#37569)

Seen failing in https://buildkite.com/materialize/spec-sheet/builds/21

Test run: https://buildkite.com/materialize/spec-sheet/builds/22

Reverts MaterializeInc#33955

Replacement stack for MaterializeInc#36982. This is part 7 of 10.

Summary:
- Adds proto round-trip fuzz targets for catalog-protos and
storage-types.
- Adds external table-desc proto round-trip targets for MySQL,
PostgreSQL, and SQL Server utilities.
- Adds a CSV decode target in storage-types.

Review notes:
- Focus review on round-trip invariants and decode error handling for
serialized metadata.

Verification:
- Split from original commit 082e197 and reapplied onto current
upstream/main.
- Top of stack has the same touched-file set as MaterializeInc#36982 and passes git
diff --check.

Stack:
1. MaterializeInc#37381 tests: Add shared cargo-fuzz runner
2. MaterializeInc#37382 tests: Expose fuzz-only internals
3. MaterializeInc#37383 sql-parser: Add cargo-fuzz targets
4. MaterializeInc#37384 expr, transform: Add cargo-fuzz targets
5. MaterializeInc#37385 repr, pgwire: Add cargo-fuzz targets
6. MaterializeInc#37386 avro, interchange: Add cargo-fuzz targets
7. MaterializeInc#37387 storage-types: Add proto cargo-fuzz targets
8. MaterializeInc#37388 persist-client: Add cargo-fuzz targets
9. MaterializeInc#37389 storage: Add upsert cargo-fuzz targets
10. MaterializeInc#37390 ci: Run cargo-fuzz in release qualification

The original monolithic PR is MaterializeInc#36982.

Replacement stack for MaterializeInc#36982. This is part 9 of 10.

Summary:
- Adds storage cargo-fuzz crate.
- Adds upsert consolidation, runtime, state consolidation, and value
round-trip targets.

Review notes:
- The targets use the fuzz-only upsert hook from part 2.
- Focus review on the modeled upsert state machine and the generated
command sequences.

Verification:
- Split from original commit 082e197 and reapplied onto current
upstream/main.
- Top of stack has the same touched-file set as MaterializeInc#36982 and passes git
diff --check.

Stack:
1. MaterializeInc#37381 tests: Add shared cargo-fuzz runner
2. MaterializeInc#37382 tests: Expose fuzz-only internals
3. MaterializeInc#37383 sql-parser: Add cargo-fuzz targets
4. MaterializeInc#37384 expr, transform: Add cargo-fuzz targets
5. MaterializeInc#37385 repr, pgwire: Add cargo-fuzz targets
6. MaterializeInc#37386 avro, interchange: Add cargo-fuzz targets
7. MaterializeInc#37387 storage-types: Add proto cargo-fuzz targets
8. MaterializeInc#37388 persist-client: Add cargo-fuzz targets
9. MaterializeInc#37389 storage: Add upsert cargo-fuzz targets
10. MaterializeInc#37390 ci: Run cargo-fuzz in release qualification

The original monolithic PR is MaterializeInc#36982.

…erializeInc#37557)

### Motivation

The MCP developer endpoint's `query` tool now accepts an optional
`cluster_replica` parameter, making `EXPLAIN ANALYZE` usable on clusters
with
more than one replica. Previously the tool could target a cluster but
not a
replica, and since introspection reads must target a specific replica on
multi-replica clusters, `EXPLAIN ANALYZE` through the MCP server failed
on
exactly the clusters where it is most interesting. Hit in practice while
debugging with the developer endpoint (QAR-136).

### Description

- `QueryParams` gains `cluster_replica: Option<String>`.
- `execute_query` emits `SET CLUSTER_REPLICA = <replica>` after the
existing
`SET CLUSTER` inside the same `BEGIN READ ONLY; ...; COMMIT;` frame, so
the
replica pin is scoped to the single read and cannot leak into the
session.
  The replica name passes through `escaped_string_literal`, the same
injection-safety treatment the cluster name already gets. The `SET`
clause
assembly is extracted into a `query_set_clause` helper for testability.
- The developer tool's JSON schema advertises the new optional property,
and
  the user docs' parameter table for the `query` tool
  (`integrations/mcp-server/mcp-developer-tools.md`) documents it.
- The **agent** endpoint's `query` tool is deliberately unchanged: its
dispatch
  arm passes `None`, so a `cluster_replica` field supplied to the agent
endpoint is dropped rather than honored (replica pinning is not part of
the
agent surface). `QueryParams` is shared by both endpoints, which is why
the
  scoping lives in the dispatch arms.

An invalid replica name fails with the engine's normal unknown-replica
error,
surfaced as a clean `ExecutionError`. An untargeted `EXPLAIN ANALYZE` on
a
multi-replica cluster keeps its existing error, which names the remedy.

### Verification

- New end-to-end case in `test/mcp/mzcompose.py`
  (`developer_query_cluster_replica_pinning`): on a 2-replica cluster,
  untargeted `EXPLAIN ANALYZE CLUSTER MEMORY` fails with an error naming
replica targeting, the same call pinned to `r1` succeeds, and a
nonexistent
  replica surfaces a clean `ExecutionError`.
- New unit tests for `query_set_clause`: replica present, replica
absent, and
  an adversarial replica name requiring escaping (mirrors the existing
  cluster-name escaping test).
- Existing `http::mcp` unit tests pass locally (`cargo test -p
  mz-environmentd --lib -- http::mcp`).

Fixes QAR-136.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <[email protected]>

For included items with PR links see:
https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.32.0/release_notes.md

For omitted items with PR links see:
https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.32.0/omitted.md

**Borderline PRs omitted: 10** — [review borderline
calls](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.32.0/omitted.md#borderline)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>

…ma (MaterializeInc#37516)

Adds `publish_glue_schema to storage-client`, the Glue-backed
counterpart to CSR's publish_kafka_schema.

A key correctness concern is around Glue's asynchronous version
validation. A freshly registered version comes back `PENDING` and only
later transitions to `AVAILABLE` or `FAILURE`. The published schema
version must be `AVAILABLE` before MZ publishes the messages to ensure
downstream consumers can decode the messages.

For publish:
- Reuse the schema definition match to avoid duplicate versions on sink
restart, but skip matches where version is `FAILURE`/`DELETING` (Glue
matches by definition regardless of lifecycle status).
`FAILURE`/`DELETING` re-registers the schema instead.
- Create the schema on first publish and set compatibility (defaulting
to Glue's `BACKWARD`). Handles the create/register race with another
writer by retrying the register on `AlreadyExists`.
- Never overwrite an existing schema's compatibility. The compatibility
is set once at creation. Producers warn (best-effort) when the existing
policy differs from the intended one.
- Poll `get_schema_version_by_id` until the version is `AVAILABLE`,
erroring out on `FAILURE`/`DELETING` or if status fails to resolve
within 30s.


This also needed some glue client API changes
(aws-glue-schema-registry):

- The write-path methods now return a new `RegisteredSchemaVersion { id,
lifecycle_status }` instead of a bare `Uuid`, so callers can gate on
lifecycle status.
- Added a `test-util` feature exposing `Client::from_sdk_client` for
injecting a mocked SDK client (easiest way to test the async lifecycle
of schema version status).

MaterializeInc#37565)

### Motivation

`StateWatchNotifier` fired a single `broadcast::Sender<SeqNo>` on every
strict seqno advance, and `Machine::wait_for_upper_past` (the wait
behind `Listen`, `Subscribe`, and snapshots) armed on it.
A shard's seqno bumps for many non-data reasons: GC, rollups,
since-downgrades including each reader's own lease heartbeat, and other
writers' compare-and-appends.
One notifier per shard fans out to every reader on that shard, so each
such bump woke every upper waiter, which re-read the unchanged upper,
emitted nothing, and re-armed.
On compute replicas this unparks the timely workers hundreds of times a
second doing no-op work, which pins a per-cluster idle CPU floor that is
flat across replica size.

### Change

Split the notifier into two channels.
The existing seqno-gated channel is still used by the lease/since
machinery via `wait_for_seqno_ge`.
A new upper-gated channel fires only when the shard upper strictly
advances, and `wait_for_upper_past` routes to a new
`wait_for_upper_seqno_ge` on it.
GC, rollup, and since-downgrade churn therefore no longer re-activates
upper waiters.

Gating happens at the single write choke point,
`LockingTypedState::write_lock`, which covers both local
compare-and-appends and pubsub `push_diff` from other processes.
It compares the shard upper before and after the mutation and reports
whether it advanced.
Subscription happens before reading the seqno in `StateWatch::new`, and
every upper advance goes through this choke point, so a waiter cannot
miss a wakeup, including advances learned via pubsub.
The existing backoff-sleep arm remains as the fallback for the
pubsub-down case.

Adds the `mz_persist_watch_notify_upper_sent` internal metric to observe
the upper channel.

### Tests

Adds a `mz-persist-client` unit test that asserts a seqno bump which
does not advance the upper (a since-downgrade) does not wake an upper
waiter, while an upper advance does.
The existing `state_watch`, `state_watch_listen_snapshot`, and
`seqno_leases` tests continue to encode the seqno-channel and liveness
behavior.

Draft: pending behavioral verification on a loaded cluster (expected
drop in the tokio-main `remote_schedule` rate and the no-op timely
wakeup rate on an idle/frozen-input cluster) and a persist owner review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…eInc#37542)

### Motivation

Fixes SS-329.

Upsert sources on diskless replicas keep their working state in a
RocksDB
running in a process-shared in-memory `Env`. When an individual source
dataflow restarts in place (health-triggered `SuspendAndRestart`, not a
replica restart), the state at the instance path is supposed to be wiped
and
rebuilt from persist. On diskless replicas it never was, so the
restarted
dataflow inherited state from the previous incarnation. Rehydration then
merged the persist snapshot on top of the leftover values, tripping the
`invalid upsert state: non 0/1 diff_sum` panic and crashing the whole
storage process.

### Description

`DB::destroy` resolves paths through the `Env` carried by the options
passed to it. Both cleanup sites in `mz_rocksdb` passed
`RocksDBOptions::default()`, which only ever touches the host filesystem
and never sees state written through the in-memory `Env`. The startup
cleanup in `rocksdb_core_loop` was additionally gated on
`instance_path.exists()`, a host-filesystem check that says nothing
about
in-memory state.

The fix:

* Destroy with options that carry the instance `Env` in both cleanup
sites
  (new `destroy_options` helper).
* Run the startup cleanup unconditionally instead of gating on the host
  path existing. A host-filesystem destroy errors when the directory is
  missing, so create the directory first (rust-rocksdb's `DB::open` does
  the same). Destroying an empty directory succeeds.

Disk-backed replicas are unaffected in practice: their instance `Env` is
the default one, so destroy already targeted the right filesystem there.

### Verification

Two regression tests, adapted from the reproducers in SS-329:

* `mem_env_cleanup_on_reopen` (`src/rocksdb/tests/basic.rs`): write a
key, close, reopen the same path in the same in-memory `Env`, assert the
  second instance starts empty. Fails without the fix (the stale value
  survives) and passes with it.
* `upsert::rocksdb::tests::stale_state_corrupts_rehydration`
  (`src/storage/src/upsert/rocksdb.rs`): reproduces the full panic path
  through `UpsertState::consolidate_chunk` and `ensure_decoded` with a
value that survives an in-process restart. This is the test whose panic
is
  quoted in SS-329, so it is known to fail without the fix.

Generated with [Claude Code](https://claude.com/claude-code)

…Inc#37237)

## Summary

The soft-assert at the tail of `gc_and_truncate` (\"earliest state
fetched during GC did not have corresponding rollup\") checks that
`states.state()` references a rollup for the earliest fetched seqno.
That premise holds only when `states` was built from
`fetch_all_live_states`, where iterating to the end leaves
`states.state()` at the genuine current state.

Under active GC (`persist_gc_use_active_gc`, on by default in CI for
versions after v0.143.0-dev) GC instead fetches diffs only through
`seqno_since`, so `states.state()` is the reconstructed state at
`seqno_since`, **not** the current state. A rollup's entry is inserted
into the rollups map by the `add_rollup` transition at a seqno strictly
greater than the seqno it rolls up. When `seqno_since` precedes that
insertion, the reconstructed state legitimately lacks its own rollup
entry and the assert false-fires.

The check is a `soft_assert_or_log!`, so production only logs to Sentry
— the impact is a recurring **test blocker** (Antithesis, nightly, Long
Workload Replay), not a customer-facing crash. The PER-16 `add_rollup`
fix (MaterializeInc#36740) addressed an adjacent but distinct invariant and did not
stop this.

## Fix

Gate the check to the classic path, where it is sound. The active-GC
path already validates the same invariant against fresh data when it
resolves the rollup for the earliest seqno (the early-returns at the top
of that branch), so the tripwire is preserved on the classic path for
any genuine regression.

## Tests

Adds the datadriven regression test
`tests/machine/regression_gc_active_rollup_assert`, which reproduces the
panic under active GC on the unfixed code and passes with the fix.

Fixes database-issues#10092, PER-10

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Part of the Frontegg → Ory cloud auth migration
(cloud/doc/design/20260710_ory_auth_migration.md). For sessions whose
access token was issued by Ory Network, the app-passwords page manages
Ory Talos API keys through the new cloud global API endpoints
(GET/POST /api/app-passwords, DELETE /api/app-passwords/{key_id})
instead of the Frontegg api-tokens API.

- cloudGlobalApi: discoverAuthProvider (POST /api/auth/discovery, for
  email-first login routing driven by the ory-auth-enabled LaunchDarkly
  flag) and app-password CRUD.
- global-api schema: hand-spliced only the new endpoints' types; a full
  regen from cloud main pulls in unrelated spec drift (e.g. Regions →
  Paginated_Region) that breaks existing call sites and belongs in its
  own change.
- sessionAuthProvider: derives frontegg|ory from the session token's
  iss claim.
- OryAppPasswordsPage: create/list/revoke UI for Talos-backed app
  passwords (single-secret model - no clientId/secret pair, secret is
  shown once at creation).

Not yet wired: the Ory login flow itself (email-first discovery routing
to Ory instead of Frontegg); this PR covers the app-password surface so
it can be exercised against a personal cloud stack with an Ory-backed
org.

Co-Authored-By: Claude Fable 5 <[email protected]>

Covers issuer extraction from malformed tokens, Ory project issuer
detection, lookalike-domain rejection (suffix match on the registrable
hostname, not substring), and the fail-safe Frontegg default.

Co-Authored-By: Claude Fable 5 <[email protected]>