Skip to content

SS-496 add option ALL COLUMNS NULLABLE to postgres CTFS - #39680

Merged
patrickwwbutler merged 6 commits into
mainfrom
patrick/pg-nullable
Oct 14, 2026
Merged

patrickwwbutler merged 6 commits into
mainfrom
patrick/pg-nullable

Conversation

@patrickwwbutler

Copy link
Copy Markdown
Contributor

Adds a new WITH option ALL COLUMNS NULLABLE to the Postgres CREATE TABLE FROM SOURCE syntax, allowing users to specify that any NOT NULL constraints should be ignored, are droppable, and the corresponding Materialize columns are nullable.

Adds sql-parser ddl test cases, and adds to the pg-cdc/exclude-constraints.td testdrive file with cases covering this logic.

@patrickwwbutler
patrickwwbutler requested a review from a team October 8, 2026 18:13
@patrickwwbutler
patrickwwbutler requested a review from a team as a code owner October 8, 2026 18:13
@linear-code

linear-code Bot commented Oct 8, 2026

Copy link
Copy Markdown

SS-496

@def-

def- commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- ALL COLUMNS NULLABLE is silently accepted and ignored for MySQL, SQL Server, Kafka and load-generator sources

src/sql/src/pure.rs:1931

purify_create_table_from_source only passes all_columns_nullable to the Postgres branch and never rejects it for other source types, so CREATE TABLE ... FROM SOURCE <mysql_source> (...) WITH (ALL COLUMNS NULLABLE) succeeds, keeps every upstream NOT NULL, and SHOW CREATE TABLE still shows the option. A user who opted in to survive an upstream DROP NOT NULL gets the permanent table stall the option is meant to prevent, and only finds out when it happens.

Details

The sibling options are already guarded: EXCLUDE CONSTRAINTS / EXCLUDE ALL CONSTRAINTS bail with "EXCLUDE CONSTRAINTS is not supported for {} sources" for anything except Postgres and MySQL (src/sql/src/pure.rs:1972-1983). The new flag has no matching check, the MySQL / SQL Server / Kafka / load-generator branches never read it, and plan_create_table_from_source drops it (all_columns_nullable: _, src/sql/src/plan/statement/ddl.rs:1789). humanize_sql_for_show_create keeps it (src/sql/src/plan/statement/show.rs:1172), so the stored and displayed SQL claim it is in effect.

For MySQL this has a direct consequence: MySqlColumnDesc::get_incompatible_schema_change returns NotNullDropped when a stored non-nullable column becomes nullable upstream (src/mysql-util/src/desc.rs:330). The table errors permanently, and the recovery hint then says to recreate it with EXCLUDE ALL CONSTRAINTS (src/source-schema-change/src/lib.rs:98). enable_exclude_constraints_option defaults to true (src/sql/src/session/vars/definitions.rs:2276), so the option can be used without changing any flag.

Fix: reject the option for non-Postgres connections next to the existing EXCLUDE CONSTRAINTS guard (or implement it in mysql::purify_source_exports, which already handles FilterConstraints::ExcludeAll the same way), and add a negative testdrive case for a non-Postgres source.

Comment thread src/sql/src/pure/postgres.rs Resolved
Comment thread src/sql/src/pure/postgres.rs Resolved
@def-

def- commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. HIGH -- The break 'c to continue 'c change re-lands the reverted #39598 for source exports, so ingestion replicas crash-loop after upgrade

src/sql/src/plan/statement/ddl.rs:1557

Existing subsources and tables-from-source are re-planned from their stored create_sql at boot. Any export whose constraints had a nullable non-NND UNIQUE ahead of an all-NOT-NULL UNIQUE now gets an extra key. That RelationDesc no longer equals the schema registered on its persist shard, so after an upgrade the replica that ingests it panics and restarts in a loop. This is the same change #39660 reverted for exactly this crash.

Details

Persist stores the whole ProtoRelationDesc as the shard schema, keys included (src/storage-types/src/sources.rs:1147). It finds a writer's schema by equality (src/persist-client/src/internal/state.rs:1622) and returns None once a different schema is registered. Source exports are not schema-evolved at bootstrap. evolve_nullability_for_bootstrap only covers compute collections (src/adapter/src/coord.rs:3486). persist_sink opens its writer with the re-planned desc (src/storage/src/render/persist_sink.rs:534), and the append then hits assert!(..., "fetching a schema id should only fail when the shard is tombstoned") at src/persist-client/src/write.rs:746.

A common upstream shape is enough. Take CREATE TABLE s (id int PRIMARY KEY, a int UNIQUE, b int NOT NULL UNIQUE) with an existing CREATE TABLE s FROM SOURCE pg (REFERENCE s). Constraints are emitted PK first, then in OID order (src/sql/src/pure/postgres.rs:317), so keys change from [[id]] to [[id],[b]]. MySQL and SQL Server exports and legacy FOR TABLES subsources take the same path (ddl.rs:1628). In 0dt the failure only shows after promotion, because read-only sources do not append.

The new option does not need this change. Postgres always puts the PK first, so break 'c cannot hide it, and the testdrive expectations are the same either way. Fix: revert the ddl.rs hunk and keep break 'c. Fixing the dropped constraints needs a catalog migration or a per-export flag that keeps existing descs stable, as the #39660 description says.

Comment thread test/pg-cdc/exclude-constraints.td Resolved

@martykulma martykulma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have a SHOW CREATE TABLE round trip test as well.

Otherwise, lgtm!

Comment thread src/sql/src/pure.rs Resolved
@patrickwwbutler
patrickwwbutler merged commit 012cf0a into main Oct 14, 2026
82 checks passed
@patrickwwbutler
patrickwwbutler deleted the patrick/pg-nullable branch October 14, 2026 22:36
peterdukelarsen added a commit that referenced this pull request Oct 16, 2026
### Motivation

#39680 added `EXCLUDE ALL NOT NULL CONSTRAINTS` to `CREATE TABLE .. FROM
SOURCE`. The parser, option plumbing, and feature flag gate are
source-agnostic, but only the Postgres purifier applied it, so MySQL
tables silently kept their `NOT NULL` constraints.

### Description

Thread the flag through the MySQL purifier and mark every column
nullable, matching the Postgres behavior. The unsupported-source guard
in `purify_create_table_from_source` now also covers this option, so
source types without support reject it instead of ignoring it.

Columns of a retained primary key stay NOT NULL. The planner declares
them NOT NULL regardless of the purified desc, so marking them nullable
disabled the decoder's NULL check for a column persist treats as
non-nullable. MySQL schema verification compares keys by name and
columns, so a key dropped and restored while the source was behind is
not reported, and a NULL written in between panicked clusterd. Postgres
compares keys by constraint OID, so a recreated key is reported before
any such row is decoded, and its purifier is unchanged.

### Verification

Extends `test/mysql-cdc/exclude-constraints.td` with the same cases
#39680 added for Postgres: `SHOW CREATE` roundtrip, index key, column
nullability, the combination with `EXCLUDE CONSTRAINTS ('PRIMARY')`,
`DROP NOT NULL` as a non-event, and PRIMARY KEY and UNIQUE drops still
erroring with the hint. Also adds the drop-and-restore-while-paused
scenario from the review, which panicked clusterd before the second
commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <[email protected]>
@bosconi bosconi mentioned this pull request Oct 21, 2026
val-materialize added a commit that referenced this pull request Oct 21, 2026
Closes [SS-519](https://linear.app/materializeinc/issue/SS-519)

Docs for the new `EXCLUDE ALL NOT NULL CONSTRAINTS` option added in
#39680.

Merge after v26.47 ships. Behavior described here matches #39680 at
commit 56af261 and may change before merge.

Pairs with #39680.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
maheshwarip added a commit that referenced this pull request Oct 23, 2026
Release notes for the **v26.47.0** train. One commit was added per
snapshot; the `final` snapshot sets the real dates. **All six snapshots
(`rc.1`–`rc.5`, `final`) are now applied — this branch is complete and
ready for review.**

- **Content**: 2 Feature sections (`TRY_CAST`, `EXCLUDE ALL NOT NULL
CONSTRAINTS`), 7 Improvements, 13 Bug Fixes — `rc.1` contributed all of
it except the top four Bug Fixes: `rc.2` added two, and `rc.3` and
`rc.4` one each. `rc.5` added no new entries; it amended `rc.4`'s
probe-port Bug Fix instead (see below). `final` adds nothing at all —
its tracking issue holds only the unassociated `release: bump to version
v26.47.0` commit
([`bed23494d`](bed2349)),
and the `v26.47.0-rc.5` and `v26.47.0` tags are 14h28m apart with that
bump the only thing between them. Its commit is empty and exists to
record on the branch that the snapshot was applied.
- **Dates (final, no longer provisional)**: Cloud **2026-10-22**,
Self-Managed **2026-10-23**. Every RC estimated this same pair from the
`v26.47.0-rc.1` tag, and the real dates confirmed it — the five RCs
compressed into one week rather than pushing the train out — so no
published date text changed in the `final` commit. ⚠️ One thing worth a
last look before merge: **2026-10-22 is a Wednesday**, one day off the
usual Thursday Cloud date (2026-10-23 is the normal Friday Self-Managed
day). That is the supplied value and it stands, but if the release owner
expected Thursday 2026-10-23 / Friday 2026-10-24, this branch is where
to correct it.
- **Operator compatibility**: row added for `v26.47`, release date
`2026-10-23`.
- **Branch base**: `main`, whose newest section is `## v26.45.1` —
v26.46.0's section is still on its own unmerged branch (#39690). Expect
a top-of-list conflict when both merge; `v26.47.0` goes above
`v26.46.0`.

✅ **Corrections already on this branch** — a `final` verification pass
over the whole assembled section confirmed these and deliberately left
them alone:

- **`rc.1` named a syntax that does not exist.** It published "`CREATE
TABLE ... FROM SOURCE` now accepts an `ALL COLUMNS NULLABLE` option",
taking the name from
[#39680](#39680
title and body. At the `v26.47.0` tag the option is `EXCLUDE ALL NOT
NULL CONSTRAINTS`
(`TableFromSourceOptionName::ExcludeAllNotNullConstraints`), and
`NULLABLE` is not a lexer keyword at all, so the published syntax would
have been a parse error for anyone who copied it. A human editor caught
this and replaced the bullet with the
`{#v26.47-exclude-not-null-constraints}` feature section now on the
branch. The `rc.1` snapshot directory in mz-skills still carries the
wrong wording — snapshot directories are the record of what was reviewed
at the time and are left untouched.
- **`rc.5`'s cross-snapshot rewrite is applied once**, including the
"Deployments using password, SASL, or OIDC authentication have no
separate internal SQL port and continue to be probed on the external
one" clause.
- **PR citations are stripped from all entries** as a deliberate
house-style edit, matching how published sections read. PR links remain
available in the mz-skills snapshot drafts linked below.

⚠️ **Open questions for the release owner** — none of these were
resolved by `final`; this branch is the last place to act on them

- **Was any `rc.4` artifact handed to a customer or used for an upgrade
rehearsal?** This decides a published entry.
[#40036](#40036)
(published as a Bug Fix by `rc.4`) made the operator probe
`environmentd`'s internal SQL port unconditionally, which crashloops
deployments using Password, SASL, or OIDC authentication — they have no
internal SQL port.
[#40046](#40046) fixed
that roughly seven hours later, inside this same unreleased train, so no
shipped build (`v26.46.1` probes the external port unconditionally)
could have hit it. It is therefore recorded as **Borderline** rather
than published as a Bug Fix of its own, and its substance folded into
`rc.4`'s bullet as a scoping clause. If an `rc.4` build did reach
anyone, the crashloop *was* reachable and
[#40046](#40046)
should be published in its own right. Note the author wrote an explicit
positive release-note line for it, which is why this is flagged rather
than quietly dropped.
- **GA announcements — the largest judgment call in this train.** Three
docs PRs remove preview markers:
[#39831](#39831)
(source versioning, `CREATE TABLE ... FROM SOURCE`),
[#39836](#39836)
(replacement materialized views),
[#39830](#39830)
(Kafka sink headers, noted as "GA in v26.47"). None changes code, so all
three were held; drafted wording for each is in `omitted.md`. Which are
actually GA in v26.47.0 and should be announced? Note v26.46.0 published
Kafka sink message attribution as *private preview* — if the header goes
GA here, this release should say so rather than repeat the preview
framing.
- **GCS blob read throughput**
([#39906](#39906),
held in `rc.2`): restores the read throughput GCS deployments lost when
[#39351](#39351) —
already shipped — forced HTTP/2 and collapsed per-request connections
into one. It is selected by `persist_blob_http_pool`, verified to
default to `off` at the `v26.47.0-rc.2` tag, and the author notes "the
default should eventually be 'GCS-only'". **Is `persist_blob_http_pool`
being set to `gcs` for v26.47.0?** If so this is a clear win to publish
for GCP users, who carry the regression today; drafted wording is in
rc.2's `omitted.md`.
- **`SUBSCRIBE` soft assertions**
([#37611](#37611),
[#39764](#39764), both
held): drafted as Bug Fixes, then pulled back. Both failures come from
`soft_assert_or_log!`, which panics only where soft assertions are
enabled and otherwise logs; neither PR says what a release build does.
Are soft assertions on in the environments customers run? If yes, both
are publishable as drafted in `omitted.md`.
- **Console Freshness page**
([#39340](#39340),
held): a complete new Monitoring > Freshness page, gated behind
`freshness-page-CNS164`, off in both Cloud and self-managed, and the
LaunchDarkly flag is not yet created. If it is turned on for v26.47.0,
this likely deserves a Feature heading.
- **PostgreSQL sources ingesting during a snapshot**
([#39419](#39419),
[#39479](#39479),
held): adding a large table to an existing source would no longer stall
the tables already hydrated — but
`storage_persist_sink_description_lookahead` is `Duration::ZERO` at the
tag. Publish if the lookahead is being enabled.
- **Coalesced materialized-view sink batches**
([#37591](#37591),
held): measures ~67% fewer blob writes and ~54% smaller state diffs, at
~10 ms added p50 latency on small writes, but
`compute_mv_sink_shared_batch_max_bytes` is `0` at the tag. Publish
(with the latency cost stated) if the staged rollout reaches customers
in this release.
- **MCP 2026-07-28 conformance checks**
([#39694](#39694),
[#39695](#39695),
held): `enable_mcp_protocol_2026_07_28` is `false` at the tag. If the
revision is being served, these belong together as one Improvement.
- **Refresh strategy columns**
([#39597](#39597),
published as an Improvement): this renames `aligned_to` →
`aligned_to_expression` and `at` → `at_expression` and changes their
type. It is a breaking change for anyone selecting them, in a schema
documented as unstable — it may warrant more prominence than a bullet.
- **Retained input after `ALTER SINK ... SET FROM`** (from `rc.3`'s
[#37653](#37653)): the
PR records a pre-existing gap it does not fix — after `SET FROM`, the
*old* upstream relation stays held, and therefore un-compactable, until
the sink is dropped or `environmentd` restarts. It is not something this
release changed, so it is out of these notes, but it is a real footgun
for the blue/green pattern that `sql/alter-sink.md` recommends `SET
FROM` for. Should the `ALTER SINK` page or the blue/green guide call it
out?
- **Operator upgrade shows pending changes** (from `rc.4`'s
[#40036](#40036)): the
probe-port change alters the generated `environmentd` StatefulSet, so
after upgrading the operator, existing environments report pending
changes and pick up the new port on their next rollout — they are
**not** restarted on their own. That is an upgrade consequence rather
than something the release changed, so it is out of the notes, but a
Self-Managed operator who cannot account for the pending-changes state
may read it as a problem. The v26.47 upgrade guide looks like the right
venue.

Snapshot drafts (with PR links) in mz-skills:
- rc.1:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/omitted.md)
- rc.2:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/omitted.md)
- rc.3:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/omitted.md)
- rc.4:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/omitted.md)
- rc.5:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/omitted.md)
- final:
[release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/release_notes.md)
/
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/omitted.md)

**Borderline PRs omitted:** rc.1: 14 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.1/omitted.md#borderline),
rc.2: 2 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.2/omitted.md#borderline),
rc.3: 0 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.3/omitted.md#borderline),
rc.4: 0 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.4/omitted.md#borderline),
rc.5: 1 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/rc.5/omitted.md#borderline),
final: 0 —
[omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.47.0/final/omitted.md#borderline)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Valerie Chiang <[email protected]>
Co-authored-by: Pranshu Maheshwari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants