Repository navigation
adapter: refuse to drop an index that other dataflows read from - #39098
Conversation
QA LLM Review1. MEDIUM -- Console "Drop index" now silently drops every materialized view, index and sink that reads the index
The Console's delete flow always appends Details
The dialog's count comes from Example: Suggested fix: drop |
778c468 to
5d86e08
Compare
|
Based on conversations in the SQL sync, the |
8b64c6d to
9fe8c32
Compare
There was a problem hiding this comment.
Looks good, thank you!
A few asks inline, and the title and description lag the code: the title still says the drop requires CASCADE, and the description says CASCADE drops the index but not the MV, while both now fail the same way (drop_index_dependents.slt:51-53). The description also names test_drop_in_use_index_requires_cascade (it is test_drop_in_use_index_is_refused) and says hydration-status.td creates the view before the index (it drops and recreates the MV).
| an existing index. It fails with the in-use error if another object reads from | ||
| that index. | ||
|
|
||
| The Console's drop dialog issues `DROP INDEX ... CASCADE`. Since `CASCADE` still |
There was a problem hiding this comment.
The Terraform provider is affected too. On this PR's CI image with provider v0.11.10, changing the columns of a materialize_index that an MV reads now fails: the provider replaces an index by running DROP INDEX ... RESTRICT first, which returns the new 2BP01 error, and a re-run fails the same way. terraform destroy of the same config failed in 3 of 6 runs, because the provider drops the MV and the index in parallel; a re-run then succeeds. Worth a line here, and a heads-up to the provider owners before this reaches cloud: destroy only needs a retry, but replacing an index that an MV reads also needs the MV replaced, which the provider docs should explain.
There was a problem hiding this comment.
(Gábor: This might need a bit more discussion, which is why I not made this an approving review yet. But I don't know the Terraform stuff.)
There was a problem hiding this comment.
Tested on this PR's CI image with provider v0.11.10: adding lifecycle { replace_triggered_by = [materialize_index.t_idx.id] } to the MV makes Terraform drop the MV, replace the index and recreate the MV, and terraform destroy then drops the MV first. Referencing .id rather than the whole resource avoids rebuilding the MV on in-place index changes such as a comment. A plain depends_on fixes the destroy order but not the replacement. Users can add it after hitting the error, which names the MVs to update; re-running the apply then succeeds.
DROP INDEX ... CASCADE when there are dependents|
Okay, everything is addressed except for the terraform issue---I've asked Justin for someone from cloud to take a look. |
|
The Terraform provider side is ready for this:
I tested both against this PR's image. We'll keep them as drafts until this merges. |
bosconi
left a comment
There was a problem hiding this comment.
Looks good apart from the rollout story and two coverage gaps; details inline.
- The restriction is on everywhere at release, and Cloud users cannot turn it off because
enable_drop_in_use_indexneedsmz_system. The description, which feeds the release notes, should say that Terraform provider versions up to v0.11.10 fail with 2BP01 when replacing amaterialize_indexthat an MV reads, give thelifecycle { replace_triggered_by = [materialize_index.<name>.id] }workaround, and say whether the flag will be set via LaunchDarkly for any environments at release.
| > SELECT * FROM mv; | ||
| 9 | ||
| > DROP INDEX t_idx | ||
| > DROP INDEX drop_idx |
There was a problem hiding this comment.
This drops coverage for the first 0dt upgrade from a pre-PR release, where the old leader can still orphan t_idx (read by mv) and the new generation has to follow. Keep DROP INDEX t_idx and set enable_drop_in_use_index = true on mz_old as mz_system just before it. Same at L2639 in workflow_ddl.
| > CREATE MATERIALIZED VIEW drop_index_view AS SELECT f1, f2 FROM drop_index_table WHERE f1 > 0; | ||
| """)) | ||
|
|
||
| # An index cannot be dropped while a materialized view or a newer index reads |
There was a problem hiding this comment.
No platform check now covers the refusal across an upgrade or restart, where it depends on the dataflows re-planned at bootstrap. Please add one whose initialize creates an index and an MV over the indexed relation and whose validate asserts ! DROP INDEX ... / contains:still depended upon by, with a base-version gate if any scenario validates on a pre-PR build.
It is no longer possible to drop an index that non-transient dataflows read from. Index imports are an optimizer choice recorded only in the compute controller, not a catalog dependency, so the planner's RESTRICT check and CASCADE expansion never saw them and DROP INDEX left the importers running on an orphaned arrangement, which pinned LIR plans cannot survive across a restart. The new enable_unsafe_drop_index flag (default off) restores the old orphaning behavior for plain DROP INDEX, with a WARNING notice, an error-level log event, and a metric increment, while in-flight SUBSCRIBE, COPY TO, and SELECT dataflows never block a drop and are reported in a NOTICE instead. Notably, the behavior of `DROP INDEX ... CASCADE` is unchanged: running `DROP INDEX ... CASCADE` on an in-use index will fail rather than cascading to other objects. (Instead, users should manually drop those objects and then drop the index, or do a blue/green swap.) Co-Authored-By: Claude <[email protected]>
cdf85a5 to
dc5ac59
Compare
Co-authored-by: Jon Currey <[email protected]>
Co-authored-by: Jon Currey <[email protected]>
Co-authored-by: Jon Currey <[email protected]>
- 0dt: restore the flagged DROP INDEX t_idx on mz_old, so mz_new still follows an old leader that orphaned an index. - Add a DropIndexInUse platform check: DROP INDEX is refused across restarts and upgrades. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Thanks, @bosconi, for the review, and @bobbyiliev for the corresponding cloud PRs (which should be good to go now)! |
Adds `enable_drop_in_use_index`, introduced in MaterializeInc#39098 without regenerating `src/sql/system-parameters.json`, which fails `check-no-diff.sh` in lint for every branch based on current main. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ndex (#40323) ### Motivation `main` fails `check-no-diff.sh` on `src/sql/system-parameters.json`. #39098 added the `enable_drop_in_use_index` flag and #40217 added the generated catalog. Each merged against a `main` that did not contain the other, so neither build caught the missing entry. ### Description Output of `bin/gen-system-parameter-catalog` on current `main`. No code changes. ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <[email protected]>
…doc (#40475) Follow-up to #39098: the Terraform paragraph it added to the design doc has two errors. I've corrected the same paragraph in #39098's description, since the release notes are generated from it. - **Affected versions:** it said "up to v0.11.10". Every released provider version fails the replacement: v0.11.11 and v0.11.12 fail it the same way, and v0.11.13 has no index changes. - **What fixes it:** it said that [terraform-provider-materialize#952](MaterializeInc/terraform-provider-materialize#952) and [#953](MaterializeInc/terraform-provider-materialize#953) fix the provider, and that `replace_triggered_by` is the workaround until users upgrade. With both, replacing an index that a materialized view reads still fails, after a minute of retries. [#952](MaterializeInc/terraform-provider-materialize#952) only helps when the readers are dropped in the same run, such as in `terraform destroy`. `replace_triggered_by` is the way to replace such an index with any provider version, and [#953](MaterializeInc/terraform-provider-materialize#953) documents it. Verified against #39098's CI image with provider v0.11.11, v0.11.12 and a build of [terraform-provider-materialize#953](MaterializeInc/terraform-provider-materialize#953), which includes [#952](MaterializeInc/terraform-provider-materialize#952). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
DROP INDEX now fails while a newer index or a materialized view on the same table reads from the index (#39098). The test dropped its oldest index first, so it failed on builds with that change. Drop the materialized view first and the indexes newest first, and put the kept index on its own table so it doesn't read from the dropped ones.
Dates in RC snapshots are provisional until the release ships. One commit is added per snapshot; the final snapshot sets the real dates. ##⚠️ Needs the release owner before this can merge **1. This PR conflicts with `main`, and the conflict is substantive.** While this PR sat open, [#40201](#40201) ("docs: describe which SQL Server constraint Materialize records as a key") merged to `main` on 2026-11-05 and independently added its own `## v26.49.0` section — an `### Advisory {#v26.49-advisory}` plus an `### Improvements {#v26.49-improvements}` bullet. Both sides edit the same region of `_index.md`, so `mergeable` is `CONFLICTING`. Two problems ride along with the conflict: - **The same PR is described twice.** [#40201](#40201) documents the behavior of [#40131](#40131), which `rc.1` already published here as a Bug Fix ("Reduced the risk of new SQL Server source tables returning incorrect results after a unique constraint or primary key is dropped upstream…"). *Recommendation: keep `#40201`'s Advisory and Improvement and drop the `rc.1` bug-fix bullet* — the hand-written pair is more complete, carries the upgrade-impact guidance, and links `/ingest-data/sql-server/#unique-keys`. - **Anchor collision.** `main`'s `## v26.49.0` carries `{#v26.49-advisory}` and `{#v26.49-improvements}`; this branch's combined section carries `{#v26.49-improvements}`, `{#v26.49-agent-skills}`, and `{#v26.49-bug-fixes}`. Merging both defines `#v26.49-improvements` twice on one page. *Recommendation: rename this branch's three anchors to `{#v26.49.1-improvements}`, `{#v26.49.1-agent-skills}`, `{#v26.49.1-bug-fixes}`* — which is what the `v26.49.1` snapshot's review notes intended — *and leave `{#v26.49-advisory}` and `{#v26.49-improvements}` to `#40201`'s section.* `#40201`'s review links already point at `/releases/#v26.49-advisory`, and the compatibility YAML's `/releases/#v26491` target is unaffected. **2. Wording drift against the merged `v26.49.1` snapshot.** Commit `39e6b854a` landed here on 2026-11-05, before [MaterializeInc/mz-skills#355](MaterializeInc/mz-skills#355) merged on 2026-11-06. Two entries read differently on the branch than in the merged snapshot — the hydration improvement is "Investigate hydration on the cluster details page" here versus "See how a cluster hydrated, and why it did not" there, and the Freshness and Disk-column bullets differ likewise. Resolving the conflict above is the natural moment to decide which wording ships. ## Note on the section heading The `v26.49.1` patch train reused this branch (commit `39e6b854a`) rather than opening its own, so this PR now publishes a single combined `## v26.49.1` section — this train's `rc.1`–`rc.3` entries plus the three `v26.49.1` entries, with the Cloud version in parentheses. That is the house convention for a patch that ships in its parent's rollout week; compare the published `## v26.48.2` (`*Released to Materialize Cloud: 2026-10-29 (v26.48.0)*`). The compatibility YAML accordingly carries one `v26.49.1` row dated 2026-11-06, which is what actually shipped to Self-Managed that day. The `final` commit for this train is therefore **empty**: its increment carries no entries ([MaterializeInc/release#635](MaterializeInc/release#635) states "No PRs in `v26.49.0` require review", `v26.49.0-rc.3...v26.49.0` is the lone version-bump commit, and the agent-skills window is empty), and the real dates — Cloud **2026-11-05**, Self-Managed **2026-11-06** — were already set by `39e6b854a`. It exists so the branch's commit-subject ledger records that the snapshot was applied. ## Decisions required (carried over from `rc.1`) - **`DROP INDEX`** ([#39098](#39098)): it now rejects drops that used to succeed. Call it out as a breaking change rather than an Improvement? - **Console on large clusters** ([#40306](#40306)): the Cloud Console deploys separately, so this may already be live. Keep it in v26.49? - **Maintained Objects page** ([#40105](#40105)): omitted because it is behind a LaunchDarkly flag. Is it generally available? - **`mz profile init` OAuth** ([#40319](#40319)): omitted because the PR lists open items. Which `mz` release carries it? - **Upsert v2** ([#40189](#40189)): still omitted because `enable_upsert_v2` is `false`. Publish it? ## Snapshot drafts (with PR links) in mz-skills - rc.1: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.1/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.1/omitted.md), [MaterializeInc/mz-skills#351](MaterializeInc/mz-skills#351) - rc.2: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.2/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.2/omitted.md), [MaterializeInc/mz-skills#352](MaterializeInc/mz-skills#352) - rc.3: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.3/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.3/omitted.md), [MaterializeInc/mz-skills#353](MaterializeInc/mz-skills#353) - final: [release_notes.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/final/release_notes.md), [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/final/omitted.md) **Borderline PRs omitted:** rc.1: 10 — [rc.1/omitted.md#borderline](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.49.0/rc.1/omitted.md#borderline), rc.2: 0, rc.3: 0, final: 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]> Co-authored-by: Pranshu Maheshwari <[email protected]>
Motivation
Closes https://linear.app/materializeinc/issue/SQL-715/drop-index-with-dependents-should-not-be-allowed.
This corrects a longstanding mis-feature that will become more dire once we have pinned LIR.
Description
Orphaned dataflows are a confusing mis-feature: since indexes aren't first-class SQL objects, they could be freely dropped.
That's all well and good for an OLTP database, but not so an operational one.
This PR forbids orphaned dataflows.
DROP INDEXnow fails withDEPENDENT_OBJECTS_STILL_EXIST(2BP01) when a materialized view, index, or metric sink reads from the index, and the error names the readers. When one statement drops several indexes, one error names every blocked index.DROP INDEX ... CASCADEkeeps its existing meaning, the same asRESTRICT: it fails the same way, and there is no way to drop all dependents of an index in one statement. Users drop the readers themselves, or replace them with a blue/green deployment.A newer index on a relation can read from an older one on the same relation, so indexes on one relation now have to be dropped newest first.
An
enable_drop_in_use_indexflag defaults to off, but allows the old behavior, with aWARNINGnotice, an error-level log event, and a metric increment.The Terraform provider replaces an index by running
DROP INDEX ... RESTRICTfirst, so replacing amaterialize_indexthat a materialized view reads fails with the in-use error on every apply, with any provider version. Addinglifecycle { replace_triggered_by = [materialize_index.<name>.id] }to the materialized view makes Terraform drop the view first and recreate it after the index, which terraform-provider-materialize#953 documents.terraform destroycan also fail intermittently, because the provider drops the view and the index in parallel. #952 makes the provider retry the index drop while the view is being dropped.In-flight transient dataflows (
SUBSCRIBE,COPY TO, andSELECT) never block a drop and are reported in a notice instead.Verification
New tests
Existing tests changed to match the new behavior. Tests drop the readers explicitly instead of orphaning a dataflow, unless they test orphans themselves.
Other test frameworks
Gaps