Repository navigation
envd: collect and record replica swap metrics - #38788
Merged
Merged
Conversation
leedqin
enabled auto-merge (squash)
September 11, 2026 20:36
leedqin
disabled auto-merge
September 11, 2026 20:38
clusterd processes already report their swap usage as `swap_bytes` on /api/usage-metrics, but the Kubernetes orchestrator discards it after folding it into `heap_bytes` and `disk_bytes`. Plumb it through to the controller and record it in `mz_cluster_replica_metrics_history`, expose it on `mz_cluster_replica_metrics`, and derive a `swap_percent` column (swap as a share of the heap allocation) on the `mz_cluster_replica_utilization` views so the Console can show how much of a replica's memory is spilled to disk.
Adding the `swap_bytes` column changes the builtin fingerprint of `mz_cluster_replica_metrics_history`, which panics at catalog open unless the change is declared in the MIGRATIONS list. The append is backward-compatible, so the shard schema evolves in place (no data loss) and read-only 0dt instances accept it. Verified with the previously failing ZeroDowntimeUpgradeEntireMzOnce platform-checks scenario.
leedqin
force-pushed
the
replica-swap-bytes
branch
from
September 11, 2026 21:56
a6b2793 to
c8214e8
Compare
bosconi
added a commit
that referenced
this pull request
Sep 17, 2026
…rced ones (#38917) ### Motivation Since #38864 every 0dt upgrade from a catalog written before v26.41.0 to `main` crash-loops the new environmentd in read-only mode ([SQL-713](https://linear.app/materializeinc/issue/SQL-713)): ``` environmentd: fatal: cannot migrate builtin schemas from version 26.19.0 to version 26.43.0-dev.0: missing shard ID for builtin SystemObjectDescription { schema_name: "mz_internal", object_type: Table, object_name: "mz_replica_hydration_history" } ``` Six jobs on [nightly#18324](https://buildkite.com/materialize/nightly/builds/18324): all four `Postgres CDC (before source versioning, multi-version upgrade, ...)` shards (from 0.147.0 and 26.19.0) and both `Checks Self-Managed earliest to latest direct upgrade` shards (from 0.147.0). The same jobs passed on [#18320](https://buildkite.com/materialize/nightly/builds/18320), the previous `main` nightly. #38864 registers `MigrationStep::evolution("26.43.0-dev.0", ..., "mz_replica_hydration_history")`. That table first shipped in v26.41.0, so a source catalog older than that has no shard for it. `plan_migration` keeps every step above the source version, the plan reaches `migrate_evolve_one`, the shard lookup finds nothing, and the read-only branch bails. In leader mode the same case returns `Ok(())`, which is why the in-place `Checks ... upgrade` scenarios pass on the same build and only the 0dt-preflight jobs fail. #36668 fixed this exact bail for the *forced* dev-to-dev plan by filtering shardless objects out of `plan_forced_migration`: "there is nothing to evolve or replace for an object that does not yet exist in persist." The versioned plan never got the filter, so any evolution step for a builtin younger than the source version reproduces it. #38788's step for `mz_cluster_replica_metrics_history` did not, only because that table predates every supported source version. ### Description Move the filter to the one place both plans pass through. `Migration::run` now calls `drop_shardless` on the selected plan, forced or versioned, before executing it; `plan_forced_migration` loses its private copy of the same filter. A builtin with no registered shard does not exist in persist yet, whether it was added in the target version or the source version predates it entirely, and in both cases the leader creates it at the target schema during bootstrap. The read-only bail in `migrate_evolve_one` stays as the fail-fast for anything that still reaches it. This keeps the semantics #36668 chose rather than adding a new one: the read-only race the bail guards against, a leader registering the shard while a read-only process evolves it, cannot involve a builtin the source version never had. ### Verification New turmoil test `test_evolution_skips_builtin_without_shard`: one builtin table with a shard and a stale fingerprint (the pre-evolution schema), one without a shard at the current fingerprint (the shape catalog open produces for a builtin new to the durable catalog), steps at 0.2.0 and 0.3.0 over a 0.1.0 source, run once read-only and once as leader. Both runs must succeed, evolve the first table, and leave the second out of `new_fingerprints`. - Against `main`'s `builtin_schema_migration.rs` the read-only run fails with `missing shard ID for builtin SystemObjectDescription { schema_name: "schema", object_type: Table, object_name: "new_table" } (s2)`, the production message. - With this change both runs pass, and the existing randomized `test_builtin_schema_migration` harness passes. Each turmoil test was run in its own process (`cargo test -p mz-adapter --lib -- <full path> --exact`), which is what nextest does in CI. Running the module's tests together in one process fails independently of this change: `configure_tracing_for_turmoil` installs a global subscriber under a `Once`, and when `hydration_history_forced_migration_policy` (`#[mz_ore::test]`) runs first the install fails and poisons the `Once` for every turmoil test after it. Pre-existing, not addressed here. `cargo fmt` clean; `cargo clippy -p mz-adapter --tests -D warnings` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On swap-based replica sizes, the catalog only exposes the combined
heap_bytes/heap_limit. clusterd already measures swap separately (VmSwapfrom/proc/self/status) and reports it asswap_byteson/api/usage-metrics, but the Kubernetes orchestrator folds it intoheap_bytes(and, for compatibility, intodisk_bytes) and drops the standalone value, so it never reaches SQL.Deriving it downstream is unreliable:
heap_bytes - memory_bytesmixes clusterd's VmRSS with the metrics-server working set (different instruments, different sampling instants), anddisk_bytesonly equals swap on swap-enabled sizes, which are not distinguishable from public catalog relations. Recording swap first-class lets the Console show how much of a replica's memory is spilled to disk. Fixes CNS-154Description
swap_bytestoServiceProcessMetrics. The Kubernetes orchestrator keeps the clusterd-reported value (the existing swap-into-disk_bytesshim is unchanged); the process orchestrator reportsNone.swap_bytesas a nullableuint8column at the end ofREPLICA_METRICS_HISTORY_DESC. This is a backward-compatible persist schema change, so existing history is preserved and older rows readNULL(same pattern asheap_bytes/heap_limitin 96fa447).mz_internal.mz_cluster_replica_metrics_historyand expose it onmz_internal.mz_cluster_replica_metrics.swap_percenttomz_internal.mz_cluster_replica_utilizationandmz_cluster_replica_utilization_history, computed asswap_bytes / heap_limit * 100.No clusterd changes are needed: every running clusterd already reports
swap_bytes, so the metric lights up on existing replicas as soon as environmentd upgrades.Two decisions worth reviewer attention:
swap_percentdenominator:heap_limit, matching the sibling columns' "percent of the total allocation" contract, soheap_percent - swap_percentreads as the RAM share and the two stack in a gauge. The spill fraction (swap as a share ofheap_bytes) is derivable asswap_percent / heap_percentor from the raw columns.swap_bytesdirectly, the orchestrator shim that folds swap intodisk_bytescan be retired (see the comment inget_metrics).Verification
Regenerated the explain goldens (
system-cluster.slt,catalog_server_explain.slt) andautogenerated/mz_internal.slt, and added the new columns tomz_catalog_server_index_accounting.slt; all four files pass against a fresh catalog. The system-catalog docs page is updated to match the column comments.🤖 Generated with Claude Code