Skip to content

envd: collect and record replica swap metrics - #38788

Merged
leedqin merged 2 commits into
MaterializeInc:mainfrom
leedqin:replica-swap-bytes
Sep 11, 2026
Merged

leedqin merged 2 commits into
MaterializeInc:mainfrom
leedqin:replica-swap-bytes

Conversation

@leedqin

@leedqin leedqin commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On swap-based replica sizes, the catalog only exposes the combined heap_bytes / heap_limit. clusterd already measures swap separately (VmSwap from /proc/self/status) and reports it as swap_bytes on /api/usage-metrics, but the Kubernetes orchestrator folds it into heap_bytes (and, for compatibility, into disk_bytes) and drops the standalone value, so it never reaches SQL.

Deriving it downstream is unreliable: heap_bytes - memory_bytes mixes clusterd's VmRSS with the metrics-server working set (different instruments, different sampling instants), and disk_bytes only equals swap on swap-enabled sizes, which are not distinguishable from public catalog relations. Recording swap first-class lets the Console show how much of a replica's memory is spilled to disk. Fixes CNS-154

Description

  • Add swap_bytes to ServiceProcessMetrics. The Kubernetes orchestrator keeps the clusterd-reported value (the existing swap-into-disk_bytes shim is unchanged); the process orchestrator reports None.
  • Append swap_bytes as a nullable uint8 column at the end of REPLICA_METRICS_HISTORY_DESC. This is a backward-compatible persist schema change, so existing history is preserved and older rows read NULL (same pattern as heap_bytes / heap_limit in 96fa447).
  • Record it in mz_internal.mz_cluster_replica_metrics_history and expose it on mz_internal.mz_cluster_replica_metrics.
  • Add swap_percent to mz_internal.mz_cluster_replica_utilization and mz_cluster_replica_utilization_history, computed as swap_bytes / heap_limit * 100.

No clusterd changes are needed: every running clusterd already reports swap_bytes, so the metric lights up on existing replicas as soon as environmentd upgrades.

Two decisions worth reviewer attention:

  • swap_percent denominator: heap_limit, matching the sibling columns' "percent of the total allocation" contract, so heap_percent - swap_percent reads as the RAM share and the two stack in a gauge. The spill fraction (swap as a share of heap_bytes) is derivable as swap_percent / heap_percent or from the raw columns.
  • Follow-up: once the Console reads swap_bytes directly, the orchestrator shim that folds swap into disk_bytes can be retired (see the comment in get_metrics).

Verification

Regenerated the explain goldens (system-cluster.slt, catalog_server_explain.slt) and autogenerated/mz_internal.slt, and added the new columns to mz_catalog_server_index_accounting.slt; all four files pass against a fresh catalog. The system-catalog docs page is updated to match the column comments.

🤖 Generated with Claude Code

@leedqin
leedqin marked this pull request as ready for review September 11, 2026 20:32
@leedqin
leedqin requested review from a team as code owners September 11, 2026 20:32

@antiguru antiguru left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine! Thank you.

@leedqin
leedqin enabled auto-merge (squash) September 11, 2026 20:36
@leedqin
leedqin disabled auto-merge September 11, 2026 20:38
clusterd processes already report their swap usage as `swap_bytes` on
/api/usage-metrics, but the Kubernetes orchestrator discards it after
folding it into `heap_bytes` and `disk_bytes`. Plumb it through to the
controller and record it in `mz_cluster_replica_metrics_history`, expose
it on `mz_cluster_replica_metrics`, and derive a `swap_percent` column
(swap as a share of the heap allocation) on the
`mz_cluster_replica_utilization` views so the Console can show how much
of a replica's memory is spilled to disk.
Adding the `swap_bytes` column changes the builtin fingerprint of
`mz_cluster_replica_metrics_history`, which panics at catalog open unless
the change is declared in the MIGRATIONS list. The append is
backward-compatible, so the shard schema evolves in place (no data loss)
and read-only 0dt instances accept it. Verified with the previously
failing ZeroDowntimeUpgradeEntireMzOnce platform-checks scenario.
@leedqin
leedqin merged commit 8103f77 into MaterializeInc:main Sep 11, 2026
86 checks passed
bosconi added a commit that referenced this pull request Sep 17, 2026
…rced ones (#38917)

### Motivation

Since #38864 every 0dt upgrade from a catalog written before v26.41.0 to
`main` crash-loops the new environmentd in read-only mode
([SQL-713](https://linear.app/materializeinc/issue/SQL-713)):

```
environmentd: fatal: cannot migrate builtin schemas from version 26.19.0 to version 26.43.0-dev.0:
missing shard ID for builtin SystemObjectDescription { schema_name: "mz_internal", object_type: Table, object_name: "mz_replica_hydration_history" }
```

Six jobs on
[nightly#18324](https://buildkite.com/materialize/nightly/builds/18324):
all four `Postgres CDC (before source versioning, multi-version upgrade,
...)` shards (from 0.147.0 and 26.19.0) and both `Checks Self-Managed
earliest to latest direct upgrade` shards (from 0.147.0). The same jobs
passed on
[#18320](https://buildkite.com/materialize/nightly/builds/18320), the
previous `main` nightly.

#38864 registers `MigrationStep::evolution("26.43.0-dev.0", ...,
"mz_replica_hydration_history")`. That table first shipped in v26.41.0,
so a source catalog older than that has no shard for it.
`plan_migration` keeps every step above the source version, the plan
reaches `migrate_evolve_one`, the shard lookup finds nothing, and the
read-only branch bails. In leader mode the same case returns `Ok(())`,
which is why the in-place `Checks ... upgrade` scenarios pass on the
same build and only the 0dt-preflight jobs fail.

#36668 fixed this exact bail for the *forced* dev-to-dev plan by
filtering shardless objects out of `plan_forced_migration`: "there is
nothing to evolve or replace for an object that does not yet exist in
persist." The versioned plan never got the filter, so any evolution step
for a builtin younger than the source version reproduces it. #38788's
step for `mz_cluster_replica_metrics_history` did not, only because that
table predates every supported source version.

### Description

Move the filter to the one place both plans pass through.
`Migration::run` now calls `drop_shardless` on the selected plan, forced
or versioned, before executing it; `plan_forced_migration` loses its
private copy of the same filter. A builtin with no registered shard does
not exist in persist yet, whether it was added in the target version or
the source version predates it entirely, and in both cases the leader
creates it at the target schema during bootstrap. The read-only bail in
`migrate_evolve_one` stays as the fail-fast for anything that still
reaches it.

This keeps the semantics #36668 chose rather than adding a new one: the
read-only race the bail guards against, a leader registering the shard
while a read-only process evolves it, cannot involve a builtin the
source version never had.

### Verification

New turmoil test `test_evolution_skips_builtin_without_shard`: one
builtin table with a shard and a stale fingerprint (the pre-evolution
schema), one without a shard at the current fingerprint (the shape
catalog open produces for a builtin new to the durable catalog), steps
at 0.2.0 and 0.3.0 over a 0.1.0 source, run once read-only and once as
leader. Both runs must succeed, evolve the first table, and leave the
second out of `new_fingerprints`.

- Against `main`'s `builtin_schema_migration.rs` the read-only run fails
with `missing shard ID for builtin SystemObjectDescription {
schema_name: "schema", object_type: Table, object_name: "new_table" }
(s2)`, the production message.
- With this change both runs pass, and the existing randomized
`test_builtin_schema_migration` harness passes.

Each turmoil test was run in its own process (`cargo test -p mz-adapter
--lib -- <full path> --exact`), which is what nextest does in CI.
Running the module's tests together in one process fails independently
of this change: `configure_tracing_for_turmoil` installs a global
subscriber under a `Once`, and when
`hydration_history_forced_migration_policy` (`#[mz_ore::test]`) runs
first the install fails and poisons the `Once` for every turmoil test
after it. Pre-existing, not addressed here.

`cargo fmt` clean; `cargo clippy -p mz-adapter --tests -D warnings`
clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.1 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants