Skip to content

Finish the metrics story: scenario overrides, random-walk gauges, interval metrics, semconv validation#140

Merged
andrewh merged 7 commits into
mainfrom
claude/admiring-rubin-ythfki
Jun 11, 2026
Merged

Finish the metrics story: scenario overrides, random-walk gauges, interval metrics, semconv validation#140
andrewh merged 7 commits into
mainfrom
claude/admiring-rubin-ythfki

Conversation

@andrewh

@andrewh andrewh commented Jun 10, 2026

Copy link
Copy Markdown
Owner

Completes the topology-driven metrics feature from PR 135 by implementing the four follow-up issues, one commit each.

Scenario overrides for metric values (closes #136)

Scenarios can override a metric's value distribution during their window, at operation scope (gateway.handle) or service scope (bare gateway, metrics only):

scenarios:
  - name: cpu spike
    at: +2m
    duration: 5m
    override:
      gateway:
        metrics:
          gateway.cpu.utilisation:
            value: 0.95 +/- 0.02

The engine pushes resolved overrides to observers each cycle via a new OverrideObserver interface; MetricObserver applies them when sampling values for counters, updowncounters, histograms, and gauge callbacks. Overlapping scenarios merge per metric name with the existing priority semantics. Validation rejects overrides of span-derived metrics, metrics not defined at the scope, and non-metric fields on service-scope keys.

Random-walk gauge values (closes #139)

Gauges gain an optional walk: mean-reversion timescale that replaces independent per-collection samples with an Ornstein-Uhlenbeck process — consecutive samples are temporally correlated while the stationary distribution still matches the configured mean and standard deviation. Optional min/max bounds clamp gauge values:

- name: gateway.cpu.utilisation
  type: gauge
  value: 0.65 +/- 0.1
  walk: 30s
  min: 0
  max: 1

White noise remains the default; walk timescales relate to wall-clock time between collections.

Rate-driven independent metrics (closes #137)

Topology-defined counters, updowncounters, and histograms accept an interval: field that emits sampled values on a wall-clock ticker instead of per span, so a service with no traffic still produces metric data:

- name: gateway.gc.pause.duration
  type: histogram
  unit: ms
  value: 2.5 +/- 0.8
  interval: 10s

MetricObserver.Start() runs one emitter goroutine per interval instrument and stops cleanly before meter provider shutdown. Interval emitters honour scenario value overrides. Gauges (already collection-driven) and span-derived metrics reject interval at validation.

Semconv-aware metric validation (closes #138)

motel validate now checks metric names against the vendored semantic convention registry. Where a name matches a known semconv metric, mismatched instrument types and units are reported as warnings on stderr — never errors, since users may intentionally deviate:

warning: service "gateway": metric "http.server.request.duration": unit "ms" does not match semantic convention unit "s"

Adds Registry.MetricByName to pkg/semconv. This caught non-compliant units in our own docs examples, which are updated to unit: s / unit: "{request}". The semconv attribute shorthand extension mentioned in the issue is left for a separate change.

Testing

  • 28 new test cases across config validation, scenario merging, observer behaviour (override application, walk correlation and mean reversion, bounds clamping, interval emission and stop), the semconv registry, and the validate command
  • Verified end to end: a topology combining a walking bounded gauge, an interval histogram, and a scenario override produces the expected values via motel run --signals metrics --stdout
  • make test and make lint pass

https://claude.ai/code/session_01TgWavXRfzQiTWDZccuS2pf


Generated by Claude Code

claude added 4 commits June 10, 2026 19:08
Scenarios can now override the value distribution of topology-defined
metrics during their activation window, at service or operation scope.
Override keys may be a bare service name (metrics only) or a
service.operation reference. The engine pushes resolved overrides to
observers each cycle; MetricObserver applies them when sampling values
for counters, updowncounters, histograms, and gauge callbacks.

Closes #136
Gauges gain an optional walk field: a mean-reversion timescale that
turns independent per-collection samples into an Ornstein-Uhlenbeck
process. Consecutive samples are temporally correlated while the
stationary distribution matches the configured mean and standard
deviation. Optional min/max bounds clamp gauge values, keeping metrics
like CPU utilisation in range.

Closes #139
Topology-defined counters, updowncounters, and histograms gain an
optional interval field that emits sampled values on a wall-clock timer
instead of per span, decoupling metric emission from trace rate. A
service with no traffic now produces metric data points. Gauges already
emit on the collection cycle and do not accept interval; span-derived
metrics remain span-coupled. Interval emitters honour scenario value
overrides and stop cleanly before meter provider shutdown.

Closes #137
motel validate now checks topology metric names against the vendored
semantic convention registry. Where a name matches a known semconv
metric, mismatched instrument types and units are reported as warnings
on stderr rather than errors, since users may intentionally deviate.
Custom metric names are never warned about. Adds Registry.MetricByName
to index metric groups by metric_name, and updates documentation
examples to use semconv-compliant units.

Closes #138

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several enhancements to the metrics system, including support for scenario-based metric overrides at service and operation scopes, random-walk gauges with optional bounds, and interval-driven metrics that emit on a timer independent of span activity. Additionally, it adds semantic convention validation warnings for metric type and unit mismatches during configuration validation. The feedback focuses on a critical performance optimization in pkg/synth/metrics.go to return FloatDistribution by value instead of by pointer in effectiveValue, which avoids heap allocations on the hot path during span observations and interval emissions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread pkg/synth/metrics.go Outdated
Comment thread pkg/synth/metrics.go
Comment thread pkg/synth/metrics.go Outdated
Comment thread pkg/synth/metrics.go
Returning a pointer to a local copy of the override distribution forced
a heap allocation on every span observation and interval tick. Return
(FloatDistribution, bool) by value instead.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR completes the follow-up work for topology-driven metrics by adding scenario-driven metric value overrides, more realistic gauge sampling (random walk + bounds), interval-driven metric emission for non-gauge instruments, and semantic-convention-aware metric validation warnings in motel validate.

Changes:

  • Add scenario override.metrics to temporarily replace a metric’s configured value distribution (service or operation scope).
  • Add gauge walk (Ornstein–Uhlenbeck mean-reverting process) plus optional min/max clamping, and add interval emission for counters/updowncounters/histograms.
  • Add semconv metric lookup + motel validate warnings when known semconv metric names use mismatched instrument type or unit; update docs/examples to be semconv-compliant.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
pkg/synth/topology.go Extends resolved metric definitions with interval, walk, min, max and parses durations.
pkg/synth/scenario.go Adds resolved Override.Metrics and merges metric overrides across active scenarios.
pkg/synth/scenario_test.go Adds coverage for parsing and merging metric overrides.
pkg/synth/observer.go Introduces OverrideObserver and dispatch helper for active overrides.
pkg/synth/metrics.go Applies scenario value overrides during sampling; implements random-walk gauges, bounds clamping, and interval emitters with clean stop.
pkg/synth/metrics_test.go Adds end-to-end tests for overrides, walk correlation/reversion, bounds, interval emission, and stop behavior.
pkg/synth/engine.go Pushes merged overrides to observers each cycle (both realtime and non-realtime loops).
pkg/synth/config.go Adds config fields (interval, walk, min, max), validates their combinations, and validates scenario metric overrides by scope.
pkg/synth/config_test.go Adds validation tests for metric overrides, walk/bounds, and interval rules.
pkg/semconv/registry.go Adds Registry.MetricByName indexing for semconv metric groups.
pkg/semconv/registry_test.go Tests metric lookup by name.
docs/reference/synth.md Documents semconv-aware validation warnings.
docs/how-to/test-alert-thresholds.md Updates semconv metric units in examples (s, {request}).
docs/examples/topology-driven-metrics.yaml Updates example units/comments to semconv-compliant values.
cmd/motel/README.md Documents new metrics scenario override, gauge walk + bounds, interval metrics, and semconv warnings.
cmd/motel/main.go Loads semconv registry for validate, emits semconv metric warnings, and starts/stops interval metric emitters for run.
cmd/motel/main_test.go Adds validate command tests asserting semconv warnings are emitted (without failing validation).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmd/motel/main.go
Comment thread pkg/synth/engine.go
Comment thread pkg/synth/engine.go
claude and others added 2 commits June 10, 2026 19:39
The engine notified observers of resolved overrides on every loop
iteration, repeatedly taking observer locks to set an unchanged (often
nil) map. Scenario contents are static, so the merged overrides only
change when the active scenario set does; compare against the previous
active set and notify on transitions only.

Also render omitted units explicitly in semconv validation warnings
instead of printing an empty string.
@andrewh
andrewh merged commit 0d132e8 into main Jun 11, 2026
2 checks passed
@andrewh
andrewh deleted the claude/admiring-rubin-ythfki branch June 11, 2026 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants