Finish the metrics story: scenario overrides, random-walk gauges, interval metrics, semconv validation#140
Conversation
Scenarios can now override the value distribution of topology-defined metrics during their activation window, at service or operation scope. Override keys may be a bare service name (metrics only) or a service.operation reference. The engine pushes resolved overrides to observers each cycle; MetricObserver applies them when sampling values for counters, updowncounters, histograms, and gauge callbacks. Closes #136
Gauges gain an optional walk field: a mean-reversion timescale that turns independent per-collection samples into an Ornstein-Uhlenbeck process. Consecutive samples are temporally correlated while the stationary distribution matches the configured mean and standard deviation. Optional min/max bounds clamp gauge values, keeping metrics like CPU utilisation in range. Closes #139
Topology-defined counters, updowncounters, and histograms gain an optional interval field that emits sampled values on a wall-clock timer instead of per span, decoupling metric emission from trace rate. A service with no traffic now produces metric data points. Gauges already emit on the collection cycle and do not accept interval; span-derived metrics remain span-coupled. Interval emitters honour scenario value overrides and stop cleanly before meter provider shutdown. Closes #137
motel validate now checks topology metric names against the vendored semantic convention registry. Where a name matches a known semconv metric, mismatched instrument types and units are reported as warnings on stderr rather than errors, since users may intentionally deviate. Custom metric names are never warned about. Adds Registry.MetricByName to index metric groups by metric_name, and updates documentation examples to use semconv-compliant units. Closes #138
There was a problem hiding this comment.
Code Review
This pull request introduces several enhancements to the metrics system, including support for scenario-based metric overrides at service and operation scopes, random-walk gauges with optional bounds, and interval-driven metrics that emit on a timer independent of span activity. Additionally, it adds semantic convention validation warnings for metric type and unit mismatches during configuration validation. The feedback focuses on a critical performance optimization in pkg/synth/metrics.go to return FloatDistribution by value instead of by pointer in effectiveValue, which avoids heap allocations on the hot path during span observations and interval emissions.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Returning a pointer to a local copy of the override distribution forced a heap allocation on every span observation and interval tick. Return (FloatDistribution, bool) by value instead.
There was a problem hiding this comment.
Pull request overview
This PR completes the follow-up work for topology-driven metrics by adding scenario-driven metric value overrides, more realistic gauge sampling (random walk + bounds), interval-driven metric emission for non-gauge instruments, and semantic-convention-aware metric validation warnings in motel validate.
Changes:
- Add scenario
override.metricsto temporarily replace a metric’s configuredvaluedistribution (service or operation scope). - Add gauge
walk(Ornstein–Uhlenbeck mean-reverting process) plus optionalmin/maxclamping, and addintervalemission for counters/updowncounters/histograms. - Add semconv metric lookup +
motel validatewarnings when known semconv metric names use mismatched instrument type or unit; update docs/examples to be semconv-compliant.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/synth/topology.go | Extends resolved metric definitions with interval, walk, min, max and parses durations. |
| pkg/synth/scenario.go | Adds resolved Override.Metrics and merges metric overrides across active scenarios. |
| pkg/synth/scenario_test.go | Adds coverage for parsing and merging metric overrides. |
| pkg/synth/observer.go | Introduces OverrideObserver and dispatch helper for active overrides. |
| pkg/synth/metrics.go | Applies scenario value overrides during sampling; implements random-walk gauges, bounds clamping, and interval emitters with clean stop. |
| pkg/synth/metrics_test.go | Adds end-to-end tests for overrides, walk correlation/reversion, bounds, interval emission, and stop behavior. |
| pkg/synth/engine.go | Pushes merged overrides to observers each cycle (both realtime and non-realtime loops). |
| pkg/synth/config.go | Adds config fields (interval, walk, min, max), validates their combinations, and validates scenario metric overrides by scope. |
| pkg/synth/config_test.go | Adds validation tests for metric overrides, walk/bounds, and interval rules. |
| pkg/semconv/registry.go | Adds Registry.MetricByName indexing for semconv metric groups. |
| pkg/semconv/registry_test.go | Tests metric lookup by name. |
| docs/reference/synth.md | Documents semconv-aware validation warnings. |
| docs/how-to/test-alert-thresholds.md | Updates semconv metric units in examples (s, {request}). |
| docs/examples/topology-driven-metrics.yaml | Updates example units/comments to semconv-compliant values. |
| cmd/motel/README.md | Documents new metrics scenario override, gauge walk + bounds, interval metrics, and semconv warnings. |
| cmd/motel/main.go | Loads semconv registry for validate, emits semconv metric warnings, and starts/stops interval metric emitters for run. |
| cmd/motel/main_test.go | Adds validate command tests asserting semconv warnings are emitted (without failing validation). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The engine notified observers of resolved overrides on every loop iteration, repeatedly taking observer locks to set an unchanged (often nil) map. Scenario contents are static, so the merged overrides only change when the active scenario set does; compare against the previous active set and notify on transitions only. Also render omitted units explicitly in semconv validation warnings instead of printing an empty string.
Completes the topology-driven metrics feature from PR 135 by implementing the four follow-up issues, one commit each.
Scenario overrides for metric values (closes #136)
Scenarios can override a metric's
valuedistribution during their window, at operation scope (gateway.handle) or service scope (baregateway, metrics only):The engine pushes resolved overrides to observers each cycle via a new
OverrideObserverinterface;MetricObserverapplies them when sampling values for counters, updowncounters, histograms, and gauge callbacks. Overlapping scenarios merge per metric name with the existing priority semantics. Validation rejects overrides of span-derived metrics, metrics not defined at the scope, and non-metric fields on service-scope keys.Random-walk gauge values (closes #139)
Gauges gain an optional
walk:mean-reversion timescale that replaces independent per-collection samples with an Ornstein-Uhlenbeck process — consecutive samples are temporally correlated while the stationary distribution still matches the configured mean and standard deviation. Optionalmin/maxbounds clamp gauge values:White noise remains the default; walk timescales relate to wall-clock time between collections.
Rate-driven independent metrics (closes #137)
Topology-defined counters, updowncounters, and histograms accept an
interval:field that emits sampled values on a wall-clock ticker instead of per span, so a service with no traffic still produces metric data:MetricObserver.Start()runs one emitter goroutine per interval instrument and stops cleanly before meter provider shutdown. Interval emitters honour scenario value overrides. Gauges (already collection-driven) and span-derived metrics rejectintervalat validation.Semconv-aware metric validation (closes #138)
motel validatenow checks metric names against the vendored semantic convention registry. Where a name matches a known semconv metric, mismatched instrument types and units are reported as warnings on stderr — never errors, since users may intentionally deviate:Adds
Registry.MetricByNametopkg/semconv. This caught non-compliant units in our own docs examples, which are updated tounit: s/unit: "{request}". The semconv attribute shorthand extension mentioned in the issue is left for a separate change.Testing
motel run --signals metrics --stdoutmake testandmake lintpasshttps://claude.ai/code/session_01TgWavXRfzQiTWDZccuS2pf
Generated by Claude Code