Skip to content

Commit 78262a7

Browse files
authored
feat(promhttp): add CoalesceGather option to deduplicate concurrent Gather calls (#1969)
* feat(promhttp): add CoalesceGather option to deduplicate concurrent Gather calls When a collector's Collect() is slower than the scrape interval, each incoming HTTP request triggers an independent Gather() that spawns its own goroutine pipeline. With no upper bound, this causes goroutine pile-up proportional to (scrape rate / collection time) — the apparent "goroutine leak" reported in #1477. Add HandlerOpts.CoalesceGather bool. When true, HandlerForTransactional wraps the underlying TransactionalGatherer in a coalescingGatherer that allows only one Gather to run at a time. Concurrent requests join the in-flight cycle and receive the same result once it completes. Reference counting ensures the TransactionalGatherer done() callback is called exactly once, after the last handler finishes encoding. Design decisions: - Per-cycle gatherCycle object (not fields on the struct) prevents the race where a new cycle overwrites result fields while prior waiters are still reading them. - Mutex-based ref counting (not atomic) ensures c.cycle = nil is cleared before cy.done() is called, ruling out double-done on a stale pointer. - close(cy.ready) happens-before <-cy.ready returns (Go memory model), so cy.mfs/err/done are safely readable without additional locking. - Zero overhead when disabled: single if-branch in HandlerForTransactional setup, identical code path when CoalesceGather is false. Fixes #1477 Signed-off-by: Kemal Akkoyun <[email protected]> * fix(promhttp): fail coalesced joiners when the gatherer panics When CoalesceGather is enabled and the wrapped gatherer itself panics, requests that joined the in-flight cycle were unblocked with a nil error and returned an empty but successful response. They now receive a sentinel error and fail like the panicking request instead of silently reporting no metrics. The leader's panic still propagates and is handled by net/http, matching the non-coalesced path; no recover is introduced. Also document the panic behaviour on the option, make the coalescing invariant test deterministic instead of timing-dependent, and align the goroutine-leak test with the repository's WaitGroup convention. Signed-off-by: Kemal Akkoyun <[email protected]> * fix(promhttp): give coalesced callers independent metric slices Concurrent callers sharing a coalesced Gather cycle previously received the same slice header. A caller that filtered or reordered its slice (for example applying name[] query parameters) could then race another caller of the same cycle. Each caller now gets its own slices.Clone; the MetricFamily values stay shared and read-only. Also clarify the CoalesceGather documentation (shared-snapshot staleness, the Timeout interaction, and slice-versus-contents mutation) and tidy the sentinel error and coalescer test helpers. Signed-off-by: Kemal Akkoyun <[email protected]> * docs(promhttp): mark CoalesceGather as experimental Signed-off-by: Kemal Akkoyun <[email protected]> --------- Signed-off-by: Kemal Akkoyun <[email protected]>
1 parent 34e9a7f commit 78262a7

3 files changed

Lines changed: 460 additions & 2 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
* [CHANGE] prometheus: Name validation now always uses the UTF-8 scheme instead of the deprecated `model.NameValidationScheme` global. Default behavior is unchanged; code that set `NameValidationScheme = LegacyValidation` no longer gets legacy enforcement at metric, label, and push-grouping construction. #2051
44
* [FEATURE] HTTP handlers created by `promhttp` package now support metrics filtering by providing one or more `name[]` query parameters. The default behavior when none are provided remains the same, returning all metrics. #1925
5+
* [FEATURE] promhttp: Add opt-in `HandlerOpts.CoalesceGather` to deduplicate concurrent `Gather` calls so overlapping scrapes share one collection cycle, preventing goroutine pile-up when the scrape rate outpaces collection time. #1969
56
* [BUGFIX] promhttp: `InstrumentHandlerDuration` and `InstrumentHandlerCounter` no longer panic when given an observer/counter that does not implement `ExemplarObserver`/`ExemplarAdder` (e.g. a `SummaryVec`). The exemplar is dropped and the value is recorded via the plain `Observe`/`Add` path, matching the safe-cast already used by `Timer.ObserveDurationWithExemplar`. #2005
67

78
## Unreleased `exp` module

‎prometheus/promhttp/http.go‎

Lines changed: 150 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,12 @@ import (
3737
"fmt"
3838
"io"
3939
"net/http"
40+
"slices"
4041
"strconv"
4142
"sync"
4243
"time"
4344

45+
dto "github.com/prometheus/client_model/go"
4446
"github.com/prometheus/common/expfmt"
4547

4648
"github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil"
@@ -79,6 +81,113 @@ var gzipPool = sync.Pool{
7981
},
8082
}
8183

84+
// coalescingGatherer wraps a TransactionalGatherer to deduplicate concurrent
85+
// Gather calls. When a Gather is already in flight, new callers join the
86+
// existing cycle and receive the same result once it completes. The underlying
87+
// done function is called exactly once, when the last joined caller releases.
88+
//
89+
// This prevents goroutine pile-up when the scrape rate is faster than the
90+
// time collectors need to produce metrics.
91+
type coalescingGatherer struct {
92+
g prometheus.TransactionalGatherer
93+
mu sync.Mutex
94+
cycle *gatherCycle
95+
}
96+
97+
// gatherCycle tracks a single in-flight Gather and all HTTP handlers sharing it.
98+
type gatherCycle struct {
99+
ready chan struct{} // closed when Gather completes; happens-before reads of mfs/err/done
100+
mfs []*dto.MetricFamily // canonical result, set before ready is closed; callers get a slices.Clone, the element values stay shared and must not be mutated
101+
err error // set before ready is closed
102+
done func() // underlying done callback; set before ready is closed
103+
refs int // number of handlers using this cycle; protected by coalescingGatherer.mu
104+
}
105+
106+
var _ prometheus.TransactionalGatherer = (*coalescingGatherer)(nil) // compile-time interface check
107+
108+
// errGatherPanicked is returned to callers that joined an in-flight coalesced
109+
// Gather whose underlying gatherer panicked. See the panic guard in Gather for
110+
// why joiners receive this error instead of the panic itself.
111+
var errGatherPanicked = errors.New("coalesced gather panicked")
112+
113+
func (c *coalescingGatherer) Gather() ([]*dto.MetricFamily, func(), error) {
114+
c.mu.Lock()
115+
if cy := c.cycle; cy != nil {
116+
// c.cycle is non-nil while Gather runs or handlers are still consuming its results.
117+
cy.refs++
118+
c.mu.Unlock()
119+
<-cy.ready
120+
// Each caller gets its own slice header so it can filter or reorder
121+
// without racing other callers sharing this cycle. The *dto.MetricFamily
122+
// values remain shared and must not be mutated in place.
123+
return slices.Clone(cy.mfs), c.releaseFunc(cy), cy.err
124+
}
125+
cy := &gatherCycle{
126+
ready: make(chan struct{}),
127+
done: func() {},
128+
refs: 1,
129+
}
130+
c.cycle = cy
131+
c.mu.Unlock()
132+
133+
// Guard against a panic in c.g.Gather. The common case, a panicking
134+
// Collector, never reaches here: Registry.Gather recovers Collector panics
135+
// and returns them as an error. This guard only covers the rare case where
136+
// the wrapped gatherer itself panics.
137+
//
138+
// We deliberately do not recover: the leader's panic propagates and is
139+
// handled by net/http exactly as it would be without coalescing. We only
140+
// set cy.err before closing cy.ready so joiners waiting on <-cy.ready fail
141+
// with that error instead of silently returning an empty, successful
142+
// response, and we clear c.cycle so the next Gather starts a fresh cycle.
143+
//
144+
// The leader never runs its own releaseFunc on this path, so its ref is
145+
// not decremented; that is harmless because the cycle is detached (c.cycle
146+
// = nil) and cy.done is still the no-op set at construction (c.g.Gather
147+
// panicked before assigning a real done). If cy.done is ever made non-nil
148+
// before c.g.Gather runs, this path would need to release it.
149+
panicked := true
150+
defer func() {
151+
if panicked {
152+
c.mu.Lock()
153+
if c.cycle == cy {
154+
c.cycle = nil
155+
}
156+
c.mu.Unlock()
157+
cy.err = errGatherPanicked // set before close: happens-before joiners' reads
158+
close(cy.ready)
159+
}
160+
}()
161+
cy.mfs, cy.done, cy.err = c.g.Gather()
162+
panicked = false
163+
close(cy.ready) // happens-before joiners' reads of cy.mfs/err/done
164+
165+
// Clone here too so cy.mfs stays the write-once canonical slice: joiners
166+
// read it concurrently via slices.Clone, so the leader must not hand out
167+
// (and potentially reorder) the same backing array.
168+
return slices.Clone(cy.mfs), c.releaseFunc(cy), cy.err
169+
}
170+
171+
// releaseFunc returns the done callback for one caller sharing cy.
172+
// When the last caller releases, the underlying done is invoked and the
173+
// cycle is cleared so the next Gather starts fresh.
174+
func (c *coalescingGatherer) releaseFunc(cy *gatherCycle) func() {
175+
return func() {
176+
c.mu.Lock()
177+
cy.refs--
178+
if cy.refs > 0 {
179+
c.mu.Unlock()
180+
return
181+
}
182+
// Last caller.
183+
if c.cycle == cy {
184+
c.cycle = nil
185+
}
186+
c.mu.Unlock()
187+
cy.done() // called outside the lock to avoid holding it during done
188+
}
189+
}
190+
82191
// Handler returns an http.Handler for the prometheus.DefaultGatherer, using
83192
// default HandlerOpts, i.e. it reports the first error as an HTTP error, it has
84193
// no error logging, and it applies compression if requested by the client.
@@ -125,6 +234,10 @@ func HandlerFor(reg prometheus.Gatherer, opts HandlerOpts) http.Handler {
125234
// Multiple metric names can be specified by providing the parameter multiple times.
126235
// When no name[] parameters are provided, all metrics are returned.
127236
func HandlerForTransactional(reg prometheus.TransactionalGatherer, opts HandlerOpts) http.Handler {
237+
if opts.CoalesceGather {
238+
reg = &coalescingGatherer{g: reg}
239+
}
240+
128241
var (
129242
inFlightSem chan struct{}
130243
errCnt = prometheus.NewCounterVec(
@@ -428,15 +541,50 @@ type HandlerOpts struct {
428541
// Service Unavailable and a suitable message in the body. If
429542
// MaxRequestsInFlight is 0 or negative, no limit is applied.
430543
MaxRequestsInFlight int
544+
// CoalesceGather, if true, deduplicates concurrent Gather calls so that
545+
// only one collection runs at a time. Additional requests that arrive
546+
// while a Gather is in flight will receive the same result once it
547+
// completes. This prevents goroutine pile-up when the scrape rate is
548+
// faster than the time collectors need to produce metrics.
549+
//
550+
// When enabled, concurrent scrapers share a single metric snapshot per
551+
// collection cycle. Each request receives its own copy of the returned
552+
// slice, so filtering or reordering it (for example via name[] query
553+
// parameters) is safe. The pointed-to MetricFamily values are still
554+
// shared: the built-in handler only reads them, so this is safe in
555+
// practice, but a custom TransactionalGatherer that mutates the returned
556+
// families in place after Gather returns must not use this option.
557+
//
558+
// Because the snapshot is shared, a request that arrives while a cycle is
559+
// in flight receives that cycle's result even though collection began
560+
// before the request; two scrapers joined to one cycle observe the same
561+
// timestamps rather than independently gathered data.
562+
//
563+
// Consider using CoalesceGather together with Timeout. Timeout bounds the
564+
// client-facing response time and keeps at most one collection running at
565+
// a time, but it does not cancel the underlying Gather: a joined request
566+
// that times out still holds a MaxRequestsInFlight slot until the shared
567+
// collection completes.
568+
//
569+
// Panic handling: a panicking Collector is already turned into an error by
570+
// the registry, so joiners receive that error like any other. In the rare
571+
// case where the wrapped gatherer itself panics, the panicking request's
572+
// panic propagates as usual (handled by net/http), while requests that
573+
// joined the same cycle receive an error rather than an empty response.
574+
//
575+
// NOTE: This option is experimental and may change or be removed in a
576+
// future release.
577+
CoalesceGather bool
431578
// If handling a request takes longer than Timeout, it is responded to
432579
// with 503 ServiceUnavailable and a suitable Message. No timeout is
433580
// applied if Timeout is 0 or negative. Note that with the current
434581
// implementation, reaching the timeout simply ends the HTTP requests as
435582
// described above (and even that only if sending of the body hasn't
436583
// started yet), while the bulk work of gathering all the metrics keeps
437584
// running in the background (with the eventual result to be thrown
438-
// away). Until the implementation is improved, it is recommended to
439-
// implement a separate timeout in potentially slow Collectors.
585+
// away). When CoalesceGather is enabled, only one such background Gather
586+
// can be in flight at a time. It is also recommended to implement a
587+
// separate timeout in potentially slow Collectors.
440588
Timeout time.Duration
441589
// If true, the experimental OpenMetrics encoding is added to the
442590
// possible options during content negotiation. Note that Prometheus

0 commit comments

Comments
 (0)