Repository navigation
Commit 78262a7
authored
feat(promhttp): add CoalesceGather option to deduplicate concurrent Gather calls (#1969)
* feat(promhttp): add CoalesceGather option to deduplicate concurrent Gather calls
When a collector's Collect() is slower than the scrape interval, each
incoming HTTP request triggers an independent Gather() that spawns its
own goroutine pipeline. With no upper bound, this causes goroutine
pile-up proportional to (scrape rate / collection time) — the apparent
"goroutine leak" reported in #1477.
Add HandlerOpts.CoalesceGather bool. When true, HandlerForTransactional
wraps the underlying TransactionalGatherer in a coalescingGatherer that
allows only one Gather to run at a time. Concurrent requests join the
in-flight cycle and receive the same result once it completes. Reference
counting ensures the TransactionalGatherer done() callback is called
exactly once, after the last handler finishes encoding.
Design decisions:
- Per-cycle gatherCycle object (not fields on the struct) prevents the
race where a new cycle overwrites result fields while prior waiters are
still reading them.
- Mutex-based ref counting (not atomic) ensures c.cycle = nil is cleared
before cy.done() is called, ruling out double-done on a stale pointer.
- close(cy.ready) happens-before <-cy.ready returns (Go memory model),
so cy.mfs/err/done are safely readable without additional locking.
- Zero overhead when disabled: single if-branch in HandlerForTransactional
setup, identical code path when CoalesceGather is false.
Fixes #1477
Signed-off-by: Kemal Akkoyun <[email protected]>
* fix(promhttp): fail coalesced joiners when the gatherer panics
When CoalesceGather is enabled and the wrapped gatherer itself panics, requests that joined the in-flight cycle were unblocked with a nil error and returned an empty but successful response. They now receive a sentinel error and fail like the panicking request instead of silently reporting no metrics. The leader's panic still propagates and is handled by net/http, matching the non-coalesced path; no recover is introduced.
Also document the panic behaviour on the option, make the coalescing invariant test deterministic instead of timing-dependent, and align the goroutine-leak test with the repository's WaitGroup convention.
Signed-off-by: Kemal Akkoyun <[email protected]>
* fix(promhttp): give coalesced callers independent metric slices
Concurrent callers sharing a coalesced Gather cycle previously received the same slice header. A caller that filtered or reordered its slice (for example applying name[] query parameters) could then race another caller of the same cycle. Each caller now gets its own slices.Clone; the MetricFamily values stay shared and read-only.
Also clarify the CoalesceGather documentation (shared-snapshot staleness, the Timeout interaction, and slice-versus-contents mutation) and tidy the sentinel error and coalescer test helpers.
Signed-off-by: Kemal Akkoyun <[email protected]>
* docs(promhttp): mark CoalesceGather as experimental
Signed-off-by: Kemal Akkoyun <[email protected]>
---------
Signed-off-by: Kemal Akkoyun <[email protected]>1 parent 34e9a7f commit 78262a7
3 files changed
Lines changed: 460 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| 45 | + | |
44 | 46 | | |
45 | 47 | | |
46 | 48 | | |
| |||
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
82 | 191 | | |
83 | 192 | | |
84 | 193 | | |
| |||
125 | 234 | | |
126 | 235 | | |
127 | 236 | | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
128 | 241 | | |
129 | 242 | | |
130 | 243 | | |
| |||
428 | 541 | | |
429 | 542 | | |
430 | 543 | | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
431 | 578 | | |
432 | 579 | | |
433 | 580 | | |
434 | 581 | | |
435 | 582 | | |
436 | 583 | | |
437 | 584 | | |
438 | | - | |
439 | | - | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
440 | 588 | | |
441 | 589 | | |
442 | 590 | | |
| |||
0 commit comments