Stats interface: support asynchronous/observable gauge metrics #72885

Description

airflow.sdk.observability.stats.Stats (and the OTel-backed stats logger under the hood) only
exposes synchronous, push-style metric recording: Stats.gauge(name, value) records whatever
value the caller passes at the moment it's called. There's no way to register a callback that
the metrics SDK invokes on its own export/collection cadence.

This is a real gap for "current state" gauges — e.g. a thread-pool occupancy count, a
connection-pool checkout count, or a boolean saturation flag. With only a synchronous API,
producers have to hand-roll a polling loop (a while loop on a timer that reads current state
and calls Stats.gauge(...)) to get periodic samples. That has two failure modes in practice:

1. If the value is only pushed on a state transition (e.g. "just became saturated") rather than
   on a fixed interval, the metric goes silent for as long as the state doesn't change — which
   most gauge-consuming backends (Prometheus/OTel scrape-based pipelines included) treat the same
   as "no data," defeating the point of an alertable gauge.
2. Even when done correctly with a polling loop, every producer re-implements the same
   "read current value on a timer, catch exceptions so a metrics failure doesn't crash the loop"
   boilerplate that the OTel spec's ObservableGauge instrument (a callback registered once,
   invoked by the SDK itself at export time) already solves generically.

OpenTelemetry's Metrics API has supported asynchronous/observable instruments
(ObservableCounter, ObservableUpDownCounter, ObservableGauge) for a long time. Since
Airflow's OTel-backed stats path is already built on the OTel SDK, it seems feasible to expose
an equivalent on the Stats façade — e.g. Stats.gauge(name, callback=...) or a dedicated
Stats.observable_gauge(name, callback) registration API — so callers with a canonical "current
value" getter don't need to own their own timer loop, and the emitted samples are guaranteed
fresh on every collection cycle rather than only on edges.

Use case (concrete example)

Surfaced from an internal package building on Airflow's Stats interface for scheduling-latency
observability: a saturation flag ("are we at max_active_dagruns right now?") is currently only
loggable, not gauge-able, without a hand-rolled polling loop, precisely because the only available
primitive is the synchronous push Stats.gauge().

Use case

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct