Skip to content

[Bug]: one failed pod list re-homes ate.actor.stats.cpu.time increments onto a permanent pool-less series #1478

Closed

Description

@baizhenyu

What happened?

atelet's stats sweep resolves worker-pool labels with one fresh pod LIST per sweep (nodeWorkerPools in cmd/atelet/statspoller.go), and any failure — a timed-out or errored list — makes that tick's samples group without the ate.workerpool.* labels.

For the gauges that is a one-tick blink. For ate.actor.stats.cpu.time it is permanent damage: an OTel counter binds every increment to its attribute set, so one flapped sweep re-homes that tick's CPU increase onto a newly-born pool-less series. Concretely, with sweeps Δ1, Δ2 (list fails), Δ3:

  • {...,pool=P} holds Δ1+Δ3, {...} (no pool) holds Δ2 — forever; counter increments cannot be re-attributed.
  • rate(ate_actor_stats_cpu_time{workerpool="P"}) dips during every flap, indistinguishable from the workload going quiet.
  • A phantom pool-less series bursts precisely during apiserver trouble — the same incidents the telemetry exists to debug.
  • Cardinality doubles for every group that ever flapped, and the stranded series persists for the process lifetime.

A partial miss (one pod absent from the list while its ateom directory still exists) flaps just that actor's contributions — same mechanism, smaller blast radius.

Expected Behavior

A pod's pool is immutable for the pod's lifetime (the ate.dev/worker-pool label is stamped at creation; pods never move between pools), so resolutions can be cached per pod UID with zero correctness risk. A failed list should answer from the cache and keep the increments on the pooled series. The residual unlabeled case shrinks to "pod first seen while the apiserver is unreachable", which heals on the next successful list.

Steps to Reproduce

  1. Run an actor and let the stats poller sweep it with a working pod list (samples group under ate.workerpool.*).
  2. Make the next sweep's pod list fail (block the apiserver for >10s, or in a unit test return nil from the resolver).
  3. Observe that sweep's samples — including the CPU delta added to ate.actor.stats.cpu.time — land on a new label set without the pool keys, and remain there after the list recovers.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

main

Confirmation

  • I have searched existing issues and verified that this is not a duplicate.
  • I have verified that this issue occurs on the latest commit on main.

Activity

  1. added 3 commits that reference this issue on Sep 4, 2026
    ba4cd68
    7c79660
    2b978ff
  2. added a commit that references this issue on Sep 11, 2026
    bdcde26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions