Orbit: hosts Fleet already has re-enroll about 2,000 times a day on a large instance #58739

Fleet versions

• Discovered: fleet-v4.94.0 (Fleet Cloud; data from 2026-11-05)
• Reproduced: not yet

Web browser and operating system: N/A (server data)

💥 Actual behavior

On a large Fleet Cloud instance, the activity log has about 2,000 fleet_enrolled activities a day (30-day average), while fewer than 200 new hosts are created a day. Fleet records fleet_enrolled on every successful orbit enrollment, including a host it already has. For a known host, the enrollment only replaces its orbit node key, so neither created_at nor last_enrolled_at changes. So roughly 1,800 enrollments a day are hosts Fleet already has enrolling orbit again. We don't yet know whether that's many hosts once each or a few hosts over and over.

🛠️ Expected behavior

A known host enrolls orbit again only after a reinstall, reimage or reset, not repeatedly.

🧑‍💻 Steps to reproduce

These steps:

• Have been confirmed to consistently lead to reproduction in multiple Fleet instances.
• Describe the workflow that led to the error, but have not yet been reproduced in multiple Fleet instances.

1. On the instance's read replica, count orbit enrollments per host over the last 7 days (the (activity_type, created_at) index keeps this to about 14k rows):

   SELECT CASE WHEN n = 1 THEN '1' WHEN n <= 3 THEN '2-3' WHEN n <= 10 THEN '4-10' WHEN n <= 100 THEN '11-100' ELSE '100+' END AS enrollments_per_host,
          COUNT(*) AS hosts, SUM(n) AS enrollments
   FROM (SELECT JSON_EXTRACT(details, '$.host_id') AS host_id, COUNT(*) AS n
         FROM activity_past
         WHERE activity_type = 'fleet_enrolled' AND created_at >= NOW() - INTERVAL 7 DAY
         GROUP BY host_id) x
   GROUP BY enrollments_per_host
   ORDER BY MIN(n);

2. For the hosts that enrolled more than 10 times, see what they are:

   SELECT h.platform, h.hardware_model, COUNT(*) AS hosts, SUM(x.n) AS enrollments
   FROM (SELECT CAST(JSON_EXTRACT(details, '$.host_id') AS UNSIGNED) AS host_id, COUNT(*) AS n
         FROM activity_past
         WHERE activity_type = 'fleet_enrolled' AND created_at >= NOW() - INTERVAL 7 DAY
         GROUP BY host_id
         HAVING n > 10) x
   JOIN hosts h ON h.id = x.host_id
   GROUP BY h.platform, h.hardware_model
   ORDER BY enrollments DESC
   LIMIT 10;

3. Pull the orbit log from one of those hosts.

🕯️ More info (optional)

Possible causes, in the order I'd check them:

• Orbit re-runs the server enroll when only the node key write fails: one host emits dozens of fleet_enrolled activities and webhooks #55613: when orbit can't write its node key, it re-runs the whole server enrollment, six times per series, and starts a new series every few minutes. One stuck host makes dozens of enrollments an hour.
• Machines sharing an identity: two machines that report the same hardware UUID (cloned VMs, golden images) match the same host row. Each enrollment replaces the other's orbit node key, and orbit re-enrolls after 45 seconds of rejected requests (client/orbit_client.go, Orbit re-enroll improvements #47652), so the pair takes turns. Duplicate identities exist on this instance: hundreds of serial numbers appear on more than one host.
• Reinstalls, reimages or non-persistent VDI: expected, and spread across many hosts once each.

Found while building the customer profile for load test 2.0 (#56102).