Hand cluster shards off before final cleanup on shutdown (v3) - #8946

Summary

When Sharding shuts down, it now hands its shards off after their entities and singletons stop, releasing each lock individually before the final releaseAll. Live runners can take the shards over without waiting for lock leases to expire. There's no new public API: disposing the layer, e.g. on SIGTERM, starts the handoff. This backports the design of the main PR, #8941.

Changes

Sharding.ts:

• Shutdown start. Once shutdown starts, including the preemptive shutdown triggered by entity finalizers, the runner stops acquiring shards. Registration and unregistration are serialized, so a registration that's in flight can't re-advertise a runner that is shutting down.

• Handoff. The Sharding finalizer runs shutdown, then the handoff:

  • every held shard is released;
  • entities still running on those shards are forced to stop;
  • each lock is released with RunnerStorage.release once its entities and singletons have stopped.

  While handing off, the pending shards are released in one batch, so 600 shards don't each need their own release fiber. The trade-off is head-of-line blocking: one slow entity or singleton teardown delays every lock in the batch, and one slow storage release delays the ones after it. Shards whose ordinary release was already running when shutdown started are left to that release.

• Best effort. Only the handoff wait is bounded, by the shard lock interval. If it times out, the final releaseAll attempts to release the remaining locks. Total shutdown is not bounded: unregistering and the final releaseAll can still wait on storage. Finalizers are uninterruptible, so the wait is made interruptible for its timeout. Release forks run in the FiberSet, because forks into the closing sharding scope never run.

• Singletons. Shard releases now wait for the singletons on those shards to stop. That includes singletons whose registration scope is already closing. A singleton is not restarted while its previous instance is still stopping. Singleton teardown is awaited outside the singleton lock, so a singleton that registered another singleton can still be closed.

internal/entityManager.ts:

• Interrupting a shard now finds entities by address. Before, an entity with the same id on another shard could hide it, and the release waited forever. This can happen with workflow shard groups.
• interruptShard became interruptShards, which takes several shards, so each entity manager needs one fiber per batch.

Tests

In packages/cluster/test/Sharding.test.ts, under Sharding graceful shutdown, one test per behaviour change:

• hands shards off to a live peer before the final releaseAll, tearing down entities first
• waits for singleton teardown before releasing its shard
• stops acquiring shards once shutdown starts
• scope close finishes while lock storage is unhealthy
• interrupts an entity whose id is also active on a shard that was interrupted first

Validation

• Sharding.test.ts: 34/34. On unmodified v3 sources, four of the five tests above fail; the unhealthy-storage test covers the new handoff wait. Removing the shutdown acquire guard, the singleton teardown wait, or the interruptible handoff wait each fails the matching test.
• ClusterWorkflowEngine, run alone: about 2.9s for its slowest test, the same as v3 (about 2.85s).
• @effect/cluster tests other than SqlRunnerStorage passed 111/111 before the test reduction. SqlRunnerStorage needs Docker containers, which were unavailable locally; it doesn't use Sharding.
• platform-node NodeClusterSocket.test.ts: pass.
• tsc -b for @effect/cluster: pass. eslint and docgen passed before the test reduction.

Closes EFF-1884