Layer 3: Periodic capability polling for silent redeployment detection #494

Parent Issue

#478 - Capability Freshness Architecture

Objective

Add a background capability poller that periodically re-fetches tool/resource/prompt lists from downstream MCP servers and updates the registry/store if anything changed. This catches scenario 2c from the epic: silent server redeployments where no CR change occurs and no MCP notifications/tools/list_changed notification is sent.

Problem

Layer 2 (push-based notification refresh) handles scenario 2d -- servers that actively send capability-change notifications. However, when a downstream server is silently redeployed (e.g., a Kubernetes pod restart that doesn't trigger a CR update), its tool list may change without any notification being sent. The aggregator's cached tools become stale until the next full reconnection.

Terminology

This issue distinguishes two server categories based on their connection model, not their authentication mechanism:

• Non-authenticated servers (StatusConnected): Have a persistent global ServerInfo.Client. Tools stored in ServerInfo.Tools. One client serves all sessions.
• Authenticated servers (StatusAuthRequired): Have per-session clients pooled in SessionConnectionPool. Tools stored per-session in CapabilityStore. Includes manual OAuth login, token forwarding, and token exchange -- the auth mechanism is irrelevant to the polling design.

Design

New file: internal/aggregator/capability_poller.go

A CapabilityPoller struct that runs a background goroutine on a configurable interval:

type CapabilityPoller struct {
    aggregator   *AggregatorServer
    interval     time.Duration
    sfGroup      *singleflight.Group  // shared with notification subscriber
}

Polling logic by server type

Non-authenticated servers (StatusConnected with persistent ServerInfo.Client):

• Call refreshNonOAuthCapabilities(serverName) -- this method already exists in notification_subscriber.go and does exactly the right thing: re-fetches tools/resources/prompts via the persistent client, deep-compares with ServerInfo.Tools/Resources/Prompts using toolListsEqual/resourceListsEqual/promptListsEqual, and updates + notifies only if something changed.
• Singleflight-deduplicated via the existing notifRefreshGroup with key notif-caps/<serverName>, so a poll that overlaps with a notification-triggered refresh is coalesced.
• A failed ListTools is a natural health signal for Layer 4. If the persistent client cannot list tools, log at warn and skip the update (don't clear the existing tools). Optionally call client.Ping() first as a cheap connectivity check.

Authenticated servers (StatusAuthRequired with per-session pooled clients):

• Need a new method on SessionConnectionPool to iterate active sessions for a given server:

  func (p *SessionConnectionPool) GetSessionsForServer(serverName string) []PooledSession
  
  type PooledSession struct {
      SessionID string
      Client    MCPClient
  }

  This returns a snapshot of currently pooled sessions for the server. No lock is held after return; the caller must handle the case where a session was evicted between snapshot and use.

• For each active session, call refreshSessionCapabilities(ctx, serverName, sessionID, client) -- this method already exists in notification_subscriber.go and correctly: re-fetches tools/resources/prompts via the session's own client, deep-compares with CapabilityStore.Get(), and updates CapabilityStore.Set() only if something changed. Auth state in SessionAuthStore is never affected.

• Singleflight-deduplicated via the existing notifRefreshGroup with key <sessionID>/<serverName>, coalescing with notification-triggered refreshes.

• Authenticated servers with no active pooled clients are skipped entirely (no clients to poll with).

Lifecycle

• Start: Add a wg.Add(1); go a.runCapabilityPoller() in AggregatorServer.Start(), similar to the existing monitorRegistryUpdates and runSSOTrackerCleanup goroutines.
• Stop: The poller respects a.ctx.Done() and exits, counted by a.wg.
• Context: Use a.refreshContext() (same as notification subscriber) for the refresh operations.

Poll interval

• Default: 5 minutes (DefaultCapabilityPollInterval = 5 * time.Minute)
• The poller iterates all registered servers each tick. A full pass re-fetches capabilities from every connected server.
• Singleflight deduplication ensures that if a notification-triggered refresh just ran for a server, the poll's attempt for that server is a no-op.

Shared singleflight group with Layer 2

The notification subscriber already uses notifRefreshGroup for deduplication. The poller should use the same singleflight group so that:

• A notification-triggered refresh that just completed prevents the poll from redundantly re-fetching the same server.
• A poll in progress prevents a concurrent notification from triggering a duplicate re-fetch.
• The dedup keys are already well-defined: notif-caps/<serverName> for non-authenticated, <sessionID>/<serverName> for authenticated.

Code reuse from Layer 2

The poller reuses existing methods with zero duplication:

• refreshNonOAuthCapabilities(serverName) for non-authenticated servers
• refreshSessionCapabilities(ctx, serverName, sessionID, client) for authenticated servers
• toolListsEqual, resourceListsEqual, promptListsEqual for deep comparison (used internally by the refresh methods)

Interaction with Layer 4 (health tracking)

The poll results are natural health observation points. If Layer 4 is implemented (or when it is):

• Successful ListTools -> mark server healthy
• Failed ListTools (network error) -> increment failure counter
• The poller doubles as a health probe, eliminating the need for a separate health check loop

This is explicitly not in scope for this issue but is a documented extension point.

Implementation plan

1. Add GetSessionsForServer(serverName string) []PooledSession to SessionConnectionPool
2. Create capability_poller.go with runCapabilityPoller() loop
3. Wire into AggregatorServer.Start() / Stop() lifecycle
4. Add unit tests for the poller (mock server, verify polling triggers refresh)
5. Add BDD test scenario: non-authenticated server tool change detected by polling (no notification sent by the mock server, change detected by periodic poll)

Acceptance criteria

• Background poller runs every 5 minutes (configurable default) and re-fetches capabilities from all connected servers
• Non-authenticated servers: stale tools are detected and updated via refreshNonOAuthCapabilities
• Authenticated servers: active pooled sessions are iterated, stale tools detected and updated via refreshSessionCapabilities
• Authenticated servers with no active pooled clients are skipped (no-op)
• Singleflight deduplication is shared with the notification subscriber to avoid redundant fetches
• No auth state is affected by capability polling (AuthStore is never touched)
• Poller respects aggregator lifecycle (starts with Start(), stops with Stop())
• Network errors during polling are logged but do not clear existing cached capabilities
• Unit tests cover poller lifecycle, non-authenticated refresh, authenticated session iteration, and singleflight dedup
• BDD scenario validates silent redeployment detection (tool change without notification)
• Min 80% test coverage for new code

Files to create/modify

• Create: internal/aggregator/capability_poller.go -- poller loop and lifecycle
• Create: internal/aggregator/capability_poller_test.go -- unit tests
• Modify: internal/aggregator/session_connection_pool.go -- add GetSessionsForServer
• Modify: internal/aggregator/session_connection_pool_test.go -- test GetSessionsForServer
• Modify: internal/aggregator/server.go -- wire poller into Start()/Stop() lifecycle
• Create: internal/testing/scenarios/poll-capability-refresh-nonoauth.yaml -- BDD test scenario

Dependencies

• No dependency on Layer 4 (health tracking). Can be implemented in parallel.
• Uses Layer 2 infrastructure (comparison helpers, refresh methods, singleflight group).
• Uses Layer 6 infrastructure (SessionConnectionPool for authenticated server iteration).