Skip to content

[Feature]: Expose agent metadata (tool call count, token usage) in message_sending plugin hook #21184

Description

@Milofax

Summary

Allow message_sending plugin hooks to access agent-level metadata — specifically tool call count and token usage — so plugins can implement pre-send validation guards (e.g., block "done" claims without verification, warn on high context fill).

Problem to solve

Plugin hooks registered for message_sending can already cancel or modify outbound messages. However, the hook event only contains delivery-level data (to, content, metadata: { channel, accountId, mediaUrls }). There's no way to know:

  1. How many tools the agent used in the current turn — needed for verification guards ("agent says done but never called any tools")
  2. Current token usage vs context window — needed for context-size guards ("context is 72% full, agent should summarize")

This data exists in the agent subscription handler (pi-embedded-subscribe.ts: toolMetas, usageTotals, params.model.contextWindow) but is not threaded through to the delivery pipeline.

Proposed solution

Add an optional agentMetadata field to PluginHookMessageSendingEvent:

export type PluginHookMessageSendingEvent = {
  to: string;
  content: string;
  metadata?: Record<string, unknown>;
  agentMetadata?: {
    toolCallCount: number;
    tokenUsage?: { input: number; output: number; total: number };
    contextWindow?: number;
    agentId?: string;
  };
};

Threading: subscription handler → block reply callback → delivery params → runMessageSending() event.

Important timing consideration: During streaming, usageTotals may not be finalized yet (populated on message_end). Options:

  • Only populate agentMetadata for non-streaming deliveries (final response)
  • Populate with "last known" values (stale but non-zero after first turn)
  • Add a streaming: boolean flag so plugins know the data may be incomplete

Alternatives considered

  1. New internal hook message:pre-send — Rejected: internal hooks are fire-and-forget (errors swallowed), not suitable for blocking. Plugin hooks have proper cancel/modify semantics.
  2. Piggybacking on channelData — Rejected: channelData is explicitly for per-channel envelope data (Discord buttons, Telegram keyboards). Agent metadata there violates the type contract.
  3. Post-send via message:sent — Works for auditing but can't block/modify. Useful as complement, not replacement.
  4. extraDirs hooks for plugin hooks — Not possible today; extraDirs only loads internal hooks. Would be a separate feature request.

Impact

  • Affected: Self-hosted deployments with custom quality/compliance requirements
  • Severity: Medium — workaround exists (prompt-based rules via bootstrap hook, ~60-70% compliance)
  • Frequency: Every agent response
  • Consequence without: Can't build reliable pre-send validation. Prompt-only approach has no enforcement.

Evidence/examples

Use case: 12-agent deployment where agents claim tasks are "done" without calling any verification tools. Bootstrap-injected quality standards help (~60-70%) but can't enforce.

Existing code references:

  • message_sending plugin hook: src/infra/outbound/deliver.ts:498-521
  • PluginHookMessageSendingEvent: src/plugins/types.ts:447-457
  • Tool tracking: pi-embedded-subscribe.ts (toolMetas, usageTotals)
  • Context window: pi-embedded-runner/run/attempt.ts:596-601

Additional information

We're happy to contribute a PR if the approach is agreed upon. The main question is the preferred threading path for agent metadata to the delivery pipeline — whether via delivery params, a shared session store, or a different mechanism.

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions