-
-
Notifications
You must be signed in to change notification settings - Fork 39.7k
Description
Problem
sanitizeUserFacingText() in src/agents/pi-embedded-helpers/errors.ts uses regex and substring matching on error text to classify errors (billing, rate-limit, timeout, etc.). This causes false positives when non-billing errors happen to contain billing-adjacent keywords.
Example: A tool call fails with "Error: Got a 402 from the upstream API". The billing regex \b(?:got|returned|received)\s+(?:a\s+)?402\b matches, and the user receives:
"API provider returned a billing error -- your API key has run out of credits..."
...instead of the actual error. The 402 came from an upstream tool (e.g., Brave Search), not the LLM provider's billing system.
Why heuristic fixes are insufficient
At least 10 PRs have attempted to fix this with better heuristics (#12777, #12702, #12226, #8661, #12720, #11680, #12052, #13318, #13467, #15109). Each one either:
- Adds another guard that can still be fooled by certain text patterns
- Fixes one false-positive path while leaving others open
- Places the fix in the wrong scope or function
The merged errorContext guard (#12988) helps by gating rewrites behind { errorContext: true }, but the fundamental problem remains: error type information that exists at the source (provider adapters know the HTTP status code and its origin) is discarded before reaching the sanitizer.
Proposed solution: thread errorKind through the pipeline
Instead of inferring error type from text, classify it at the source and pass it through:
// Provider adapter (source of truth):
errorKind: "billing" | "rate_limit" | "timeout" | "context_overflow" | "unknown"
// Or lighter-weight:
errorSource: "provider" | "tool" | "system"The sanitizer would then use the structured classification directly instead of pattern-matching on text:
// Instead of: regex matching "402" in arbitrary text
// Do: if (opts.errorKind === "billing") return BILLING_ERROR_USER_MESSAGE;Key files/interfaces that would need changes
src/agents/pi-embedded-helpers/errors.ts—sanitizeUserFacingText()signature and billing/rate-limit rewrite logicReplyPayloadtype — adderrorKindfield- Provider adapters — classify errors at the source where HTTP status codes are available
normalize-reply.ts— pass classification through to sanitizeragent-runner-execution.ts— pass classification through to sanitizer
Scope
This is a future refactor proposal, not blocking current pragmatic fixes. PRs like #12777 (structural guard) are still valuable as incremental improvements. This issue tracks the longer-term architectural fix.
Related issues
- [Bug]:
sanitizeUserFacingText()matches "402" in normal assistant responses, replacing them with billing error warning #12711 — canonical root-cause analysis - sanitizeUserFacingText false-positive: normal assistant replies containing billing keywords get replaced with billing error message #11649 — normal replies containing billing keywords get replaced
- sanitizeUserFacingText false-positive: normal assistant responses about billing/pricing replaced with billing error warning #12676 —
sanitizeUserFacingTextfalse-positive - Bug: Tool failures (e.g., Brave Search 429) incorrectly surfaced as 'billing error' in Telegram #14245 — tool failures (Brave Search 429) surfaced as billing errors
- Billing error false positive from x402 HTTP 402 responses #13888 — x402 HTTP 402 responses triggering false positives
- [Bug]: False billing error: \b402\b regex matches legitimate content #16237 —
\b402\bregex matching legitimate content