[FEATURE]: Auto-compress context and retry on context_length_exceeded errors #37443

Feature hasn't been suggested before.

• I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request

When running long-context workflows or multi-turn agent sessions, the LLM API occasionally throws a context_length_exceeded error (typically an HTTP 400). Currently, this error immediately crashes the entire execution pipeline.

To make the system more resilient, it would be great if the engine could catch this specific error, automatically compress or truncate the historical messages, and then retry the task seamlessly instead of failing out completely.

Suggested Solution

We could introduce an automatic recovery mechanism in the LLM request/middleware layer:

1. Error Interception: Catch the API exception and check if the error code or message indicates a context_length_exceeded event.
2. Context Reduction: Trigger a configurable compression strategy on the current message list, such as:
   • Sliding Window: Drop the oldest N rounds of dialogue (while preserving the system prompt).
   • Summary Fallback: Summarize older messages using a faster model and replace the raw history with this summary.
3. Auto-Retry: Re-estimate tokens and automatically re-submit the request to complete the pending task.

Expected Behavior

Ideally, this can be handled via a simple configuration. For example:

{
  "llm_retry_on_context_limit": true,
  "context_compression_strategy": "sliding_window",
  "max_retry_attempts": 2
}

When enabled, the system should log a warning (e.g., Context limit exceeded. Retrying with compressed history...) and proceed to return the final response to the user without throwing an unhandled exception.

Alternatives Considered

Handling this manually at the application level requires writing repetitive boilerplate code around every single LLM call. Supporting this natively in the core request loop would be a much cleaner and more robust solution for everyone.