[Bug]: Unhandled HTTP 401 in RestApiTool traps chat sessions in unrecoverable auth loops #7085

🔴 Required Information

Describe the Bug:
When an authenticated RestApiTool (e.g., from OpenAPIToolset) encounters an HTTP 401 Unauthorized response from an upstream API because the access token has expired or was revoked and refresh fails/is unavailable, RestApiTool.call() catches httpx.HTTPStatusError generically and returns a text error string ({"error": "Tool ... execution failed ... Status Code: 401 ..."}) to the LLM agent.

This applies equally to both OpenID Connect (OPEN_ID_CONNECT) and standard OAuth 2.0 (OAUTH2), which share the same underlying token exchange, bearer authorization, and OAuth2CredentialRefresher architecture in Google ADK.

ADK does not evict the dead token from tool_context.state. Because the dead credential remains cached in the session state (persisted across turns via DatabaseSessionService), all subsequent user messages in that chat session continue reusing the expired token. The session enters an unrecoverable 401 loop, never re-prompting the user for authorization (adk_request_credential).

Beyond trapping the individual user in a broken session, this persistent 401 loop causes trust degeneration with security firewalls, API gateways, and WAFs (e.g., automated threat detection flagging repeated unauthorized requests). In multi-tenant environments where agent runners share egress IPs or gateways, firewall-level IP penalties and rate-limiting can cascade and disrupt other active users who have valid tokens.

Crucially, in oauth2_credential_refresher.py (lines 125–127), the code explicitly states:

except (AuthlibBaseError, requests.RequestException) as e:
    # Non-fatal: keep the stale token so its eventual 401
    # re-triggers auth.
    logger.error("Failed to refresh OAuth2 tokens: %s", e)
    return auth_credential

ADK framework designers explicitly intended for an eventual 401 to re-trigger auth, but RestApiTool never executes credential eviction or auth re-triggering. Furthermore, ToolContextCredentialStore.remove_credential() is dead code that calls del self.tool_context.state[key], which crashes at runtime with AttributeError: __delitem__ because google.adk.sessions.state.State does not implement __delitem__.

Steps to Reproduce:

1. Configure an agent with an OpenAPI toolset (RestApiTool) requiring OIDC or OAuth2 authentication (AuthCredentialTypes.OPEN_ID_CONNECT or AuthCredentialTypes.OAUTH2).
2. Run the agent using adk api_server with persistent storage (e.g. DatabaseSessionService / PostgreSQL).
3. Send a user query requiring the tool. ADK emits adk_request_credential.
4. Supply credentials via FunctionResponse (queryAuthConfirm). The tool executes successfully and credentials are saved in session state under openIdConnect_<scheme_hash>_... or oauth2_<scheme_hash>_....
5. Invalidate the access token (or let access and refresh tokens expire, e.g., leave the session idle overnight).
6. Send another query in the same session requiring the authenticated tool.

Expected Behavior:

1. On receiving HTTP 401 from an authenticated endpoint (whether OIDC or OAuth2), RestApiTool / ToolAuthHandler should evict or tombstone the stale credential key from tool_context.state.
2. It should either attempt an immediate refresh via OAuth2CredentialRefresher, or pause execution by calling tool_context.request_credential() and returning {"pending": True} so the runner emits adk_request_credential.
3. If not re-triggered in the active turn, the subsequent turn should at least see an empty credential store and trigger re-authorization, rather than continuing to blast the protected API with invalid credentials.

Observed Behavior:

• RestApiTool catches httpx.HTTPStatusError and returns {"error": "Tool <name> execution failed... Status Code: 401..."} to the LLM.
• The LLM outputs an apology to the user ("Authentication failed").
• The dead token remains cached in tool_context.state and in persistent database session storage.
• When the user sends any follow-up message, ToolAuthHandler._get_existing_credential() re-fetches the expired token from state and repeatedly attempts the call with the exact same invalid token.
• The chat session is stuck in a permanent HTTP 401 loop until the entire session is manually deleted.
• Continuous unauthorized calls degrade the client/gateway trust score with upstream enterprise firewalls, risking IP blocking and impacting concurrent active users with valid credentials.

Environment Details:

• ADK Library Version: google-adk 2.8.0
• Desktop OS: Windows 11 (AMD64) / Debian GNU/Linux 12 (Docker)
• Python Version: 3.13.14

Model Information:

• Are you using LiteLLM: No
• Which model is being used: gemini-2.5-flash / gemini-2.5-pro

🟡 Optional Information

Regression:
Unknown. However, inspection of google.adk.tools.openapi_tool.openapi_spec_parser.tool_auth_handler.ToolContextCredentialStore reveals:

def remove_credential(self, key: str):
    del self.tool_context.state[key]

This method is dead code (never called anywhere in ADK). Moreover, State in google.adk.sessions.state does not implement __delitem__, meaning executing this method raises AttributeError: __delitem__.

Logs:

WARNING:google.adk.tools.openapi_tool.openapi_spec_parser.rest_api_tool:API call failed for tool qualicision_qai_tool: Status 401 - {"detail":"Invalid or expired access token"}
DEBUG:google.adk.flows.llm_flows.functions:Function execution failed with error: Tool qualicision_qai_tool execution failed. Analyze this execution error and your inputs. Retry with adjustments if applicable. But make sure don't retry more than 3 times. Execution Error: Status Code: 401, {"detail":"Invalid or expired access token"}

Additional Context:

• Unified Impact across OIDC & OAuth2:
  Because OIDC is built directly on OAuth 2.0, Google ADK uses the exact same OAuth2Credential model, the same OAuth2CredentialRefresher, and the same bearer header injection for both OPEN_ID_CONNECT and OAUTH2 schemes. This defect affects all token-based OpenAPI/REST tools equally.
• Security & Network Risk (Firewall Trust Degeneration):
  Repeated 401 Unauthorized bursts from the agent container to protected APIs trigger anomaly detection and rate-limiting rules in enterprise Web Application Firewalls (WAF) and API gateways. In environments where the agent runs behind a shared gateway or egress IP, this trust score degeneration can lead to automated IP-level throttling or blocking, causing outages for other active users who have valid tokens.
• Session State Key Formats:
  openIdConnect_<scheme_hash>_openIdConnect_<cred_hash>_existing_exchanged_credential
oauth2_<scheme_hash>_oauth2_<cred_hash>_existing_exchanged_credential
• Workaround:
  In our API gateway/proxy layer, intercepting the 401 event and sending PATCH /apps/{app_name}/users/{user_id}/sessions/{session_id} with {"state_delta": {"<credential_key>": {}}} successfully clears the cache and allows the session to recover on the next prompt.
• Root Cause Code Locations:
  1. google/adk/tools/openapi_tool/openapi_spec_parser/rest_api_tool.py: Lines 582–600 (call method does not inspect 401 status or call auth eviction).
  2. google/adk/tools/openapi_tool/openapi_spec_parser/tool_auth_handler.py: Lines 159–161 (remove_credential never called, and calls unsupported del state[key]).
  3. google/adk/sessions/state.py: Lines 61–136 (State class lacks __delitem__ and delta tombstone support).

Minimal Reproduction Code:

This issue consists of two coupled defects, both verified with minimal isolated scripts:

Repro 1: RestApiTool swallows HTTP 401 without evicting the token from state

import asyncio
import httpx
from unittest.mock import MagicMock
from google.adk.tools.openapi_tool.openapi_spec_parser.rest_api_tool import RestApiTool
from google.adk.tools.openapi_tool.openapi_spec_parser.openapi_spec_parser import OperationEndpoint, Operation

async def test_rest_api_tool_swallows_401():
    # Mock upstream API returning 401 Unauthorized
    def mock_factory():
        transport = httpx.MockTransport(lambda req: httpx.Response(401, json={"detail": "Token expired"}))
        return httpx.AsyncClient(transport=transport)

    endpoint = OperationEndpoint(base_url="http://example.com", path="/data", method="get")
    operation = Operation(responses={"200": {"description": "OK"}})

    tool = RestApiTool(
        name="test_api_tool",
        description="test tool",
        endpoint=endpoint,
        operation=operation,
        httpx_client_factory=mock_factory
    )

    mock_ctx = MagicMock()
    mock_ctx.state = {"openIdConnect_credential_key": {"access_token": "expired_token"}}
    mock_ctx.actions = MagicMock()
    mock_ctx.function_call_id = "call_abc"

    result = await tool.call(args={}, tool_context=mock_ctx)
    
    # 1. Swallowed into error dictionary for LLM
    assert "error" in result
    assert "Status Code: 401" in result["error"]
    
    # 2. Defect: The dead token is NOT evicted from session state
    assert "openIdConnect_credential_key" in mock_ctx.state

asyncio.run(test_rest_api_tool_swallows_401())

Repro 2: ToolContextCredentialStore.remove_credential() crashes with AttributeError on State

import pytest
from google.adk.sessions.state import State
from google.adk.tools.openapi_tool.openapi_spec_parser.tool_auth_handler import ToolContextCredentialStore
from unittest.mock import MagicMock

def test_state_does_not_support_delitem():
    """Demonstrates that ToolContextCredentialStore.remove_credential() crashes on State."""
    state = State(
        value={"openIdConnect_foo_existing_exchanged_credential": {"access_token": "expired"}},
        delta={},
    )
    tool_context = MagicMock()
    tool_context.state = state

    store = ToolContextCredentialStore(tool_context)
    
    # This raises AttributeError: __delitem__
    with pytest.raises(AttributeError):
        store.remove_credential("openIdConnect_foo_existing_exchanged_credential")

How often has this issue occurred?:

• Always (100%) whenever an authenticated session's access and refresh tokens expire or are revoked upstream.