Skip to content

fix: retry web searches after key-specific failures - #10877

Merged
Soulter merged 4 commits into
AstrBotDevs:masterfrom
wozhenbang2004:fix/10871-search-key-failover
Nov 1, 2026
Merged

Soulter merged 4 commits into
AstrBotDevs:masterfrom
wozhenbang2004:fix/10871-search-key-failover

Conversation

@wozhenbang2004

@wozhenbang2004 wozhenbang2004 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #10871.

A request can currently fail on an exhausted or invalid search key even when another configured key works. This adds the missing provider-specific failures to key failover and applies the existing bounded retry pattern to Exa and DeepSeek.

Modifications

  • Retry Firecrawl 402 and Tavily 433 responses with the next key for both search and extraction.

  • Retry Brave 422 only when the response identifies SUBSCRIPTION_TOKEN_INVALID; validation errors and malformed error bodies still fail immediately.

  • Add Exa search/contents and DeepSeek search/fetch failover for 401, 402 and 429. Preserve the last error after all configured keys fail, and keep other errors, including DeepSeek 403 refusals, non-retryable.

  • Add regression coverage for successful failover, exhausted keys, unchanged request payloads and non-key errors.

  • This is not a breaking change.

Test Results

Changes were made in the GitHub web editor. The submitted source was checked against the intended changes and the diff is limited to the search tools and their unit tests. Pytest and Ruff have not been run locally; repository CI is awaiting maintainer approval.

Verification steps:

uv run pytest tests/unit/test_web_search_tools.py
uv run ruff format --check astrbot/core/tools/web_search_tools.py
uv run ruff check astrbot/core/tools/web_search_tools.py

Checklist

  • The tests and verification commands above have passed.
  • No WebUI navigation, labels or documentation workflows change, so no WebUI documentation update is needed.
  • No new dependencies are introduced.
  • The changes do not introduce malicious code.

Summary by Sourcery

Improve web-search reliability by failing over between configured API keys only for provider-specific authentication, quota, and rate-limit failures.

Bug Fixes:

  • Retry key-specific failures across configured web-search provider keys, including Firecrawl, Tavily, Exa, and DeepSeek, while preserving non-key errors and the final failure details.
  • Restrict Brave 422 failover to responses identifying an invalid subscription token.

Enhancements:

  • Make key rotation request-local and resilient to duplicate keys or concurrent requests while retaining round-robin behavior.

Tests:

  • Add regression coverage for provider failover, exhausted keys, request payload preservation, interleaved key rotation, and non-retryable errors.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/tools/web_search_tools.py" line_range="258" />
<code_context>
                 reason = await response.text()
                 # Retryable errors are saved so the final failure is meaningful.
-                if response.status in _RETRYABLE_HTTP_STATUSES:
+                if response.status in _TAVILY_RETRYABLE_HTTP_STATUSES:
                     last_error = Exception(
                         f"Tavily web search failed: {reason}, status: {response.status}",
</code_context>
<issue_to_address>
**Usable search keys go untried**

When concurrent requests advance the same provider's rotator between a failed attempt and its retry, the Tavily, Exa, and DeepSeek retry loops call the shared `_KeyRotator.get` again, so a request can reuse a key it already tried and exhaust its attempts without trying a usable configured key; the caller sees a search or fetch failure.

Track keys tried by each request and select an untried key for each retry instead of reacquiring one from the shared rotator.

Also at `astrbot/core/tools/web_search_tools.py:314`, `astrbot/core/tools/web_search_tools.py:442-453`, `astrbot/core/tools/web_search_tools.py:1069-1070`, `astrbot/core/tools/web_search_tools.py:1125`, `astrbot/core/tools/web_search_tools.py:1636-1637`, `astrbot/core/tools/web_search_tools.py:1654`, `astrbot/core/tools/web_search_tools.py:1698-1699`, `astrbot/core/tools/web_search_tools.py:1716`, `astrbot/core/tools/web_search_tools.py:2172-2173`, `astrbot/core/tools/web_search_tools.py:2190`, `astrbot/core/tools/web_search_tools.py:2278-2279`, `astrbot/core/tools/web_search_tools.py:2296`.
</issue_to_address>

### Comment 2
<location path="tests/unit/test_web_search_tools.py" line_range="1837" />
<code_context>
+        with pytest.raises(Exception, match=f"last key failed, status: {status}"):
+            await search_function(settings, payload)
+    else:
+        assert await search_function(settings, payload)
+
+    header = "x-api-key" if provider == "exa" else "Authorization"
</code_context>
<issue_to_address>
**Wrong success result passes**

When a retry returns a nonempty but incorrect value, or a string containing “Example” that is not the expected result, `test_search_key_errors_try_next_key` accepts any truthy result at line 1837, and `test_deepseek_key_errors_try_next_key` accepts any string containing “Example” at line 1950; a bogus fallback or error string therefore passes despite the second-key result being wrong.

Assert the expected result from the mocked successful response, rather than only truthiness or a substring.

Also at `tests/unit/test_web_search_tools.py:1950`.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and on selected provider errors, the tool now sends the same search or fetch request using additional configured API keys; if a retry succeeds, the provider may meter that request. Reverting stops future retries but cannot undo requests already sent or any usage charges they incurred.

Blocking findings: astrbot/core/tools/web_search_tools.py:258, tests/unit/test_web_search_tools.py:1837


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread astrbot/core/tools/web_search_tools.py Resolved
Comment thread tests/unit/test_web_search_tools.py Resolved
…keys after concurrent rotation

Updated the get method to include tried_keys parameter for tracking used API keys.
Cover concurrent cursor advancement, duplicate configured keys, exhaustion errors, and exact successful provider outputs.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sourcery assessment

Approved.

@Soulter
Soulter merged commit 59e44a7 into AstrBotDevs:master Nov 1, 2026
23 checks passed
@Soulter Soulter moved this to Done in AstrBot Roadmap Nov 2, 2026
ilya-bogin-keenable added a commit to keenableai/AstrBot that referenced this pull request Nov 3, 2026
Follow the request-local tried-keys rotation from AstrBotDevs#10877, so duplicate keys and concurrent requests do not make a failover retry a key it already tried.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Bug] 部分网页搜索多 Key 轮询遇到额度耗尽或 Key 失效时不重试

2 participants