[fix] Deduplicate llms.txt pages before applying the URL limit - #10671

Summary

Fixes #10670.

With llms.txt entries A, A, B and max_urls=2, the reader fetches A twice, emits it twice, and skips B. Deduplicate resolved URLs before applying the limit in both sync and async reads. Preserve the first entry's metadata and ordering; leave the index parser unchanged.

Type of change

• Bug fix
• New feature
• Breaking change
• Improvement
• Model update
• Other:

Checklist

• Code complies with style guidelines
• Ran format/validation scripts (./scripts/format.sh and ./scripts/validate.sh)
• Self-review completed
• Documentation updated (comments, docstrings)
• Examples and guides: Relevant cookbook examples have been included or updated (if applicable)
• Tested in clean environment
• Tests added/updated (if applicable)

Duplicate and AI-Generated PR Check

• I have searched existing open pull requests and confirmed that no other PR already addresses this issue
• If a similar PR exists, I have explained below why this PR is a better approach
• Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.)

Additional Notes

Nine new cases cover sync/async reads, limits of 0/1/2/20, relative/absolute aliases, duplicates across sections, request counts, document contents, metadata, and unchanged index-parser output. HTTP responses use httpx.MockTransport, without live network or model calls.

• New tests on unchanged main: 4 failed, 5 passed.
• python -m pytest libs/agno/tests/unit/reader/test_llms_txt_duplicate_urls.py libs/agno/tests/unit/tools/test_llms_txt.py -q: 73 passed.
• ./scripts/format.sh, ./scripts/validate.sh, and git diff --check: passed, including full Agno/agnoctl mypy and cookbook checks.

Validation used the existing macOS ARM64/Python 3.12.14 development environment. The full repository test suite and live integrations were not run. URL equality is based on the parser's resolved URL strings; this change does not introduce broader URL canonicalization or change allowed-host policy. Existing PR #10207 addresses asynchronous chunking, a separate behavior; its source diff does not deduplicate links.

Generated by Codex with the account holder's authorization. Codex performed reproduction, code review, and checks; no independent human code review is claimed. No public API or cookbook change is needed.