[fix] Count eligible pages toward the sitemap discovery cap - #10668

Summary

Fixes #10667.

With max_pages=2, a sitemap index whose first shard lists the same page twice currently skips the second shard and returns only one page. Off-host entries can similarly consume the entire budget and prevent valid later pages from being read.

Count unique, host-allowed canonical page URLs when deciding whether to open another shard. Keep this additional set bounded by max_pages. The existing final output ordering and incomplete-discovery handling continue to apply.

Type of change

• Bug fix
• New feature
• Breaking change
• Improvement
• Model update
• Other:

Checklist

• Code complies with style guidelines
• Ran format/validation scripts (./scripts/format.sh and ./scripts/validate.sh)
• Self-review completed
• Documentation updated (comments, docstrings)
• Examples and guides: Relevant cookbook examples have been included or updated (if applicable)
• Tested in clean environment
• Tests added/updated (if applicable)

Duplicate and AI-Generated PR Check

• I have searched existing open pull requests and confirmed that no other PR already addresses this issue
• If a similar PR exists, I have explained below why this PR is a better approach
• Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.)

Additional Notes

Ten new regression cases exercise sync and async reads with duplicate URLs, canonical aliases, filtered hosts, mixed entries, and duplicates across shards. They also verify page order, complete-discovery metadata, and that later shards remain unread once the actual page cap is reached. All HTTP is mocked at the transport boundary; the real discovery and page-fetching code runs.

• Unchanged main (4db4180), sitemap suite with added cases: 10 failed, 52 passed.
• Fixed sitemap and page-fetcher suites: 91 passed.
• ./scripts/format.sh, ./scripts/validate.sh, and git diff --check: passed.

Validation environment: macOS ARM64, Python 3.12.14, pytest 9.1.1, Ruff 0.15.20, mypy 2.1.0, SQLAlchemy 2.0.54. Full static validation covers Agno, agnoctl, and cookbook checks. The full repository test suite and live integrations were not run locally.

Generated by Codex with the account holder's authorization; Codex performed the reproduction, diff review, and checks. No independent human code review is claimed. No public API or cookbook changes are needed.