You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7e76f20
Browse filesBrowse the repository at this point in the historyBrowse files
fix(deep-crawl): cap each BFS level to the remaining max_pages budget
Siblings in a level queue links before the level's own pages are all
counted, so the next level could still exceed max_pages in a real batch
crawl (83 pages for max_pages=50). Cut each level to the budget left
before fetching it, in both batch and stream mode. Add tests that run the
real batch and stream loops against a fake crawler.
0 commit comments