core(is-crawlable): only warn if some bots are blocked #14550
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamraine
approved these changes
Jan 11, 2023
| /** | ||
| * Returns true if any of provided directives blocks page from being indexed | ||
| * @param {string} directives | ||
| * @param {string} directives assumes no user-agent prefix |
Contributor
There was a problem hiding this comment.
Suggested change
| * @param {string} directives assumes no user-agent prefix | |
| * @param {string} directives (assumes no user-agent prefix) |
Collaborator
Author
There was a problem hiding this comment.
This is proper jsdoc syntax. IDE makes it look like this on hover:
and not hovered color should be different
connorjclark
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ref #5329 (can't close until we update docs)
This audit will no longer fail if some known crawler is explicitly allowed while others are blocked. Instead, it will emit a warning say which user agents have been blocked. Details items are only present when the audit is failing, and only show blocking directives wrt a generic UA.
here's a really big list of crawler UAs ... https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json . What all should we explicitly look for?