feat(docs): allow to configure noIndex per doc version#7963
Conversation
✅ [V2]
To edit notification comments on pull requests, go to your Netlify site settings. |
⚡️ Lighthouse report for the deploy preview of this PR
|
Josh-Cena
left a comment
Josh-Cena
left a comment
There was a problem hiding this comment.
Question: how does this play with the sitemap? Over there, we only test tag.props.name === 'robots' && tag.props.content === 'noindex'. Maybe it should be tag.props.content.includes("noindex") instead?
✅ [V2]
To edit notification comments on pull requests, go to your Netlify site settings. |
|
Size Change: +45 B (0%) Total Size: 813 kB
ℹ️ View Unchanged
|
Yes definitively! I thought it was already the case actually 😅 will send another PR and edited our sitemap plugin so that we can see the problem. Will merge a sitemap plugin fix before this PR, ensuring we can see the problem fixed here. |
|
BTW I guess having a noIndex: true flag for plugin instances in a generic way could be useful 🤷♂️ we also want to have a label, wonder what else makes sense as a general option for a plugin instance. Not going to implement this here, but let's keep this in mind |
slorber
added
the
to backport
|
working fine now those tag urls are still there in the sitemap and indexed, will do another refactor to fix that <url>
<loc>https://docusaurus.io/tests/docs/tags/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/a/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/b/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/c/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/d-custom-permalink/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/e/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/some-tag/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url> |
slorber
added
backported
3 participants
Motivation
You might want to prevent crawlers and google to index older versions of your docs
See also #7960
Test Plan
unit tests + local tests + dogfood on test docs instance (as it's not really useful to index it)
Test links
Deploy preview:
Limitation: the versioned docs tag pages won't be noIndexed. But a further refactor will fix it soon. (fixed in #7966)