Skip to content

fix(drift): recognize none as a noindex directive - #404

Merged
AgriciDaniel merged 1 commit into
AgriciDaniel:mainfrom
Afloat16:fix/drift-none-directive-20261103
Nov 10, 2026
Merged

AgriciDaniel merged 1 commit into
AgriciDaniel:mainfrom
Afloat16:fix/drift-none-directive-20261103

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

The noindex drift rule only searches for the substring noindex. It therefore misses a newly added none directive, and reports a false new restriction when an existing none is replaced with noindex. Google documents none as equivalent to noindex, nofollow: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#none

Compare complete, case-insensitive directives and treat none and noindex as equivalent indexing restrictions. Parameterized values such as max-image-preview:none and max-image-preview: none do not become indexing restrictions. Independent directives still work with comma or whitespace separation. Regression cases also check that a previous parameter value cannot hide a newly added none.

Validation

  • Before the fix: 6 new regression cases fail.
  • python -m pytest -q tests/test_drift_robots_directives.py tests/test_drift_portability.py: 17 passed.
  • ruff check scripts/drift_compare.py tests/test_drift_robots_directives.py: passed.
  • git diff --check: passed.

Tests cover the rule directly without network access.

@AgriciDaniel
AgriciDaniel merged commit 6675c49 into AgriciDaniel:main Nov 10, 2026
myforkiq Bot pushed a commit to bguisad/claude-seo that referenced this pull request Nov 10, 2026
… BOM handling

Follow-ups to AgriciDaniel#332, AgriciDaniel#385, AgriciDaniel#345, AgriciDaniel#384 and AgriciDaniel#404:
- one malformed URL in an image or link no longer aborts the parse; bad
  values are listed under malformed_urls
- the image probe reports a malformed src or a non-image response as
  unverified and no longer labels arbitrary XML as SVG
- <base href> is honoured for http and https only; fragment-only links
  follow a base that points elsewhere (raised by agammann on AgriciDaniel#385)
- <br> in headings and link text reads as a space
- a plain-text sitemap that starts with a UTF-8 BOM is valid
- drift reads semicolon-separated robots directives

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants