{"id":951,"date":"2026-05-11T09:02:52","date_gmt":"2026-05-11T09:02:52","guid":{"rendered":"https:\/\/docs.wpultimatesecurity.com\/?post_type=docs&p=951"},"modified":"2026-05-11T09:02:53","modified_gmt":"2026-05-11T09:02:53","password":"","slug":"block-crawlers-wp-paths","status":"publish","type":"docs","link":"https:\/\/docs.wpultimatesecurity.com\/docs\/setup-rules\/block-crawlers-wp-paths\/","title":{"rendered":"Block Crawlers & WP Paths"},"content":{"rendered":"\n

The Block Crawlers & WP Paths<\/strong> feature helps protect your WordPress site from aggressive bots, exploit scanners, suspicious crawlers, and requests targeting sensitive WordPress files. It runs before blocking rules so good bots are never caught by them.<\/p>\n\n\n\n

\"block<\/figure>\n\n\n\n

Enable Block Aggressive Crawlers & WP Paths<\/h2>\n\n\n\n

Turn on the main toggle to activate this protection module. Once enabled, Ultimate Security will start applying the selected crawler and path protection rules. The settings have an Enable All<\/strong> and Disable All<\/strong> button for quick bulk control, plus individual toggles for each service.<\/p>\n\n\n\n

Aggressive Crawlers<\/h2>\n\n\n\n

This section blocks known crawlers and bots that may heavily scan your website, consume bandwidth, or collect data aggressively.<\/p>\n\n\n\n

\"aggressive<\/figure>\n\n\n\n

Available Rules<\/strong>:<\/p>\n\n\n\n

    \n
  • Yandex<\/strong> (Safe to block unless you target Russian-speaking audiences.)<\/li>\n\n\n\n
  • Sogou<\/strong> (Safe to block unless you target Chinese-speaking audiences.)<\/li>\n\n\n\n
  • SEMrush<\/strong> (If you use SEMrush, allow it in Allow Good Bots instead.)<\/li>\n\n\n\n
  • Ahrefs<\/strong> (If you use Ahrefs, allow it in Allow Good Bots instead.)<\/li>\n\n\n\n
  • Baidu<\/strong> (Safe to block unless you need Chinese search visibility.)<\/li>\n\n\n\n
  • Neevabot<\/strong> (Safe to block because the service is no longer operational.)<\/li>\n<\/ul>\n\n\n\n

    Generic Bot Patterns<\/h2>\n\n\n\n

    This section blocks suspicious requests that match common bot-related patterns.<\/p>\n\n\n\n

    \"generic<\/figure>\n\n\n\n

    Available Rules<\/strong>:<\/p>\n\n\n\n

      \n
    • Python Requests<\/strong> (Blocks requests using the python-requests user agent. Most legitimate services use custom agents.)<\/li>\n\n\n\n
    • Generic “crawl” in User-Agent<\/strong> (Blocks user agents containing “crawl” except verified Cloudflare bots.)<\/li>\n\n\n\n
    • Generic “bot” in User-Agent<\/strong> (Blocks user agents containing “bot” except verified Cloudflare bots. Monitor for false positives.)<\/li>\n\n\n\n
    • Generic “spider” in User-Agent<\/strong> (Blocks user agents containing “spider” except verified Cloudflare bots.)<\/li>\n<\/ul>\n\n\n\n

      Exploit Scanners<\/h2>\n\n\n\n

      This section blocks well-known vulnerability scanners and penetration testing tools commonly used by attackers.<\/p>\n\n\n\n

      \"exploit<\/figure>\n\n\n\n

      Available Rules<\/strong><\/p>\n\n\n\n

        \n
      • Nikto<\/strong> (Blocks the Nikto vulnerability scanner.)<\/li>\n\n\n\n
      • SQLMap<\/strong> (Blocks SQL injection testing tools.)<\/li>\n\n\n\n
      • Masscan<\/strong> (Blocks Masscan network scanning requests.)<\/li>\n\n\n\n
      • Nmap<\/strong> (Blocks requests related to the Nmap scanner.)<\/li>\n<\/ul>\n\n\n\n

        WordPress Path Protection<\/h1>\n\n\n\n

        This section protects sensitive WordPress files and endpoints that attackers frequently target.<\/p>\n\n\n\n

        \"wordpress<\/figure>\n\n\n\n

        Available Rules<\/strong><\/p>\n\n\n\n

          \n
        • Block XML-RPC<\/strong> (Blocks access to the XML-RPC endpoint. Helps prevent brute-force attacks and pingback abuse.)<\/li>\n\n\n\n
        • Block wp-config<\/strong> (Blocks attempts to access the wp-config.php<\/code> file.)<\/li>\n\n\n\n
        • Block WP-JSON (REST API)<\/strong> (Restricts access to the WordPress REST API endpoint.)<\/li>\n\n\n\n
        • Block install.php<\/strong> (Blocks install.php to reduce the risk of reinstallation exposure on production sites.)<\/li>\n\n\n\n
        • Block WLW Manifest<\/strong> (Blocks wlwmanifest.xml, which mostly exposes WordPress metadata. Safe to block.)<\/li>\n\n\n\n
        • Block readme.html<\/strong> (Blocks readme.html, which can reveal your WordPress version.)<\/li>\n\n\n\n
        • Block license.txt<\/strong> (Blocks license.txt to reduce WordPress fingerprinting and information disclosure.)<\/li>\n<\/ul>\n\n\n\n

          Attack Patterns<\/h2>\n\n\n\n

          This section blocks requests commonly associated with advanced attack techniques.<\/p>\n\n\n\n

          \"attack<\/figure>\n\n\n\n

          Available Rules<\/strong><\/p>\n\n\n\n

            \n
          • Time-delay \/ Blind SQLi Primitives<\/strong> (Blocks time-based blind SQL injection payloads like pg_sleep and waitfor delay.)<\/li>\n\n\n\n
          • Encoded Path Traversal \/ LFI<\/strong> (Blocks URL-encoded path traversal attempts like ..\/etc\/passwd and similar LFI probes.)<\/li>\n<\/ul>\n\n\n\n
            \n

            Enable both protections for improved WAF coverage.<\/em><\/p>\n<\/blockquote>\n\n\n\n

            Deploy to Cloudflare<\/h2>\n\n\n\n

            After configuring your bot whitelist, you must save and deploy<\/strong> to make it active on Cloudflare.<\/p>\n\n\n\n

            \"deploy<\/figure>\n\n\n\n
            <\/div>\n\n\n\n

            Deploy Rules<\/strong>: Pushes your saved settings to Cloudflare and activates them live
            Preview Rules<\/strong>: Shows you the exact rule expressions that will be generated. Review before deploying
            Remove Plugin Rules<\/strong>: Removes all WAF rules created by this plugin from Cloudflare
            Zone Selector:<\/strong> Choose which Cloudflare domain (zone) to deploy to.<\/p>\n\n\n\n

            How Deployment Works From the Plugin:<\/strong><\/p>\n\n\n\n

              \n
            1. Save your WAF settings first<\/strong> using the Save Changes<\/strong> button at the bottom of the page<\/li>\n\n\n\n
            2. Select the Cloudflare zone<\/strong> you want to protect<\/li>\n\n\n\n
            3. Preview Rules<\/strong> shows the current draft output, including source tags for each generated rule<\/li>\n\n\n\n
            4. Deploy Rules<\/strong> pushes only the saved plugin-managed rules and preserves unrelated Cloudflare rules<\/li>\n<\/ol>\n\n\n\n
              \n

              The plugin only manages its own rules. It won’t delete or overwrite any rules you created manually in Cloudflare.<\/em><\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"

              The Block Crawlers & WP Paths feature helps protect your WordPress site from aggressive bots, exploit scanners, suspicious crawlers, and […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"doc_category":[75],"doc_tag":[],"class_list":["post-951","docs","type-docs","status-publish","hentry","doc_category-setup-rules"],"year_month":"2026-06","word_count":596,"total_views":0,"reactions":{"happy":0,"normal":0,"sad":0},"author_info":{"name":"mishu","author_nicename":"mishu","author_url":"https:\/\/docs.wpultimatesecurity.com\/author\/mishu\/"},"doc_category_info":[{"term_name":"Setup Rules","term_url":"https:\/\/docs.wpultimatesecurity.com\/docs-category\/setup-rules\/"}],"doc_tag_info":[],"_links":{"self":[{"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/docs\/951","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/docs"}],"about":[{"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/types\/docs"}],"author":[{"embeddable":true,"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/comments?post=951"}],"version-history":[{"count":5,"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/docs\/951\/revisions"}],"predecessor-version":[{"id":1024,"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/docs\/951\/revisions\/1024"}],"wp:attachment":[{"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/media?parent=951"}],"wp:term":[{"taxonomy":"doc_category","embeddable":true,"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/doc_category?post=951"},{"taxonomy":"doc_tag","embeddable":true,"href":"https:\/\/docs.wpultimatesecurity.com\/wp-json\/wp\/v2\/doc_tag?post=951"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}