Forum Replies Created

Viewing 3 replies - 1 through 3 (of 3 total)
  • I’ll leave my previous response, only to suggest that the following support link was immediately successful in solving the problem.

    Limiting excessive Google Crawls

    The recommendation of adding the following code stopped the Google Bot within 10 seconds of uploading the updated robots.txt file.

    Disallow: /calendar/action:posterboard/
    Disallow: /calendar/action:agenda/
    Disallow: /calendar/action:oneday/
    Disallow: /calendar/action:month/
    Disallow: /calendar/action:week/
    Disallow: /calendar/action:map/

    I would really suggest that anyone experiencing high CPU usage try this solution first. It might even be worth mentioning this in an overlay during install and after updates. It really is likely the single largest factor in poor site performance due to heavy loads from unseen traffic.

    Agreed. It took months to track down the issue, but after looking at average CPU utilization we’ve identified the one website that was using AI1EC.

    It’s a great program, but blocking bots is not a solution. Using WordFence, we’ve seen the GoogleBot consistently crawling for updates.

    Hostname: crawl-66-249-74-130.googlebot.com 77 hits
    Hostname: crawl-66-249-74-137.googlebot.com 66 hits
    Hostname: crawl-66-249-74-128.googlebot.com 61 hits
    Hostname: crawl-66-249-74-139.googlebot.com 60 hits

    This was over a 5 minute period. I think we’ll have to find a different solution as we don’t have time to rewrite the program to reduce bot loads.

    Thread Starter thewebwing

    (@thewebwing)

    Sorry, the WP version is 3.5 as stated above, not 3.4.2 as in the sidebar. I saw that just as I was hitting submit.

Viewing 3 replies - 1 through 3 (of 3 total)