Adding my own robots.txt

Resolved Gomle
(@gomle)

2 years, 7 months ago

I asked about robots.txt, and you answered.

I saw your answer only in my email, as my post and your answer were unfortunately both deleted as I hijacked the thread. I am sorry about that.

This is your answer to my question if I could use my own robots.txt file on the server root. (copy/pasted from my email)

Sybre Waaijer wrote:

Hello!

1. For most (read 99%) of WordPress websites, what you want to do isn’t an issue. However, WordPress Multisite configurations could generate a new robots.txt for each subsite; when you upload a file, that feature can no longer work. For example, these two outputs are different but are on the exact same WordPress website:
– https://premium.theseoframework.com/robots.txt
– https://theseoframework.com/robots.txt

2. Google ignores the delay directive in robots.txt — they figure this out automatically, but you can temporarily configure it via their website, see https://support.google.com/webmasters/answer/48620.
Bing does read the delay directive, but you can also configure Bing via their website.

If the 3000-4000 requests/day limits your server’s uptime/performance/bandwidth, you may want to speak to your hosting partner about that; a well-configured server should be able to serve about 2~3 requests every second per thread without issue (source).

Still, 4000 requests daily from Google and Bing are a lot, so I assume the website is large.

I notice you wrote (source) in regards to what the server should be able to handle. I am using Cloudways / Vultr, and I am not really impressed. I think my smaller Litespeed server did a better job before I moved to Cloudways.

And I wondered if you could please give me the source-link you were talking about once more, as this is something I’m interested in investigating.

Viewing 1 replies (of 1 total)

Plugin Author Sybre Waaijer
(@cybr)

2 years, 7 months ago

Hi again! Sorry for the delay.

The (source) link was this: https://kinsta.com/blog/php-benchmarks/#wordpress-59rc2 — I took the numbers where they have WooCommerce installed and divided that by 60. The derivation is that they have 60 CPU threads available on 30 cores, whereas most hosts skimp on users providing a measly single thread (“PHP Workers”) on a single (shared) CPU core.

Kinsta’s WordPress hosting provides 2 PHP workers for $35/month (so 6 hits per second) on C2 compute units.
To get that performance at Cloudways, you’d have to pay $84/month via Google Cloud. I had to guess this number because they used fallacies to prove they’re “better” on the single data page they have.

With that, I’m not surprised if you got better performance out of your Litespeed server. For example, at a middle-road price of $48/month at Linode, you get 4 cores when deploying Litespeed. This should easily achieve double the performance of those I mentioned above — and considering it’s Litespeed, likely triple.

Of course, assuming you have knowledge of server administration, you could also consider TransIP (in Amsterdam and Delft, what I use), Azure (globally, what I also use…), and many more.

Viewing 1 replies (of 1 total)

The topic ‘Adding my own robots.txt’ is closed to new replies.