Hello!
Could you tell me why you’d wish to edit robots.txt? It often isn’t the most effective tool for the job. See the warning on this page to learn why: https://developers.google.com/search/docs/crawling-indexing/robots/intro.
Now, TSF doesn’t provide an option to edit robots.txt (aside from adding a sitemap link) — it is planned, however, since legitimate crawlers like GPTBot are now stealing content without permission.
In the meantime, you can upload a robots.txt file to the root folder of your website to set custom robots rules. However, this will block WordPress from virtually outputting it.
Alternatively, you can use this plugin: https://wordpress.org/plugins/robots-txt-quick-editor/. I just tested it, and it works as intended: either append or override; both features work.
Thread Starter
Justin
(@static1635)
Hi, thanks for the info.
My stats plugin “WP Statistics” asked me to add “Disallow: /wp-json” to my robots.txt
Hello Justin Tuijl! Adding the “Disallow: /wp-json” directive does not make sense, since the /wp-json page contains the HTTP header X-Robots-Tag: noindex, which prevents indexing by search engines.
I also recommend installing the Disable WP REST API plugin to disable REST/JSON for non-logged in visitors.
Here are some good reasons to disable the REST API for unregistered users:
– The REST API may not be needed for non-logged users,
– Disabling the REST API conserves server resources,
– Disabling the REST API minimizes potential attack vectors,
– Disabling the REST API prevents content scraping and plagiarism.
Thread Starter
Justin
(@static1635)
Ah, thank you for this info. Very useful. The plugin disabling Rest Api for non-logged in is great.
-
This reply was modified 1 year, 9 months ago by
Justin.
