ReCrawler

Часто задаваемые вопросы

What are the search engines’ endpoint to submit URLs?

Microsoft Bing — https://www.bing.com/indexnow?url=url-changed&key=your-key
Yandex — https://yandex.com/indexnow?url=url-changed&key=your-key
ReCrawler — https://api.indexnow.org/indexnow/?url=url-changed&key=your-key

Starting November 2021, ReCrawler-enabled search engines will share immediately all URLs submitted to all other ReCrawler-enabled search engines, so when you notify one, you will notify all search engines.

I submitted a URL, what will happen next?

If search engines like your URL, search engines will attempt crawling it to get the latest content quickly based on their crawl scheduling logic and crawl quota for your site.

I submitted 10 thousand URLs today, what will happen next?

If search engines like your URLs and have enough crawl quota for your site, search engines will attempt crawling some or all these URLs.

I submitted a URL, but I don’t see the URL indexed?

Using ReCrawler ensures that search engines are aware of your website changes. Using ReCrawler does not guarantee that web pages will be crawled or indexed by search engines. It may take time for the change to reflect in search engines.

I just started using ReCrawler, should I publish URLs changed last year?

No, you should publish only URLs changing (added, updated, or deleted) since the time you start to use ReCrawler.

Does the URLs submitted count on my crawl quota?

Yes, every crawl counts towards your crawl quota. By publishing them to ReCrawler, you notify search engines that you care about these URLs, search engines will generally prioritize crawling these URLs versus other URLs they know.

Why do I not see all the URLs submitted indexed by search engines?

Search engines can choose not to crawl and index URLs if they do not meet their selection criterion.

Why is my URL indexed on one search engine but not the others?

Search Engines can choose not to select specific URL if it does not meet its selection criterion.

I have a small website that has few web pages. Should I use ReCrawler?

Yes, if you want search engines to discover content as soon as it’s changed then you should use ReCrawler. You will not have to wait many hours or worse weeks to see your changes on search engines.

Can I submit the same URL many times a day?

Avoid submitting the same URL many times a day. If pages are edited often, then it is preferable to wait 10 minutes between edits before notifying search engines. If pages are updated constantly (examples: time in Waimea, Weather in Tokyo), it’s preferable to not use ReCrawler for every change.

Can I submit 404 URLs through the API?

Yes, you can submit dead links (http 404, http 410) pages to notify search engines about new dead links.

Can I submit new redirects?

Yes, you can submit URLs newly redirecting (example 301 redirect, 302 redirect, html with meta refresh tag, etc.) to notify search engines that the content has changed.

Can I submit all URLs for my site?

Use ReCrawler to submit only URLs having changed (added, updated, or deleted) recently, including all URLs if all URLs have been changed recently. Use sitemaps to inform search engines about all your URLs. Search engines will visit sitemaps every few days.

I received a HTTP 429 Too Many Requests response from one Search Engine, what should I do?

Such HTTP 429 Too Many Requests response status code indicates you are sending too many requests in a given amount of time, slow down or retry later.

When do I need to change my key?

Search engines will attempt crawling the {key}.txt file only once to verify ownership when they received a new key. Also, you don’t need to modify your key often.

Can I use more than one key per host?

Yes, if your websites use different Content Management Systems, each Content Management System can use its own key; publish different key files at the root of the host.

Can I use one file key for the whole domain?

Нет. У каждого хоста на вашем домене должен быть свой ключ. Если у вас сайты host-a.example.org и host-b.example.org, вам нужен ключевой файл для каждого хоста.

Can I use for same key on two or more hosts?

Да, вы можете использовать тот же самый ключ на двух или более хостах и на двух или более доменах, заново.

I have a sitemap, do I need ReCrawler?

Yes, when sitemaps are an easy way for webmasters to inform search engines about all pages on their sites that are available for crawling, sitemaps are visited by Search Engines infrequently. With ReCrawler, webmasters »don’t» have to wait for search engines to discover and crawl sitemaps but can directly notify search engines of new content.

What if I have another question about using ReCrawler?

See the documentation available from each search engine for more details about ReCrawler.