Scribd
Upload
34 views2 pages

Robots

The document outlines the robots.txt configuration for a website, specifying which directories and files are allowed or disallowed for crawling by web crawlers. It includes specific rules for the Google Image crawler, allowing access to product images while disallowing other paths. Additionally, it provides a link to the website's sitemap for better indexing.

Uploaded by

tREnS Pirar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
34 views2 pages

Robots

The document outlines the robots.txt configuration for a website, specifying which directories and files are allowed or disallowed for crawling by web crawlers. It includes specific rules for the Google Image crawler, allowing access to product images while disallowing other paths. Additionally, it provides a link to the website's sitemap for better indexing.

Uploaded by

tREnS Pirar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

# Google Image Crawler Setup

User-agent: *
#Crawl-delay: 10

# Allowable Index
Allow: /*?p=

Allow: /media/

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Allow: /media/catalog/product/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)


Disallow: /[Link]/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Files
Disallow: /[Link]
Disallow: /[Link]
Disallow: /error_log
Disallow: /[Link]
Disallow: /[Link]
Disallow: /[Link]
Disallow: /LICENSE_AFL.txt
Disallow: /[Link]

# Paths (no clean URLs)


#Disallow: /*.js$
#Disallow: /*.css$
Disallow: /*.php$
Allow: /*?SID=

User-agent: Googlebot-Image
Disallow: /
Allow: /media/catalog/product/

Sitemap: [Link]

You might also like