indexaccion google

afabiani

Nuevo usuario
Buenos días a todos, les escribo porque estuve varios dias investigando y probando cosas pero no logro dar con el resultado. El tema es asi, google esta indexando los filtros de magento /catalogsearch/result/. Lo que hice fue modificar el archivo robots.txt pero sigue indexando mal. Alguien le ha pasado lo mismo?

Mi robots.txt es este:
# Crawlers Setup
User-agent: *
Crawl-delay: 1
# Google Image Crawler Setup
User-agent: Googlebot-Image
Disallow:
# BING
User-agent: msnbot
Crawl-delay: 10

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/
Disallow: /captcha/refresh/
Disallow: /catalogsearch/result/
Disallow: /catalogsearch/*
Disallow: /onestepcheckout/
Disallow: /customer/account/
Disallow: /customer/account/login/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?dir=
Disallow: /*?q=
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=*
Disallow: /*mode=*
Disallow: /*cat=*
Disallow: /*manufacturer=*
Disallow: /*futura_size=*
Disallow: /*?SID=Sitemap:http://magento.instantsearchplus.com/ext_sitemap?u=2c4c36e1-0910-49e1-b889-4b4f504b75e8
 

Eds Daniels

Nuevo usuario
Veo que solo estas bloqueando la indexacion al crawler de imagenes de google pero no el de contenido o el de Ads. Te recomiendo poner una configuración por cada crawler que quieras que no indexe.
Te comparto una programación de ejemplo:


User-agent:Adsbot-Google
Allow: /js/
Allow: /lib/
Allow: /media/

DisAllow: /404/
DisAllow: /app/
DisAllow: /cgi-bin/
DisAllow: /downloader/
DisAllow: /errors/
DisAllow: /includes/
DisAllow: /pkginfo/
DisAllow: /shell/
DisAllow: /skin/
DisAllow: /var/
DisAllow: /lib/
DisAllow: /magento/
DisAllow: /media/captcha/

# Paths (clean URLs)

DisAllow: /index.php/
DisAllow: /catalogsearch/result/
DisAllow: /catalog/product/gallery/
DisAllow: /catalog/product_compare/
DisAllow: /catalog/category/view/
DisAllow: /catalog/product/view/
DisAllow: /catalogsearch/
DisAllow: /checkout/
DisAllow: /control/
DisAllow: /contacts/
DisAllow: /customer/
DisAllow: /customize/
DisAllow: /newsletter/
DisAllow: /poll/
DisAllow: /review/
DisAllow: /review/product/
DisAllow: /review/product/list/
DisAllow: /sendfriend/
DisAllow: /tag/
DisAllow: /wishlist/
DisAllow: /index.php/catalogsearch/result/
DisAllow: /index.php/catalog/product_compare/
DisAllow: /index.php/catalog/category/view/
DisAllow: /index.php/catalog/product/view/
DisAllow: /index.php/catalogsearch/
DisAllow: /index.php/checkout/
DisAllow: /index.php/control/
DisAllow: /index.php/contacts/
DisAllow: /index.php/customer/
DisAllow: /index.php/customize/
DisAllow: /index.php/newsletter/
DisAllow: /index.php/poll/
DisAllow: /index.php/review/
DisAllow: /index.php/sendfriend/
DisAllow: /index.php/tag/
DisAllow: /index.php/wishlist/
DisAllow: /sendfriend/
DisAllow: /ajaxcart/
DisAllow: /ajax/
DisAllow: /quickview/
DisAllow: /productalert/
DisAllow: /sales/guest/form/

# Files

DisAllow: /api.php
DisAllow: /get.php
DisAllow: /cron.php
DisAllow: /cron.sh
DisAllow: /error_log
DisAllow: /install.php
DisAllow: /LICENSE.html
DisAllow: /LICENSE.txt
DisAllow: /LICENSE_AFL.txt
DisAllow: /STATUS.txt
DisAllow: /README.txt
DisAllow: /RELEASE_NOTES.txt

# Paths (no clean URLs)

#DisAllow: /*.js$
#DisAllow: /*.css$

DisAllow: /*.php$
DisAllow: /*?SID=
DisAllow: /*PHPSESSID

# Do not index the page subcategories that are sorted or filtered.

DisAllow: /*?
DisAllow: /*?q
DisAllow: /*?price
DisAllow: /*?tipo_producto
DisAllow: /*?shopdesign_brand
DisAllow: /*?super_color
DisAllow: /*?super_material
DisAllow: /*?estilo
DisAllow: /*?shopdesign_designer
DisAllow: /*?cat
DisAllow: /*&
DisAllow: /*&q
DisAllow: /*&price
DisAllow: /*&tipo_producto
DisAllow: /*&shopdesign_brand
DisAllow: /*&super_color
DisAllow: /*&super_material
DisAllow: /*&estilo
DisAllow: /*&shopdesign_designer
DisAllow: /*?dir
DisAllow: /*&dir
DisAllow: /*?mode
DisAllow: /*&mode
DisAllow: /*?order
DisAllow: /*&order
DisAllow: /*?limit
DisAllow: /*&limit
DisAllow: /*%C3%A
DisAllow: /*%2F
DisAllow: /*+
DisAllow: /*=

User-agent: Googlebot
Allow: /js/
Allow: /lib/
Allow: /media/

DisAllow: /404/
DisAllow: /app/
DisAllow: /cgi-bin/
DisAllow: /downloader/
DisAllow: /errors/
DisAllow: /includes/
DisAllow: /pkginfo/
DisAllow: /shell/
DisAllow: /skin/
DisAllow: /var/
DisAllow: /lib/
DisAllow: /magento/
DisAllow: /media/captcha/

# Paths (clean URLs)

DisAllow: /index.php/
DisAllow: /catalogsearch/result/
DisAllow: /catalog/product/gallery/
DisAllow: /catalog/product_compare/
DisAllow: /catalog/category/view/
DisAllow: /catalog/product/view/
DisAllow: /catalogsearch/
DisAllow: /checkout/
DisAllow: /control/
DisAllow: /contacts/
DisAllow: /customer/
DisAllow: /customize/
DisAllow: /newsletter/
DisAllow: /poll/
DisAllow: /review/
DisAllow: /review/product/
DisAllow: /review/product/list/
DisAllow: /sendfriend/
DisAllow: /tag/
DisAllow: /wishlist/
DisAllow: /index.php/catalogsearch/result/
DisAllow: /index.php/catalog/product_compare/
DisAllow: /index.php/catalog/category/view/
DisAllow: /index.php/catalog/product/view/
DisAllow: /index.php/catalogsearch/
DisAllow: /index.php/checkout/
DisAllow: /index.php/control/
DisAllow: /index.php/contacts/
DisAllow: /index.php/customer/
DisAllow: /index.php/customize/
DisAllow: /index.php/newsletter/
DisAllow: /index.php/poll/
DisAllow: /index.php/review/
DisAllow: /index.php/sendfriend/
DisAllow: /index.php/tag/
DisAllow: /index.php/wishlist/
DisAllow: /sendfriend/
DisAllow: /ajaxcart/
DisAllow: /ajax/
DisAllow: /quickview/
DisAllow: /productalert/
DisAllow: /sales/guest/form/

# Files

DisAllow: /api.php
DisAllow: /get.php
DisAllow: /cron.php
DisAllow: /cron.sh
DisAllow: /error_log
DisAllow: /install.php
DisAllow: /LICENSE.html
DisAllow: /LICENSE.txt
DisAllow: /LICENSE_AFL.txt
DisAllow: /STATUS.txt
DisAllow: /README.txt
DisAllow: /RELEASE_NOTES.txt

# Paths (no clean URLs)

#DisAllow: /*.js$
#DisAllow: /*.css$

DisAllow: /*.php$
DisAllow: /*?SID=
DisAllow: /*PHPSESSID

# Do not index the page subcategories that are sorted or filtered.

DisAllow: /*?
DisAllow: /*?q
DisAllow: /*?price
DisAllow: /*?tipo_producto
DisAllow: /*?shopdesign_brand
DisAllow: /*?super_color
DisAllow: /*?super_material
DisAllow: /*?estilo
DisAllow: /*?shopdesign_designer
DisAllow: /*?cat
DisAllow: /*&
DisAllow: /*&q
DisAllow: /*&price
DisAllow: /*&tipo_producto
DisAllow: /*&shopdesign_brand
DisAllow: /*&super_color
DisAllow: /*&super_material
DisAllow: /*&estilo
DisAllow: /*&shopdesign_designer
DisAllow: /*?dir
DisAllow: /*&dir
DisAllow: /*?mode
DisAllow: /*&mode
DisAllow: /*?order
DisAllow: /*&order
DisAllow: /*?limit
DisAllow: /*&limit
DisAllow: /*%C3%A
DisAllow: /*%2F
DisAllow: /*+
DisAllow: /*=

User-agent: Googlebot-Image
Allow: /js/
Allow: /lib/
Allow: /media/

DisAllow: /404/
DisAllow: /app/
DisAllow: /cgi-bin/
DisAllow: /downloader/
DisAllow: /errors/
DisAllow: /includes/
DisAllow: /pkginfo/
DisAllow: /shell/
DisAllow: /skin/
DisAllow: /var/
DisAllow: /lib/
DisAllow: /magento/
DisAllow: /media/captcha/

# Paths (clean URLs)

DisAllow: /index.php/
DisAllow: /catalogsearch/result/
DisAllow: /catalog/product/gallery/
DisAllow: /catalog/product_compare/
DisAllow: /catalog/category/view/
DisAllow: /catalog/product/view/
DisAllow: /catalogsearch/
DisAllow: /checkout/
DisAllow: /control/
DisAllow: /contacts/
DisAllow: /customer/
DisAllow: /customize/
DisAllow: /newsletter/
DisAllow: /poll/
DisAllow: /review/
DisAllow: /review/product/
DisAllow: /review/product/list/
DisAllow: /sendfriend/
DisAllow: /tag/
DisAllow: /wishlist/
DisAllow: /index.php/catalogsearch/result/
DisAllow: /index.php/catalog/product_compare/
DisAllow: /index.php/catalog/category/view/
DisAllow: /index.php/catalog/product/view/
DisAllow: /index.php/catalogsearch/
DisAllow: /index.php/checkout/
DisAllow: /index.php/control/
DisAllow: /index.php/contacts/
DisAllow: /index.php/customer/
DisAllow: /index.php/customize/
DisAllow: /index.php/newsletter/
DisAllow: /index.php/poll/
DisAllow: /index.php/review/
DisAllow: /index.php/sendfriend/
DisAllow: /index.php/tag/
DisAllow: /index.php/wishlist/
DisAllow: /sendfriend/
DisAllow: /ajaxcart/
DisAllow: /ajax/
DisAllow: /quickview/
DisAllow: /productalert/
DisAllow: /sales/guest/form/

# Files

DisAllow: /api.php
DisAllow: /get.php
DisAllow: /cron.php
DisAllow: /cron.sh
DisAllow: /error_log
DisAllow: /install.php
DisAllow: /LICENSE.html
DisAllow: /LICENSE.txt
DisAllow: /LICENSE_AFL.txt
DisAllow: /STATUS.txt
DisAllow: /README.txt
DisAllow: /RELEASE_NOTES.txt

# Paths (no clean URLs)

#DisAllow: /*.js$
#DisAllow: /*.css$

DisAllow: /*.php$
DisAllow: /*?SID=
DisAllow: /*PHPSESSID

# Do not index the page subcategories that are sorted or filtered.

DisAllow: /*?
DisAllow: /*?q
DisAllow: /*?price
DisAllow: /*?tipo_producto
DisAllow: /*?shopdesign_brand
DisAllow: /*?super_color
DisAllow: /*?super_material
DisAllow: /*?estilo
DisAllow: /*?shopdesign_designer
DisAllow: /*?cat
DisAllow: /*&
DisAllow: /*&q
DisAllow: /*&price
DisAllow: /*&tipo_producto
DisAllow: /*&shopdesign_brand
DisAllow: /*&super_color
DisAllow: /*&super_material
DisAllow: /*&estilo
DisAllow: /*&shopdesign_designer
DisAllow: /*?dir
DisAllow: /*&dir
DisAllow: /*?mode
DisAllow: /*&mode
DisAllow: /*?order
DisAllow: /*&order
DisAllow: /*?limit
DisAllow: /*&limit
DisAllow: /*%C3%A
DisAllow: /*%2F
DisAllow: /*+
DisAllow: /*=
 
Arriba