Se você tem a oportunidade de editar o arquivo robots.txt tenha muito cuidado.

Não é algo que você deve fazer com frequência, mas vale a pena utilizar as configurações corretas a seu favor.

Primeiros passos com arquivos robots? Então utilize esse exemplo

User-agent: *
Disallow:

Aqui estamos permitindo tudo de seu site

outro bom primeiro exemplo é:

User-agent: *Disallow:Sitemap: https://www.meusite.com.br/sitemaps/sitemap.xml.gz

Abaixo vou dar 3 exemplos de sitemaps de sites populares no Google e de sites bem indexados nos buscadores.

User-agent: *Disallow: */busca*{"id":*{"id":*Disallow: /avaliacao
Disallow: /parceiros
Disallow: /product-description
Disallow: /portal
Disallow: /garantia
Disallow: /estaticapopDisallow: *{"id":*{"id":*{"id":*{"id":*
Disallow: */f/*/*/Disallow: /*cage.htmlSitemap: https://www.americanas.com.br/sitemaps/americanas.xml.gz

Um exemplo do site da americanas https://www.americanas.com.br/robots.txt

#Robots despegar default
User-agent: *
Disallow:   /2001/
Disallow:   /2002/
Disallow:   /2003/
Disallow:   /cgi-bin/
Disallow:   /cobranded/
Disallow:   /desp/
Disallow:   /despegarCorp/
Disallow:   /entelpcs/
Disallow:   /site/
Disallow:   /span/
Disallow:   /STS/
Disallow:   /temp
Disallow:   /wap
Disallow:   /wapold
Disallow:   /Yahoo/
Disallow:   /yahoobr/
Disallow:   /ASPxGrid/
Disallow:   /backnet/
Disallow:   /Cancun2005/
Disallow:   /Checkout
Disallow:   /citibank/
Disallow:   /entelpcs/
Disallow:   /Error
Disallow:   /estatico/
Disallow:   /estilos/
Disallow:   /galeria/
Disallow:   /general/
Disallow:   /grhomes/
Disallow:   /grtemplates/
Disallow:   /hl
Disallow:   /ict
Disallow:   /imagen_t1msn/
Disallow:   /ImagesMapQuest/
Disallow:   /imagesNN/
Disallow:   /img/
Disallow:   /PingSabre/
Disallow:   /recomendacionpt_files/
Disallow:   /Agendaes_files/
Disallow:   /desp/
Disallow:   /despegarCorp/
Disallow:   /infoHotelpt_files/
Disallow:   /infoPuntaCana_files
Disallow:   /infoRDominicanapt_files
Disallow:   /infoVisaspt_files
Disallow:   /isapi
Allow:      /vuelos-baratos
Disallow:   /paginas
Disallow:   /Despegar.Business.Services/
Disallow:   /ezpublish/
Disallow:   /georigen/
Disallow:   /savecookie.asp
Disallow:   /search/
Disallow:   /Search/
Disallow:   /SEARCH/
Disallow:   /shop/
Disallow:   /Shop/
Disallow:   /SHOP/
Disallow:   /shopping/
Disallow:   /Shopping/
Disallow:   /SHOPPING/
Disallow:   /booking/
Disallow:   /Booking/
Disallow:   /BOOKING/
Disallow:   /beneficios/
Disallow:   /Beneficios/
Disallow:   /BENEFICIOS/
Disallow:   /contents/
Disallow:   /Contents/
Disallow:   /CONTENTS/
Disallow:   /credito
Disallow:   /Credito
Disallow:   /CREDITO
Allow:      /Search/Flights/AdvancedSearch
Disallow:   /gds-services-web/
Disallow:   /homes/
Disallow:   /Homes/
Disallow:   /HOMES/
Disallow:   /benefit-service/
Disallow:   /it/
Disallow:   /voos.html
Disallow:   /micrositios/
Disallow:   /Micrositios/
Disallow:   /MICROSITIOS/
Disallow:   /cruises/
Disallow:   /Cruises/
Disallow:   /CRUISES/
Disallow:   /advertising/
Disallow:   /Advertising/
Disallow:   /ADVERTISING/
Disallow:   /lpfrontend/
Disallow:   /Lpfrontend/
Disallow:   /LPFRONTEND/
Disallow:   /travelnetwork_terminos.asp
Disallow:   /travelnetwork_terminos_en.asp
Disallow:   /travelnetwork_terminos_pt.asp
Disallow:   /yahoo/
Disallow:   /Yahoo/
Disallow:   /YAHOO/
Disallow:   /yahoobr/
Disallow:   /Yahoobr/
Disallow:   /YAHOOBR/
Disallow:   /IpService/
Disallow:   /ipservice/
Disallow:   /IPSERVICE/
Disallow:   /IT/
Disallow:   /It/
Disallow:   /turismo/SiteServices/
Disallow:   /search/hotels/
Disallow:   /Hotels.Services/
Disallow:   /loggerjs/
Disallow:   /Loggerjs/
Disallow:   /hoteles/landinghotels/
Disallow:   /Hoteles/landinghotels/
Disallow:   /landing-hotels-web/
Allow:/carros/
Disallow:   /autos/
Disallow:   /cars/shop/
Disallow:   /Cars/shop/
Disallow:   /que-hacer-en-orlando/
#Habilita el flow de Cars
Allow: /cars/shop/city/
Allow: /cars/shop/airport/
Disallow:   /book/
Disallow:   /Book/
Disallow:   /BOOK/
Disallow:   /commercial-web/
Disallow:   /packages/
Disallow:   /PACKAGES/
Disallow:   /Packages/
Disallow:   /vuelos/handler/
Disallow:   /vuelos/Handler/
Disallow:   /vuelos/HANDLER/
Disallow:   /vuelos/sem/
Disallow:   /vuelos/Sem/
Disallow:   /vuelos/SEM/
Disallow:   /viajes_en_oferta/
Disallow:   /Taca/
Disallow:   /contactenos/
Disallow:   /homesv2/
Disallow:   /content/
Disallow:   /content/
Disallow:   /Lineas%20Aereas/
Disallow:   /amadeuscars/
Disallow:   /hoteles/search/
Disallow:   /hoteles/detail/
Disallow:   /hoteles/map/
#Los siguientes 2 son de las nuevas landings de hoteles en destinos
#Disallow:   /hoteles/ciudad/
#Disallow:   /hoteis/cidade/
Disallow:   /hoteis/search/
Disallow:   /hoteis/detail/
Disallow:   /hoteis/map/
Disallow:   /emt-service/
Disallow:   /packages-static-content/
#Bloquea las landings de hoteles en pais
Disallow:   /hoteles/pais/
#Disallow:   /hoteis/pais/
#Los siguientes 3 son por la estructura de urls de mobile
Disallow:   /vuelos/detail/
Disallow:   /passagens-aereas/detail/
Disallow:   /vuelos/oneway/
Disallow:   /passagens-aereas/oneway/
Disallow:   /vuelos/roundtrip/
Disallow:   /passagens-aereas/roundtrip/
#Allow para hoteles search
Allow:   /search/hotel/
Disallow:   /search/hotel/fullSearch/
Disallow:   /search/hotel/details/lastViewedHotels
Disallow:   /search/hotel/quickSearch
Disallow:   /search/hotel/filteredSearch
Disallow:   /search/hotel/details/availability/
Disallow:   /search/hotel/details/closestHotels/
Disallow:   /search/hotel/details/suggestedHotels
Disallow:   /search/hotel/quickSearch/
Disallow:   /search/hotel/summariesAndReviews/
Disallow:   /search/hotel/details/streetView/
Disallow:   /search/hotel/details/summary/
Disallow:   /search/hotel/details/reviews/
Allow:   /Search/hotel/
Disallow:   /Search/hotel/fullSearch/
Disallow:   /Search/hotel/details/lastViewedHotels
Disallow:   /Search/hotel/quickSearch
Disallow:   /Search/hotel/filteredSearch
Disallow:   /Search/hotel/details/availability/
Disallow:   /Search/hotel/details/closestHotels/
Disallow:   /Search/hotel/details/suggestedHotels
Disallow:   /Search/hotel/quickSearch/
Disallow:   /Search/hotel/summariesAndReviews/
Disallow:   /Search/hotel/details/streetView/
Disallow:   /Search/hotel/details/summary/
Disallow:   /Search/hotel/details/reviews/
Allow:   /search/Hotel/
Disallow:   /search/Hotel/fullSearch/
Disallow:   /search/Hotel/details/lastViewedHotels
Disallow:   /search/Hotel/quickSearch
Disallow:   /search/Hotel/filteredSearch
Disallow:   /search/Hotel/details/availability/
Disallow:   /search/Hotel/details/closestHotels/
Disallow:   /search/Hotel/details/suggestedHotels
Disallow:   /search/Hotel/quickSearch/
Disallow:   /search/Hotel/summariesAndReviews/
Disallow:   /search/Hotel/details/streetView/
Disallow:   /search/Hotel/details/summary/
Disallow:   /search/Hotel/details/reviews/
Allow:   /Search/Hotel/
Disallow:   /Search/Hotel/fullSearch/
Disallow:   /Search/Hotel/details/lastViewedHotels
Disallow:   /Search/Hotel/quickSearch
Disallow:   /Search/Hotel/filteredSearch
Disallow:   /Search/Hotel/details/availability/
Disallow:   /Search/Hotel/details/closestHotels/
Disallow:   /Search/Hotel/details/suggestedHotels
Disallow:   /Search/Hotel/quickSearch/
Disallow:   /Search/Hotel/summariesAndReviews/
Disallow:   /Search/Hotel/details/streetView/
Disallow:   /Search/Hotel/details/summary/
Disallow:   /Search/Hotel/details/reviews/
Disallow:  /comunidadafiliados/resources/
Allow:  /shop/flights/
Disallow: /shop/flights/print/
Disallow: /shop/flights/wishlist/
Disallow: /shop/flights/data/
Disallow: /shop/flights/results/
Allow: /cruises/shop/search/
#Bloquea paginas de hoteles version para imprimir
Disallow: /hoteles/h/print/
Disallow: /hotels/h/print/
Disallow: /hoteis/h/print/#Paquetes
Allow: /pacotes/
Disallow: /paquetes/sem/
Disallow: /pacotes/sem/
Disallow: /packages/
Disallow: /paquetes/#Actividades
Allow: /atracoes-turisticas/
Disallow: /actividades/Disallow: /pasajes-en-micro/shop
Disallow: /passagens-de-onibus/shop#Special clients
Disallow: /AE_beneficios/
Disallow: /beneficio_empleados/
Disallow: /beneficio_topclub/
Disallow: /beneficio-empleados/
Disallow: /cajalosandes_beneficios/
Disallow: /clubadelanteviajes/
Disallow: /icbc-viajes/
Disallow: /jumbomas/
Disallow: /macro_premia/
Disallow: /puntosbancomer/
Disallow: /puntoscencosud/
Disallow: /sams_beneficios/
Disallow: /santanderplus/
Disallow: /scotiabank_beneficios/
Disallow: /superclub_canje/#Bloqueos especificos para el bot de QualityScore de SEM
User-agent: AdsBot-Google
Disallow:   /book/
#Bloqueo Baidu:
User-agent: Baiduspider
Disallow:   /Disallow: /turismo/Disallow: /detail/hotel/services/reviews/summary/User-agent: google-hoteladsverifier
Disallow:User-agent: *
Disallow: /cache/
Disallow: /configs/
Disallow: /templates/
Disallow: /util/
Sitemap: https://www.decolar.com/hf/sitemap/sitemap_index_BR.xml

O site decolar.com é um belo exemplo de paginas que devem ser ignoradas pelos buscadores

User-agent: *
Disallow: /conta/*
Disallow: /carrinho/index
Disallow: /carrinho/produto/*
Disallow: /buscar
Disallow: /documentacao
Disallow: /api/produto/calcular_frete
Disallow: /*fq=
Crawl-delay: 10User-agent: DirBuster-0.12
Disallow: /User-agent: Baiduspider
Disallow: /User-agent: Baiduspider-video
Disallow: /User-agent: Baiduspider-image
Disallow: /User-agent: Baiduspider+
Disallow: /User-agent: TwengaBot-Discover
Disallow: /User-agent: TwengaBot
Disallow: /User-agent: TwengaBot-2.0
Disallow: /User-agent: MJ12bot
Disallow: /User-agent: BDCbot
Disallow: /User-agent: spbot
Disallow: /User-agent: LinkpadBot
Disallow: /User-agent: WBSearchBot
Disallow: /User-agent: AddThis.com robot [email protected]
Disallow: /User-agent: AddThis.com
Disallow: /User-agent: Exabot
Disallow: /User-agent: YandexImages
Disallow: /User-agent: Yandex
Disallow: /User-agent: Slurp
Disallow: /User-agent: spbot
Disallow: /User-agent: AhrefsBot
Disallow: /User-agent: EveryoneSocialBot
Disallow: /Sitemap: https://www.lojaintegrada-loja.com.br/sitemap.xml

Algumas plataformas impedem você editar o arquivo robots.txt e tem uma configurações padrão para todos os sites, estes é um exemplo de qualquer loja virtual hospedada na Loja Integrada, que tem uma peculiar filtro de bots.

Encontrou a sua resposta?