Como Criar um Robots.txt

Olá!

Se você tem a oportunidade de editar o arquivo robots.txt tenha muito cuidado.

Não é algo que você deve fazer com frequência, mas vale a pena utilizar as configurações corretas a seu favor.

Primeiros passos com arquivos robots? Então utilize esse exemplo:

User-agent: * Disallow:

Aqui estamos permitindo tudo de seu site

outro bom primeiro exemplo é:

User-agent: *Disallow:Sitemap: https://www.meusite.com.br/sitemaps/sitemap.xml.gz

Abaixo vou dar 3 exemplos de sitemaps de sites populares no Google e de sites bem indexados nos buscadores.

Um exemplo do site da americanas https://www.americanas.com.br/robots.txt

User-agent: *Disallow: */busca*{"id":*{"id":*Disallow: /avaliacao Disallow: /parceiros Disallow: /product-description Disallow: /portal Disallow: /garantia Disallow: /estaticapopDisallow: *{"id":*{"id":*{"id":*{"id":* Disallow: */f/*/*/Disallow: /*cage.html

Sitemap: https://www.americanas.com.br/sitemaps/americanas.xml.gz

Sitemap: https://www.decolar.com/hf/sitemap/sitemap_index_BR.xml

O site decolar.com é um belo exemplo de paginas que devem ser ignoradas pelos buscadores

#Robots despegar default User-agent: * Disallow:   /2001/ Disallow:   /2002/ Disallow:   /2003/ Disallow:   /cgi-bin/ Disallow:   /cobranded/ Disallow:   /desp/ Disallow:   /despegarCorp/ Disallow:   /entelpcs/ Disallow:   /site/ Disallow:   /span/ Disallow:   /STS/ Disallow:   /temp Disallow:   /wap Disallow:   /wapold Disallow:   /Yahoo/ Disallow:   /yahoobr/ Disallow:   /ASPxGrid/ Disallow:   /backnet/ Disallow:   /Cancun2005/ Disallow:   /Checkout Disallow:   /citibank/ Disallow:   /entelpcs/ Disallow:   /Error Disallow:   /estatico/ Disallow:   /estilos/ Disallow:   /galeria/ Disallow:   /general/ Disallow:   /grhomes/ Disallow:   /grtemplates/ Disallow:   /hl Disallow:   /ict Disallow:   /imagen_t1msn/ Disallow:   /ImagesMapQuest/ Disallow:   /imagesNN/ Disallow:   /img/ Disallow:   /PingSabre/ Disallow:   /recomendacionpt_files/ Disallow:   /Agendaes_files/ Disallow:   /desp/ Disallow:   /despegarCorp/ Disallow:   /infoHotelpt_files/ Disallow:   /infoPuntaCana_files Disallow:   /infoRDominicanapt_files Disallow:   /infoVisaspt_files Disallow:   /isapi Allow:      /vuelos-baratos Disallow:   /paginas Disallow:   /Despegar.Business.Services/ Disallow:   /ezpublish/ Disallow:   /georigen/ Disallow:   /savecookie.asp Disallow:   /search/ Disallow:   /Search/ Disallow:   /SEARCH/ Disallow:   /shop/ Disallow:   /Shop/ Disallow:   /SHOP/ Disallow:   /shopping/ Disallow:   /Shopping/ Disallow:   /SHOPPING/ Disallow:   /booking/ Disallow:   /Booking/ Disallow:   /BOOKING/ Disallow:   /beneficios/ Disallow:   /Beneficios/ Disallow:   /BENEFICIOS/ Disallow:   /contents/ Disallow:   /Contents/ Disallow:   /CONTENTS/ Disallow:   /credito Disallow:   /Credito Disallow:   /CREDITO Allow:      /Search/Flights/AdvancedSearch Disallow:   /gds-services-web/ Disallow:   /homes/ Disallow:   /Homes/ Disallow:   /HOMES/ Disallow:   /benefit-service/ Disallow:   /it/ Disallow:   /voos.html Disallow:   /micrositios/ Disallow:   /Micrositios/ Disallow:   /MICROSITIOS/ Disallow:   /cruises/ Disallow:   /Cruises/ Disallow:   /CRUISES/ Disallow:   /advertising/ Disallow:   /Advertising/ Disallow:   /ADVERTISING/ Disallow:   /lpfrontend/ Disallow:   /Lpfrontend/ Disallow:   /LPFRONTEND/ Disallow:   /travelnetwork_terminos.asp Disallow:   /travelnetwork_terminos_en.asp Disallow:   /travelnetwork_terminos_pt.asp Disallow:   /yahoo/ Disallow:   /Yahoo/ Disallow:   /YAHOO/ Disallow:   /yahoobr/ Disallow:   /Yahoobr/ Disallow:   /YAHOOBR/ Disallow:   /IpService/ Disallow:   /ipservice/ Disallow:   /IPSERVICE/ Disallow:   /IT/ Disallow:   /It/ Disallow:   /turismo/SiteServices/ Disallow:   /search/hotels/ Disallow:   /Hotels.Services/ Disallow:   /loggerjs/ Disallow:   /Loggerjs/ Disallow:   /hoteles/landinghotels/ Disallow:   /Hoteles/landinghotels/ Disallow:   /landing-hotels-web/ Allow:/carros/ Disallow:   /autos/ Disallow:   /cars/shop/ Disallow:   /Cars/shop/ Disallow:   /que-hacer-en-orlando/ #Habilita el flow de Cars Allow: /cars/shop/city/ Allow: /cars/shop/airport/ Disallow:   /book/ Disallow:   /Book/ Disallow:   /BOOK/ Disallow:   /commercial-web/ Disallow:   /packages/ Disallow:   /PACKAGES/ Disallow:   /Packages/ Disallow:   /vuelos/handler/ Disallow:   /vuelos/Handler/ Disallow:   /vuelos/HANDLER/ Disallow:   /vuelos/sem/ Disallow:   /vuelos/Sem/ Disallow:   /vuelos/SEM/ Disallow:   /viajes_en_oferta/ Disallow:   /Taca/ Disallow:   /contactenos/ Disallow:   /homesv2/ Disallow:   /content/ Disallow:   /content/ Disallow:   /Lineas%20Aereas/ Disallow:   /amadeuscars/ Disallow:   /hoteles/search/ Disallow:   /hoteles/detail/ Disallow:   /hoteles/map/ #Los siguientes 2 son de las nuevas landings de hoteles en destinos #Disallow:   /hoteles/ciudad/ #Disallow:   /hoteis/cidade/ Disallow:   /hoteis/search/ Disallow:   /hoteis/detail/ Disallow:   /hoteis/map/ Disallow:   /emt-service/ Disallow:   /packages-static-content/ #Bloquea las landings de hoteles en pais Disallow:   /hoteles/pais/ #Disallow:   /hoteis/pais/ #Los siguientes 3 son por la estructura de urls de mobile Disallow:   /vuelos/detail/ Disallow:   /passagens-aereas/detail/ Disallow:   /vuelos/oneway/ Disallow:   /passagens-aereas/oneway/ Disallow:   /vuelos/roundtrip/ Disallow:   /passagens-aereas/roundtrip/ #Allow para hoteles search Allow:   /search/hotel/ Disallow:   /search/hotel/fullSearch/ Disallow:   /search/hotel/details/lastViewedHotels Disallow:   /search/hotel/quickSearch Disallow:   /search/hotel/filteredSearch Disallow:   /search/hotel/details/availability/ Disallow:   /search/hotel/details/closestHotels/ Disallow:   /search/hotel/details/suggestedHotels Disallow:   /search/hotel/quickSearch/ Disallow:   /search/hotel/summariesAndReviews/ Disallow:   /search/hotel/details/streetView/ Disallow:   /search/hotel/details/summary/ Disallow:   /search/hotel/details/reviews/ Allow:   /Search/hotel/ Disallow:   /Search/hotel/fullSearch/ Disallow:   /Search/hotel/details/lastViewedHotels Disallow:   /Search/hotel/quickSearch Disallow:   /Search/hotel/filteredSearch Disallow:   /Search/hotel/details/availability/ Disallow:   /Search/hotel/details/closestHotels/ Disallow:   /Search/hotel/details/suggestedHotels Disallow:   /Search/hotel/quickSearch/ Disallow:   /Search/hotel/summariesAndReviews/ Disallow:   /Search/hotel/details/streetView/ Disallow:   /Search/hotel/details/summary/ Disallow:   /Search/hotel/details/reviews/ Allow:   /search/Hotel/ Disallow:   /search/Hotel/fullSearch/ Disallow:   /search/Hotel/details/lastViewedHotels Disallow:   /search/Hotel/quickSearch Disallow:   /search/Hotel/filteredSearch Disallow:   /search/Hotel/details/availability/ Disallow:   /search/Hotel/details/closestHotels/ Disallow:   /search/Hotel/details/suggestedHotels Disallow:   /search/Hotel/quickSearch/ Disallow:   /search/Hotel/summariesAndReviews/ Disallow:   /search/Hotel/details/streetView/ Disallow:   /search/Hotel/details/summary/ Disallow:   /search/Hotel/details/reviews/ Allow:   /Search/Hotel/ Disallow:   /Search/Hotel/fullSearch/ Disallow:   /Search/Hotel/details/lastViewedHotels Disallow:   /Search/Hotel/quickSearch Disallow:   /Search/Hotel/filteredSearch Disallow:   /Search/Hotel/details/availability/ Disallow:   /Search/Hotel/details/closestHotels/ Disallow:   /Search/Hotel/details/suggestedHotels Disallow:   /Search/Hotel/quickSearch/ Disallow:   /Search/Hotel/summariesAndReviews/ Disallow:   /Search/Hotel/details/streetView/ Disallow:   /Search/Hotel/details/summary/ Disallow:   /Search/Hotel/details/reviews/ Disallow:  /comunidadafiliados/resources/ Allow:  /shop/flights/ Disallow: /shop/flights/print/ Disallow: /shop/flights/wishlist/ Disallow: /shop/flights/data/ Disallow: /shop/flights/results/ Allow: /cruises/shop/search/ #Bloquea paginas de hoteles version para imprimir Disallow: /hoteles/h/print/ Disallow: /hotels/h/print/ Disallow: /hoteis/h/print/#Paquetes Allow: /pacotes/ Disallow: /paquetes/sem/ Disallow: /pacotes/sem/ Disallow: /packages/ Disallow: /paquetes/#Actividades Allow: /atracoes-turisticas/ Disallow: /actividades/Disallow: /pasajes-en-micro/shop Disallow: /passagens-de-onibus/shop#Special clients Disallow: /AE_beneficios/ Disallow: /beneficio_empleados/ Disallow: /beneficio_topclub/ Disallow: /beneficio-empleados/ Disallow: /cajalosandes_beneficios/ Disallow: /clubadelanteviajes/ Disallow: /icbc-viajes/ Disallow: /jumbomas/ Disallow: /macro_premia/ Disallow: /puntosbancomer/ Disallow: /puntoscencosud/ Disallow: /sams_beneficios/ Disallow: /santanderplus/ Disallow: /scotiabank_beneficios/ Disallow: /superclub_canje/#Bloqueos especificos para el bot de QualityScore de SEM User-agent: AdsBot-Google Disallow:   /book/ #Bloqueo Baidu: User-agent: Baiduspider Disallow:   /Disallow: /turismo/Disallow: /detail/hotel/services/reviews/summary/User-agent: google-hoteladsverifier Disallow:User-agent: * Disallow: /cache/ Disallow: /configs/ Disallow: /templates/ 

Exemplo Sitemap: https://www.lojaintegrada-loja.com.br/sitemap.xml
User-agent: * Disallow: /conta/* Disallow: /carrinho/index Disallow: /carrinho/produto/* Disallow: /buscar Disallow: /documentacao Disallow: /api/produto/calcular_frete Disallow: /*fq= Crawl-delay: 10User-agent: DirBuster-0.12 Disallow: /User-agent: Baiduspider Disallow: /User-agent: Baiduspider-video Disallow: /User-agent: Baiduspider-image Disallow: /User-agent: Baiduspider+ Disallow: /User-agent: TwengaBot-Discover Disallow: /User-agent: TwengaBot Disallow: /User-agent: TwengaBot-2.0 Disallow: /User-agent: MJ12bot Disallow: /User-agent: BDCbot Disallow: /User-agent: spbot Disallow: /User-agent: LinkpadBot Disallow: /User-agent: WBSearchBot Disallow: /User-agent: AddThis.com robot tech.support@clearspring.com Disallow: /User-agent: AddThis.com Disallow: /User-agent: Exabot Disallow: /User-agent: YandexImages Disallow: /User-agent: Yandex Disallow: /User-agent: Slurp Disallow: /User-agent: spbot Disallow: /User-agent: AhrefsBot Disallow: /User-agent: EveryoneSocialBot Disallow: /

Sitemap: https://www.lojaintegrada-loja.com.br/sitemap.xml

Ferramentas para Gerar Robots.txt

Quer agilizar esse processo e não cometer erros? Utilize o gerador de robots da WebPeak. Esse recurso é grátis, basta se cadastrar na plataforma pelo site www.webpeak.com.br

Algumas plataformas impedem você editar o arquivo robots.txt e tem uma configurações padrão para todos os sites, estes é um exemplo de qualquer loja virtual hospedada na Loja Integrada, que tem uma peculiar filtro de bots.

Até a próxima!

• Related Articles

• Como Criar um Robots.TXT em sua loja Tray Commerce

  Olá, tudo bem? Neste artigo vamos explicar como configurar e trabalhar da melhor maneira seu arquivo Robots.txt, em específico dentro da plataforma Tray Commerce. Robots.txt é o arquivo em formato de texto inserido no site para dar “orientações” aos ...

• Auditoria Técnica

  Olá! :) A ferramenta de SEO da WebPeak é uma ferramenta completa que facilita o caminho do empreendedor digital, com grande facilidade e integrações para os lojistas virtuais. Basicamente ela é um conglomerado das principais ferramentas, para ...

• Sitemap, Robots.txt e Metatags SEO

  Olá! Nesse artigo vamos falar sobre alguns arquivos e tags indispensáveis para sua loja. Acompanhe! Sitemap.xml O sitemap como o próprio nome diz é o mapa de seu site (loja virtual), ou seja, será utilizado tais caminhos (mapa) pelos robôs dos sites ...

• Robots e Sitemap - WordPress

  Olá! Aqui vamos falar de: Plataforma: Wordpress Ferramenta: Sitemap Tire todas as suas dúvidas de forma dinâmica e rápida, respondendo dentro do tópico. É extremamente importante otimizar produtos e categorias dentro de sua plataforma de e-commerce ...

• Como gerar backlink

  Olá! Temos em nossa ferramenta acesso a portais para geração de backlinks, veja o exemplo que preparamos para você: Para gerar um backlink em portais nacionais e internacionais: acesse a categoria ‘’Perfil de Backlink’’ e o módulo ‘’Insira seu Link ...