• White Hat SEO

  Robots.txt

  Posted by seohelper on April 9, 2020 at 6:25 pm

  Hello,

  Im making an audit for a client and one of the topics is Robots.txt .We usually write something like “Your robots.txt file is not full”

  So i wondered – Is Robots.txt individually written for every site or it has to be written the same way for every site?

  guanacazo replied 5 years, 1 month ago 1 Member · 3 Replies
• 3 Replies

• 

  guanacazo

  Guest

  April 9, 2020 at 6:30 pm

  You can use the same for the bots that actually respect it, if it is a very basic one. If you want to list pages you dont want indexed then you need one for every site. Anyway the bad guys always go for the pages you dont want listed. Lol.

• 

  pixgarden

  Guest

  April 9, 2020 at 9:26 pm

  >Is Robots.txt individually written for every site or it has to be written the same way for every site?

  the most likely scenarios are:

  * There is no robots.txt and that’s not necessarily a problem
  * the cms or plugin cms create robots.txt file, that’s why many of them follow the same format.

• 

  Anastasya99

  Guest

  April 10, 2020 at 11:27 am

  **Can this one be used as a basic for most sites?**

  User-agent:

  * Disallow: /administrator/ Disallow:

  /scripts/ Disallow: /cache/

  Disallow: /components/

  Disallow: /includes/

  Disallow: /installation/

  Disallow: /language/

  Disallow: /libraries/

  Disallow: /media/

  Disallow: /modules/

  Disallow: /templates/

  Disallow: /tmp/

  Disallow: /xmlrpc/

  Disallow: /catalog/view/theme/bearstore_v1/flash/tagcloud.xml

  Disallow: /unsubscribed-from-reviews

  Disallow: /product/compare

  Disallow: /*search&tag

  Disallow: /*search&filter_tag

  Disallow: /*sort=

  Disallow: /*limit=

  Disallow: /*order=

  Disallow: /plugins/feedback.php Sitemap: https://*******.com/feed/google_sitemap

Log In to Reply