• White Hat SEO

  Robots.txt need suggestions!!!

  Posted by seohelper on October 14, 2021 at 6:32 am

  Is there anything wrong with this robot file, I am little confused here because the indexation is little low for this website and the site itself has more than 19,000 pages but only 5000 are currenlty indexed.

  User-agent: *

  Disallow:

  User-agent: ia_archiver-web.archive.org

  Disallow: /

  User-agent: ia_archiver

  Disallow: /

  User-agent: TurnitinBot/2.1

  Disallow: /

  User-agent: TurnitinBot

  Disallow: /

  User-agent: SurveyBot

  Disallow: /

  User-agent: EmailWolf

  Disallow: /

  User-agent: EmailSiphon

  Disallow: /

  User-agent: EmailCollector

  Disallow: /

  User-agent: AboutUsBot

  Disallow: /

  User-agent: robtexbot

  Disallow: /

  User-agent: spybot

  Disallow: /

  ​

  User-agent: spyfu

  Disallow: /

  mehul1989 replied 3 years, 7 months ago 1 Member · 2 Replies
• 2 Replies

• 

  julienguil

  Guest

  October 14, 2021 at 9:03 am

  Hello,

  The first rule is useless :

  User-agent: *
  Disallow:

  -> (for all bots, block nothing)

  And other rules can not impact the indexation of your 14K other pages. So, your issue is outside the robots.txt.

• 

  mehul1989

  Guest

  October 14, 2021 at 11:59 am

  Nothing wrong with robots.txt.

Log In to Reply