Forums Forums White Hat SEO Robots.txt need suggestions!!!

  • Robots.txt need suggestions!!!

    Posted by seohelper on October 14, 2021 at 6:32 am

    Is there anything wrong with this robot file, I am little confused here because the indexation is little low for this website and the site itself has more than 19,000 pages but only 5000 are currenlty indexed.

    User-agent: *

    Disallow:

    User-agent: ia_archiver-web.archive.org

    Disallow: /

    User-agent: ia_archiver

    Disallow: /

    User-agent: TurnitinBot/2.1

    Disallow: /

    User-agent: TurnitinBot

    Disallow: /

    User-agent: SurveyBot

    Disallow: /

    User-agent: EmailWolf

    Disallow: /

    User-agent: EmailSiphon

    Disallow: /

    User-agent: EmailCollector

    Disallow: /

    User-agent: AboutUsBot

    Disallow: /

    User-agent: robtexbot

    Disallow: /

    User-agent: spybot

    Disallow: /

    ​

    User-agent: spyfu

    Disallow: /

    mehul1989 replied 11 months, 3 weeks ago 1 Member · 2 Replies
  • 2 Replies
  • julienguil

    Guest
    October 14, 2021 at 9:03 am

    Hello,

    The first rule is useless :

    User-agent: *
    Disallow:

    -> (for all bots, block nothing)

    And other rules can not impact the indexation of your 14K other pages. So, your issue is outside the robots.txt.

  • mehul1989

    Guest
    October 14, 2021 at 11:59 am

    Nothing wrong with robots.txt.