Robots.txt need suggestions!!!

Forum White Hat SEO Robots.txt need suggestions!!!

  • This topic has 2 replies, 1 voice, and was last updated 7 months ago by mehul1989.
Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #26851
    seohelper
    Keymaster

    Is there anything wrong with this robot file, I am little confused here because the indexation is little low for this website and the site itself has more than 19,000 pages but only 5000 are currenlty indexed.

    User-agent: *

    Disallow:

    User-agent: ia_archiver-web.archive.org

    Disallow: /

    User-agent: ia_archiver

    Disallow: /

    User-agent: TurnitinBot/2.1

    Disallow: /

    User-agent: TurnitinBot

    Disallow: /

    User-agent: SurveyBot

    Disallow: /

    User-agent: EmailWolf

    Disallow: /

    User-agent: EmailSiphon

    Disallow: /

    User-agent: EmailCollector

    Disallow: /

    User-agent: AboutUsBot

    Disallow: /

    User-agent: robtexbot

    Disallow: /

    User-agent: spybot

    Disallow: /

    ​

    User-agent: spyfu

    Disallow: /

    #26852
    julienguil

    Hello,

    The first rule is useless :

    User-agent: *
    Disallow:

    -> (for all bots, block nothing)

    And other rules can not impact the indexation of your 14K other pages. So, your issue is outside the robots.txt.

    #26853
    mehul1989

    Nothing wrong with robots.txt.

Viewing 3 posts - 1 through 3 (of 3 total)
  • You must be logged in to reply to this topic.
Scroll to Top