Will this code block most SEO bots?

Forum White Hat SEO Will this code block most SEO bots?

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #12447
    seohelper
    Keymaster

    I was going to put this in my htaccess:

    SetEnvIfNoCase User-Agent .*aboundexbot.* botstop
    SetEnvIfNoCase User-Agent .*ahrefsbot.* botstop
    SetEnvIfNoCase User-Agent .*backlinkcrawler.* botstop
    SetEnvIfNoCase User-Agent .*blekkobo.* botstop
    SetEnvIfNoCase User-Agent .*blexbot.* botstop
    SetEnvIfNoCase User-Agent .*dotbot.* botstop
    SetEnvIfNoCase User-Agent .*dsearch.* botstop
    SetEnvIfNoCase User-Agent .*exabot.* botstop
    SetEnvIfNoCase User-Agent .*ezooms.* botstop
    SetEnvIfNoCase User-Agent .*gigabot.* botstop
    SetEnvIfNoCase User-Agent .*ia_archiver.* botstop
    SetEnvIfNoCase User-Agent .*linkdexbot.* botstop
    SetEnvIfNoCase User-Agent .*lipperhey spider.* botstop
    SetEnvIfNoCase User-Agent .*majestic-12.* botstop
    SetEnvIfNoCase User-Agent .*majestic-seo.* botstop
    SetEnvIfNoCase User-Agent .*meanpathbot.* botstop
    SetEnvIfNoCase User-Agent .*megaindex.* botstop
    SetEnvIfNoCase User-Agent .*mj12bot.* botstop
    SetEnvIfNoCase User-Agent .*ncbot.* botstop
    SetEnvIfNoCase User-Agent .*nutch.* botstop
    SetEnvIfNoCase User-Agent .*pagesinventory.* botstop
    SetEnvIfNoCase User-Agent .*rogerbot.* botstop
    SetEnvIfNoCase User-Agent .*scoutjet.* botstop
    SetEnvIfNoCase User-Agent .*searchmetricsbot.* botstop
    SetEnvIfNoCase User-Agent .*semrushbot.* botstop
    SetEnvIfNoCase User-Agent .*seokicks-robot.* botstop
    SetEnvIfNoCase User-Agent .*sistrix.* botstop
    SetEnvIfNoCase User-Agent .*sitebot.* botstop
    SetEnvIfNoCase User-Agent .*spbot.* botstop
    <Limit GET POST HEAD>
    Order Allow,Deny
    Allow from all
    Deny from env=botstop
    </Limit>

    I was wondering if that includes all the major players.

    I am sick of all my competitors big lazy stupid *&$! copycats!

    They copy my stuff and because they have more site authority they outrank me. So lame.

    Now I know someone will say “it won’t work”. Blah blah to you.

    #12450
    boostedgts

    Don’t forget **SEO-PowerSuite-Bot**.

    #12448
    ModemSEO

    I’m curious, what are you trying to block them from seeing exactly? They will still be able to see your backlinks and keyword rankings. It may make it harder for them to figure out which content on your site is helping drive your traffic, but they will still be able to see all your public site content.

    #12451
    Edward_Morbius

    It will only stop the polite bots that identify themselves.

    #12449
    josourcing

    There are too many bots to list them all like that. That’s why I block all the ones that have “bot” in their name, like this:

    RewriteCond %{HTTP_USER_AGENT} !^(.*)googlebot(.*)$ [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(.*)twitterbot(.*)$ [NC]
    RewriteCond %{HTTP_USER_AGENT} ^(.*)bot(.*)$ [NC]
    RewriteRule ^(.*)$ – [F,L]

    In that example, Google and Twitter are exempted. All the others go to hell.

Viewing 5 posts - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.
Scroll to Top