-
Will this code block most SEO bots?
I was going to put this in my htaccess:
SetEnvIfNoCase User-Agent .*aboundexbot.* botstop
SetEnvIfNoCase User-Agent .*ahrefsbot.* botstop
SetEnvIfNoCase User-Agent .*backlinkcrawler.* botstop
SetEnvIfNoCase User-Agent .*blekkobo.* botstop
SetEnvIfNoCase User-Agent .*blexbot.* botstop
SetEnvIfNoCase User-Agent .*dotbot.* botstop
SetEnvIfNoCase User-Agent .*dsearch.* botstop
SetEnvIfNoCase User-Agent .*exabot.* botstop
SetEnvIfNoCase User-Agent .*ezooms.* botstop
SetEnvIfNoCase User-Agent .*gigabot.* botstop
SetEnvIfNoCase User-Agent .*ia_archiver.* botstop
SetEnvIfNoCase User-Agent .*linkdexbot.* botstop
SetEnvIfNoCase User-Agent .*lipperhey spider.* botstop
SetEnvIfNoCase User-Agent .*majestic-12.* botstop
SetEnvIfNoCase User-Agent .*majestic-seo.* botstop
SetEnvIfNoCase User-Agent .*meanpathbot.* botstop
SetEnvIfNoCase User-Agent .*megaindex.* botstop
SetEnvIfNoCase User-Agent .*mj12bot.* botstop
SetEnvIfNoCase User-Agent .*ncbot.* botstop
SetEnvIfNoCase User-Agent .*nutch.* botstop
SetEnvIfNoCase User-Agent .*pagesinventory.* botstop
SetEnvIfNoCase User-Agent .*rogerbot.* botstop
SetEnvIfNoCase User-Agent .*scoutjet.* botstop
SetEnvIfNoCase User-Agent .*searchmetricsbot.* botstop
SetEnvIfNoCase User-Agent .*semrushbot.* botstop
SetEnvIfNoCase User-Agent .*seokicks-robot.* botstop
SetEnvIfNoCase User-Agent .*sistrix.* botstop
SetEnvIfNoCase User-Agent .*sitebot.* botstop
SetEnvIfNoCase User-Agent .*spbot.* botstop
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=botstop
</Limit>I was wondering if that includes all the major players.
I am sick of all my competitors big lazy stupid *&$! copycats!
They copy my stuff and because they have more site authority they outrank me. So lame.
Now I know someone will say “it won’t work”. Blah blah to you.
-
4 Replies
-
Don’t forget **SEO-PowerSuite-Bot**.
-
I’m curious, what are you trying to block them from seeing exactly? They will still be able to see your backlinks and keyword rankings. It may make it harder for them to figure out which content on your site is helping drive your traffic, but they will still be able to see all your public site content.
-
It will only stop the polite bots that identify themselves.
-
There are too many bots to list them all like that. That’s why I block all the ones that have “bot” in their name, like this:
RewriteCond %{HTTP_USER_AGENT} !^(.*)googlebot(.*)$ [NC]
RewriteCond %{HTTP_USER_AGENT} !^(.*)twitterbot(.*)$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^(.*)bot(.*)$ [NC]
RewriteRule ^(.*)$ – [F,L]In that example, Google and Twitter are exempted. All the others go to hell.
Log in to reply.