Reply To: Will this code block most SEO bots?

Forum White Hat SEO Will this code block most SEO bots? Reply To: Will this code block most SEO bots?

#12449
josourcing

There are too many bots to list them all like that. That’s why I block all the ones that have “bot” in their name, like this:

RewriteCond %{HTTP_USER_AGENT} !^(.*)googlebot(.*)$ [NC]
RewriteCond %{HTTP_USER_AGENT} !^(.*)twitterbot(.*)$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^(.*)bot(.*)$ [NC]
RewriteRule ^(.*)$ – [F,L]

In that example, Google and Twitter are exempted. All the others go to hell.

Scroll to Top