Forums › Forums › White Hat SEO › Will this code block most SEO bots? › Reply To: Will this code block most SEO bots?

  • josourcing

    Guest
    November 10, 2020 at 1:40 am

    There are too many bots to list them all like that. That’s why I block all the ones that have “bot” in their name, like this:

    RewriteCond %{HTTP_USER_AGENT} !^(.*)googlebot(.*)$ [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(.*)twitterbot(.*)$ [NC]
    RewriteCond %{HTTP_USER_AGENT} ^(.*)bot(.*)$ [NC]
    RewriteRule ^(.*)$ – [F,L]

    In that example, Google and Twitter are exempted. All the others go to hell.