Block Bad Bots with Htaccess

Posted on July 26, 2013 at 7:18 pm

Bots are automated scripts that are used to crawl your website. Legit bots are, for example, Google Bot or Bing Bot used to index the pages on your website. Bad bots are scripts used to find vulnerabilities on your website by crawling web pages to search for vulnerable scripts, or spam bots that steal email addresses found in the HTML code in plain text format.

To block bad bots add this in your .htaccess file:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^(curl|wget).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|scan).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR] 
RewriteRule ^(.+)$ - [F]
</IfModule>

If you want to log all blocked connections, use this:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^(curl|wget).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|scan).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR] 
RewriteRule ^(.+)$ - [env=bots:true]
</IfModule>
CustomLog /path/to/logs/bots.log combined env=bots

Updated on October 22, 2013 at 3:57 pm

Receive updates via email

Other Posts

Updated Posts