Снижаем нагрузку на сервер, заблокировав ботов


AhrefsBot - Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
MJ12bot - Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Detectify - Mozilla/5.0 (compatible; Detectify) +https://detectify.com/bot/
DotBot - Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
Riddler - Riddler (http://riddler.io/about)
SemrushBot - Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
LinkpadBot - Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
BLEXBot Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
FlipboardProxy - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2), Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
aiHitBot - Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
trovitBot - Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)

Просто заблокируйте мусор через htaccess:

RewriteCond %{HTTP_USER_AGENT} AhrefsBot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} MJ12bot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} Detectify
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} dotbot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} Riddler
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} SemrushBot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} LinkpadBot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} BLEXBot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} FlipboardProxy
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} aiHitBot
RewriteRule (.*) - [F,L]
RewriteCond %{HTTP_USER_AGENT} trovitBot
RewriteRule (.*) - [F,L]