Another Reason Web Analytics Software Should Incorporate An Industrial-strength Bot List

I wrote a real simple script to mess with some of my friends which basically makes it seem like Googlebot visited their site, well, as many times as I wanted them to think. Don't worry, I'll provide the script in a minute. But it reminded me of a tweet from @fantomaster in which he shared how important it is to incorporate a bot list (i.e. a spider IP database) with a web analytics software/program. I can't seem to locate the tweet now (if you can, please drop it in a comment). So here's another example that further makes the case of how important it actually is...

This is a screenshot of the "Robots/Spiders visitors" report from Awstats (the default web analytics program included with cPanel). As you can see, Googlebot visited this particular domain 1,126 times since February 14th 2010, which is today. If Awstats cross-checked the IP address from which my script ran against a bot list like the one provided at searchbotBase, it would not have logged the hits. As an SEO, you can of course see the problem here in relying on such statistics. For example, this could make it real difficult for an SEO who cannot install Google Webmaster Tools and needs to diagnose crawl rates. A victim of a more calculated attack by this script might think everything is hunky dory when, in reality, their site could be experiencing real problems that may get overlooked.

The Googlebot Traffic Generation Script

#!/usr/bin/php
<?php

$referer = $argv[1];
$url = $argv[2];

$useragent = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";

for ($i =0; $i < rand(20,80); $i++) {
	system('curl -v -A '.$useragent.' -e '.$referer.' -L '.$url);
}