Monday, January 24, 2011

Craigs Directory - Clean-up Update

As mentioned in my previous "Craigs Directory - The Cleanup" post, I developed a webbot to trawl all the websites listed in Craigs Directory.

Over the last 4 days it has managed to trawl over 23 thousand websites(I purposely gave it a 15 second 'wait' period between websites) and it doesn't look too bad, so far it has found;

  1. ±250 "404 not Found"
  2. ±120 "500+ Errors"
  3. and ±80 DNS issues

I have left it running, to hopefully finish the rest of the 56 thousand websites before the end of the week.

Google has also been trawling about 2.5 thousand pages of Craigs Directory per day and reported a few websites (about 140) with issues.

All the "problem" websites have been removed from the index, so once again, if you find that your website has been removed from Craigs Directory, please re-submit it.

Regards,
Craig

No comments:

Post a Comment