Monday, January 31, 2011

Craigs Directory - Clean-up Complete?

Well, it took almost 2 weeks, by my little webbot has finished it's work and checked over 60 thousand websites submitted to Craigs Directory.

In the end, 2,585 websites have been removed from the indexes. Quite a few of these websites were removed due to 404's (page not found) but most of the errors were DNS (Domain Name Server) errors - i.e. URL not recognized?

Now the hard work starts... I have to manually go through all approved websites and ensure that they still adhere to the submission guidelines. I have found that some individuals submit valid websites, have them approved and then modify their submission to allow some of their "suspect" websites to be indexed - I will find you...


Friday, January 28, 2011

Is Good Design out of Date?

If you have been visiting Craigs Directory regularly over the last 18 months, you will have noticed that it under-went a major face-lift in the beginning of 2010.

I don't think of myself as artistic, sure I can see what it glaringly not right, but balance, poise and "Wow" are not what I do. Anyway, I am quite proud of the design I have for Craigs Directory... then I met a web designer yesterday whos main focus is on usability and focus. Seems it's not only look and feel side of things that I need to understand?

Anyway, watch this space, it looks like Craigs Directory may be in for another face-lift.


Thursday, January 27, 2011

Think carefully before you agree

I got an email from my ISP yesterday saying that thay noticed that I send out a lot of emails from my website and would I be interested in one of their "Email Quota" packages.

Hmmm, sound interesting, so I had a look at the offers...

  1. 2000 email recipients per month $4.95
  2. 4000 email recipients per month $6.95
  3. 6000 email recipients per month $8.95

Doesn't seem too bad? But then I had a look at my current email package...

  1. its included in my hosting package
  2. its limited to 200 emails per hour

So, hypothetically, I can currently send out 200 emails per hour. In one day that would be 4800 emails! Over the course of the month 144,000 emails.

I think I'll pass thanks.


Wednesday, January 26, 2011

Craigs Directory - Clean-up Update II

The webbot I wrote last week has now been running against Craigs Directory for 5 days and so far has checked ±39,000 of the ±55,000 websites. This check basically ensures that the website submitted to the web directory is still available, and if not, removes it from the index. No point linking to websites that no longer exist?

So far, of the ±39,000 websites checked, over 2,000 websites have been removed!

I find this trend quite worrying. People get these great ideas (or are sold lemons) and setup a website using free ISP's, blogs, wordpress, etc. They give it a few months and then give up, letting the website die.

Keep trying, be persistent, it will work.


Tuesday, January 25, 2011

Craigs Directory - Lessons Learned

Craigs Directory has been live for about 18 months now. It started as a simple web directory that ranked websites by credits. I admit I am a "techie" person and originally built it as a challenge and maybe because I was a little bored. However, its success has taught me quite a bit over the last 18 months, some good lessons and some bad.


Search Engine Optimization (SEO) is not a one-time deal, it is ongoing. You need to keep at it, keep tweaking and keep promoting.

Google PageRank

This is probably only important to me because most people who submit their websites to Craigs Directory are keen to increase their own Google Ranking. However, I believe PageRank is always a few months out-of-date (in the Google toolbar) and Google wants us to focus more on ROI and not on visible pagerank.

That being said, Google penalized me in October 2010 for not adhereing to their strict guidelines - by removing my visible pagerank. My search results have not changed, so I know that Google has not removed me from their indexes, but I have noticed a 25% drop in daily submissions (±200 down to ±150 per day).

I have now corrected the reason for my penalty and hope to have my visible pagerank restored, but I am not too concerned as it has not effected me too much. BTW, Google still crawls about 2,500 pages of Craigs Directory per day.


Email is still a good communication tool. Through Craigs Directory I send out numerouos emails, the purpose being to keep people up to date on the rank of their website. Unfortunately, there have been times where I have had to send out emails to let people know that their website has been declined for not adhering to my submission guidelines (basically no sex, drugs or hacking content). Suffice to say, there are some people out there who take things far too personally, enough said.

Whether it be through rejection or just plain malice, there have been attempts to damage Craigs Directory or is reputation. Over the last 18 months I have had to deal with really offensive emails, hacking attempts and even having my email being black-listed by Yahoo (that battle is still ongoing).

I have learnt a lot over the last 18 months, but mostly I have learned about people. SEO is all about ranking in the search engines, but we tend to forget that it is "people" that type in the search query.


Monday, January 24, 2011

Craigs Directory - Clean-up Update

As mentioned in my previous "Craigs Directory - The Cleanup" post, I developed a webbot to trawl all the websites listed in Craigs Directory.

Over the last 4 days it has managed to trawl over 23 thousand websites(I purposely gave it a 15 second 'wait' period between websites) and it doesn't look too bad, so far it has found;

  1. ±250 "404 not Found"
  2. ±120 "500+ Errors"
  3. and ±80 DNS issues

I have left it running, to hopefully finish the rest of the 56 thousand websites before the end of the week.

Google has also been trawling about 2.5 thousand pages of Craigs Directory per day and reported a few websites (about 140) with issues.

All the "problem" websites have been removed from the index, so once again, if you find that your website has been removed from Craigs Directory, please re-submit it.


Friday, January 21, 2011

Craigs Directory - Top Boosters

In my "Craigs Directory - The History" post, I mentioned as part of my updates, that I was going to update the status of my Twitter, Facebook and Google Buzz accounts with the list of websites that have boosted their rank in Craigs Directory with 2,000 or more credits.

I finally managed to complete this task this morning for all the historic submissions. Going forward, this will become a daily task as more websites boost their rank in Craigs Directory.

The complete (and hopefully growing) list of these websites is this;

If you want your website listed here, submit it to Craigs Directory and select 2000 or more credits in the "Instant" listing option.


Thursday, January 20, 2011

Craig's Directory - Clean-up

I decided yesterday, to have a look at a sample of the websites I had approved over the last 18 months. Surprisingly, quite a few of the websites no longer exist, or the page being linked to has been removed?

Any guesses how long it would take me to check over 56 thousand web page manually? My broadband provider would love me, but sadly, my wife wouldn't.

I therefore developed a small webbot to do the checking for me. After checking a website it writes the response status codes (200, 404, etc) into the database. All I have to do now is review all the negative response codes and remove the defunct websites from my listings.

So, if you find your website no longer exists in Craigs Directory, apologies, but please check the page I was linking to - it may no longer exist. If this is the case, please re-submit your website.


Wednesday, January 19, 2011

Craigs Directory - The History

Since its launch in June 2009, Craigs Directory has been visited by over 3.5 million visitors and has over 50 thousand websites listed.

This is not my first website, that was Craigs Minis, which I launched in 2004. To date it has only seen about 37 thousand visitors, but then again, it is very niche. I have developed a few other websites, but none have proved to be as succesful as Craigs Directory. Life's a funny thing, I spent 18 months developing a website that I was sure would succeed - it bombed, but with Craigs Directory I only spent about a week with the original design (sure, I have tweaked it a lot since then). Sometimes its the simple things in life that work the best.

The concept of ranking websites by credits (earned through clicks or purchases) was there from the beginning, but I must be honest in saying I never expected it to work so well. To date, over 2.7 million credits have been issued - which is well above my expectations.

Over the last year and a bit, I have made a few changes and put a few things in place to reward websites with lots of credits. These include:

  • [±July 2009] Introduced the TOP 25 websites page
  • [±May 2010] Show the top 3 websites (i.e. the highest credits) on every page of the directory
  • [±May 2010] Show the top 9 websites on the home page
  • [±May 2010] Show the top 9 websites of each category on the sub-category selection page
  • [±June 2010] Developed a completely new web directory whos sole purpose is to list all websites in Craigs Directory that have earned credits by buying them.
  • [±August 2010] Developed a Mobile version of Craigs Directory - lists/ranks all approved websites
  • [±November 2010] Change the Add a Website to https (secure server)
  • [18 January 2011] Created a Twitter and Facebook account for Craigs Directory
  • [18 January 2011] Started updating the status of the Twitter, Facebook and my Google Buzz accounts with the urls of websites who purchase 2,000 or more credits ($10)