I have been doing a lot lately to get Google more interested in my sites and have found some interesting things you may or may not be aware of. So if you are technical minded at all this weeks newsletter is definetly for you.
The TOUCH Command
A lot of search engine like to look at UPDATED content. So one way to give them that is to modify your files. Well it isn't easy to go into your website and edit each of your files every day or even every week so if you are running on a linux or unix box you can use the TOUCH command.
I am sure there is a similar command on windows boxes but I am not famiiar enough with them to know what it is.
When you TOUCH a file it updates the date of the file so any web browser looking at it will see that date and not the date you actually updated the information in the file.
If you just have one website you would go into your sites root directory and use touch *. That would hit every file in the directory. Just look up Linux Touch on any search engine and you will find all kinds of different variation to hit all the subdirectories and subdirectories or files you don't want touched. This is a good thing if you are using dynamic file creation. This pages already show up with the current date and time.
You can go one step further and set this up as a daily or weekly cron job. I have set mine up as weekly and use a pretty large variation of the command I will explain here.
Because I run my own servers I have it hitting every file almost in my web directories:
find /home/sites/ -not -path "*/web/stats/*" -not -path "*/web/sl/*" -not -path "*/web/us?/*" -not -path "*/web/st?/*" -not -path "*/web/cgibin/*" -not -path "*/web/flinks/*" -not -path "*/users/*" -not -path "*/logs/*" -not -path "*/certs/*" \( -name "*" \) -exec touch * {} \;
What happens here is it first FINDS my directory of websites located at /home/sites, the it eliminates the stats and various other directories including the us? and st? dir's which are really dynamic directories. Then it TOUCHES everything that falls within the paramaters. WHAM my dates have all changed and a lot of the search engines that don't do content checking think they have been updated. Hence higher rankings in the search results.
This does lead to some problems I will explain further below.
Spidering:
My next step was to update all the sitemaps on my site so they reflected not only the real files there but also the dynamic ones. Some of my sites have well over 200k pages so most of the web based programs just wouldn't cut it and very few of my sites have less then 500 to 700 pages which is more then most web based sites can handle. Usually their limit is about 500. I did find a really interesting program that is windows based at: TryToBreak.com.
I tested out a ton of programs to do this but this just seemed so close to what I wanted I started using it. Well.... some problems developed. First I found that if my html wasn't absolutely perfect it just wouldn't follow the links even though all the links were tested with MSIE and netscape, firefox and even AOL's version of MSIE. This is telling me that the search engines spiders really do get held up badly too when they come to some bad html. Even something as simple as missing " or a > of some type.
My next step is to install a php based sitemap program on the servers and set them up to run immediately after the cron job runs for the touch. There is a reason for that. It seems that if the dates on the sitemap don't match the pages or the date on the sitemap file is later then on the real files thats a big NO NO for the engines.
Another problem I found was the date format. This sitemap program put the last mod date something like 2007-1-23. Well I kept getting errors on it. Something struck me and i went in with a blanket find and replace program and changed all the sitemap files dates in the lastmod field to 2007-01-23 and it was accepted without error. Keep this in mind when you have a problem with getting a sitemap accepted.
Google
Ok I had now updated my files so they all looked newly modified and then created an acceptable sitemap to upload to google and what do I find?
To get any real information about crawling my pages there I have to VERIFY that I am the site owner. I can do this by putting a google meta tag in the head portion of my page. This went along ok but having to put that code in all my sites is turning out to be a heck of a job but I do not see a way around it.
Again html syntax is really really important. I had one site that I just couldn't get it to accept the tag. It kept telling me it wasn't there. So of course I FTPed the file in several times. Changed the position of the tag a couple times and even moved it to the first meta tag under the title. It still wouldn't take it. I looked at the code and didn't see anything wrong until instead of looking at it in my text editor I pulled up the page in an OLD OLD OLD version of Netscape and viewed the source of the page that way. Well... apparently when the meta tags were put on the page someone missed a closing > on the description tag. This STOPPED the entire process of google being able to find their tag. Now... As I think back I bet it also stopped their spider too. Once I fixed it it was fine and the verify said OK and it found it immediately. A strange thing was that even when the tag was before the error. It still wouldn't find it. So thats telling me that even a SMALL ERROR on your page can stop a spider DEAD IN ITS TRACKS.
The next problem I found at Google was my robots.txt tag. What was happening was that I had a crawl delay of 10 in the tag. The reason being on my old servers if a spider hit several sites with say 100k pages each it would bring the server to its knees. Well.... with the delay the spiders didn't want to wait that long so they just left. I have now modified the robots.txt file on all the servers so that it has no delay now and it seems like the spider bait is being chewed up at an amazing pace.
What I had to do though was move all the big sites to dual processor servers using the 15,000 rpm drives and I can still see the loads going up when they are being hit hard by several spiders at several different sites on each server. But that is an easy fix. Buy more servers as time goes on.
So bottom line of this discussion is:
1. Take a look at using the TOUCH command to update your sites page dates.
2. Make sure all your pages have perfectly formated html
3. Make sure your sitemaps have all the correct elements and formats in them.
4. Take a look at your robots.txt file and make sure that there isn't something in there that is turning away the bots.
Next week I will get into the Yahoo maps and pinging google with your updated sitemaps. Please take a look at the special below. It is a great deal!
Dynamic Pages
The newest experiment in Dynamic Pages seems to be rolling along really good. Here is one of the new sites I have put up using them. Fake Diplomas
I have orders for several more and all the spiders seem to be eating up the pages right now. Only time will tell how many pages get indexed one each site, but with the 3 links at the bottom of each page pointing back to your site they should do well if it indexes all the pages or not.
I am going to still honor the sales offer of 149.00 each until i get 25 more orders then the price will go back up,. i figure anyone can afford it at 149.00 and since it is still in test mode they aren't risking much. After the testing phase is over I will adjust the price back up accordingly. You can get on the list for one of these below. Please let me know the url of the site you want to promote and your primary 3 key phrases so i can register a new domain that fits your sites topic and key phrases.
ORDERS ONLY $149.00!
EMAIL SERVER WANTED
If anyone out there is doing huge amounts of emails and has a server or server space for rent, please let me know. I acquired a huge email list that is supossed to be a subscribed one but I am not really ready to start emailing over my existing T1 lines with an unknown list. So if anyone has an upline isp that doesn't mind too much or their own servers setup for mailing please let me know. I am very interested. This list that I bought is really huge and some of the people that are mailing are bringing in upwards of 1000.00 bucks a day profit from it. Of course I will believe that when I see it.
DOMAINS FOR SALE:
I have a few more for domains for sale yet. They are all listed at my EBAY STORE, but I would be happy to accept offers on them. If anyone is interested please email me at joed@joed.com Then make me an offer I can't refuse!
# YOUR-FANTASY.COM
Just a parked domain with a holder page there. Great name though just never did anything with it. I may turn this one into one of the new format sites if I don't sell it soon.
# DARKSIDEOFTHENET.COM
One of the oldest and most respected web sites out there. It has a huge following with the Gothic crowd. Lots of hits but I have left it set dormant for close to 2 years. The existing site is included with the name. I will zip up the site and email it to the buyer.
# FOXXII.COM
This has a great name that will be well rememberd no matter what site you put there. It would be great for a pic post site. Am I Foxxii?
# BIKEZINE.COM
Another old site that is really in with the biker and harley crowd. The site is included in the sale price and once again I will zip up the site for you. .
Well thats it for this week. Thanks for reading.
Joe
http://www.search-engine-info.com
