Along with identifying individual robots and counting number of their visits, statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. In resources section of end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them.

How Do They Read The Pages On Your Website? When search engine robot visits your page, it looks at visible text on page, content of various tags in your page's source code (title tag, meta tags, etc.), and hyperlinks on your page. From words and links that robot finds, search engine decides what your page is about. There are many factors used to figure out what "matters" and each search engine has its own algorithm in order to evaluate and process information. Depending on how robot is set up through search engine, information is indexed and then delivered to search engine's database.

The information delivered to databases then becomes part of search engine and directory ranking process. When search engine visitor submits their query, search engine digs through its database to give final listing that is displayed on results page.

The search engine databases update at varying times. Once you are in search engine databases, robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they have latest info. The number of times you are visited depends on how search engine sets up its visits, which can vary per search engine.

Sometimes visiting robots are unable to access website they are visiting. If your site is down, or you are experiencing huge amounts of traffic, robot may not be able to access your site. When this happens, website may not be re-indexed, depending on frequency of robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then.

Resources

SpiderSpotting - Search Engine Watch

http://searchenginewatch.com/webmasters/spiders.html

Robotstxt.org

List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

Spider-Food

Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

Spiderhunter.com

Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

Sim Spider Search Engine Robot Simulator

Search Engine World has a spider that simulates what Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

While search engine marketing person cannot guarantee you a position, what they can do is to apply years of experience to tell you what has worked in past, and to help you make it work today. In many ways, "organic" search engine optimization is really a matter of editing web pages or whole sites to make them most search engine friendly they can be. Making sure that a given page has just right combination of keywords, title, links, and so on, is really at its base simply a matter of making that page best web page it can possibly be. The page that will rank best in search engines is also page that will make most sense to human visitor. Rather than relying on tricks to try to make page rank high, it is a matter of just making page most focused and on-message that it can be. The bad news is that this doesn't guarantee which position in search engine rankings that page will occupy on a given day. The good news is that page will always rank well.

The search engines change their algorithms from time to time. If today's rule, for example, is that just right combination of text in title tag will make or break a site, and you know this is true, then all you have to do is to tweak title tag to fit within that rule, and you will automatically rank very high. Today.

Suppose that tomorrow, however, that rule is changed. Suppose that now most important factor that search engines use to rank a site is content of META Description tag. All work you went to yesterday to fix title is now useless. All of your attention is now focused on fixing that description tag.

Clearly, over time focus of search engines will vary. The best way to deal with this is to not deal with it! This means that rather than tweaking a site one way today and another way tomorrow, best way to approach optimizing a page or a whole site is to not try to beat system. Instead of trying to "psych-out" search engines, why not add value to site? A "common sense" approach to search engine optimization, looking for long term results, is way to go. When you try to help a site rank better by making it best it can be, everybody wins.