You'd think with all fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can't do certain things. Robots don't understand frames, Flash movies, images or JavaScript. They can't enter password protected areas and they can't click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation.

How Do Search Engine Robots Work? Think of search engine robots as automated data retrieval programs, traveling web to find information and links.

When you submit a web page to a search engine at "Submit a URL" page, new URL is added to robot's queue of websites to visit on its next foray out onto web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of reasons why it is important to build your link popularity and to get links from other topical sites back to yours.

When arriving at your website, automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files robot doesn't need to concern itself with.

Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow links from one page to another. The entire World Wide Web is made up of links, original idea being that you could follow links from one place to another. This is how robots get around.

The "smarts" about indexing pages online comes from search engine engineers, who devise methods used to evaluate information search engine robots retrieve. When introduced into search engine database, information is available for searchers querying search engine. When a search engine user enters their query into search engine, there are a number of quick calculations done to make sure that search engine presents just right set of results to give their visitor most relevant response to their query.

Suppose you sell widgets. You want to sell more widgets, and way to do that is to make sure that more people know about widgets, and that you are place to buy their widgets. You might decide to buy a half-page ad in a national magazine to tell your story. When you place that ad, you are "guaranteed" your position.

With a magazine advertisement, you know what magazine's circulation is, who reads it, and which page will feature your ad. The magazine can guarantee all that, because they own medium.

Search engine marketing is qualitatively different. When you work with a search engine marketing firm to promote your website, they cannot guarantee where your listing will appear. Certainly there are types of online ads where there are guarantees in place: banner ads priced at "cost per thousand impressions", pop-up ads, and so forth. These are like traditional media buys, where you are working directly with owner of medium where ads appear, but this is not search engine marketing.

Even so-called pay-per-click search engines cannot guarantee your position. In Google AdWords, for example, it is not just price you pay for a given keyword that determines where you will rank. They also bring in other factors, including how often your ad is clicked-on, to determine which ad will be listed first. Just throwing money at them will not necessarily get you into #1 spot.

The bottom line is this: search engine marketing professionals do not own search engines. They can tell you that you will achieve #1 ranking on a given search engine, or they can tell you that moon is made of green cheese, but there is no way they can make either of those happen. When you tell Time magazine you want your ad to be on back cover, and you pay them enough money, they will guarantee you back cover. If you tell your search engine marketing people you want to be #1 in AllTheWeb, they cannot guarantee you that result. They can recommend changes to your site that will increase likelihood of your ranking higher, but that is a long way from a guarantee. If you don't control medium, you can't guarantee result. Since your search engine consultant doesn't control search engine, there is no way they can guarantee your position.