Search Engine Robots - How They Work, What They Do (Part I)Written by Daria Goetsch
Automated search engine robots, sometimes called "spiders" or "crawlers", are seekers of web pages. How do they work? What is it they really do? Why are they important?
How Do Search Engine Robots Work? Think of search engine robots as automated data retrieval programs, traveling web to find information and links.
When you submit a web page to a search engine at "Submit a URL" page, new URL is added to robot's queue of websites to visit on its next foray out onto web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of reasons why it is important to build your link popularity and to get links from other topical sites back to yours.
When arriving at your website, automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files robot doesn't need to concern itself with.
Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow links from one page to another. The entire World Wide Web is made up of links, original idea being that you could follow links from one place to another. This is how robots get around.
The "smarts" about indexing pages online comes from search engine engineers, who devise methods used to evaluate information search engine robots retrieve. When introduced into search engine database, information is available for searchers querying search engine. When a search engine user enters their query into search engine, there are a number of quick calculations done to make sure that search engine presents just right set of results to give their visitor most relevant response to their query.
You can see which pages on your site search engine robots have visited by looking at your server logs or results from your log statistics program. Identifying robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google's "Googlebot"; others are bit more obscure, like Inktomi's "Slurp". Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers.
The Myth of "Guaranteed #1 Ranking" in Search Engine MarketingWritten by Dale Goetsch
You've seen ads: Guaranteed #1 Ranking! There are no guarantees in search engine marketing and website promotion. If anyone tells you different, you should check quickly to make sure they don't have their hand in your wallet.
Suppose you sell widgets. You want to sell more widgets, and way to do that is to make sure that more people know about widgets, and that you are place to buy their widgets. You might decide to buy a half-page ad in a national magazine to tell your story. When you place that ad, you are "guaranteed" your position.
With a magazine advertisement, you know what magazine's circulation is, who reads it, and which page will feature your ad. The magazine can guarantee all that, because they own medium.
Search engine marketing is qualitatively different. When you work with a search engine marketing firm to promote your website, they cannot guarantee where your listing will appear. Certainly there are types of online ads where there are guarantees in place: banner ads priced at "cost per thousand impressions", pop-up ads, and so forth. These are like traditional media buys, where you are working directly with owner of medium where ads appear, but this is not search engine marketing.
Even so-called pay-per-click search engines cannot guarantee your position. In Google AdWords, for example, it is not just price you pay for a given keyword that determines where you will rank. They also bring in other factors, including how often your ad is clicked-on, to determine which ad will be listed first. Just throwing money at them will not necessarily get you into #1 spot.
The bottom line is this: search engine marketing professionals do not own search engines. They can tell you that you will achieve #1 ranking on a given search engine, or they can tell you that moon is made of green cheese, but there is no way they can make either of those happen. When you tell Time magazine you want your ad to be on back cover, and you pay them enough money, they will guarantee you back cover. If you tell your search engine marketing people you want to be #1 in AllTheWeb, they cannot guarantee you that result. They can recommend changes to your site that will increase likelihood of your ranking higher, but that is a long way from a guarantee. If you don't control medium, you can't guarantee result. Since your search engine consultant doesn't control search engine, there is no way they can guarantee your position.
The ranking algorithms of search engines are a closely-guarded secret. The search engine wants to give top ranking to site that is best match to an individual visitor's search query, not to site that was able to "beat" system. That is where value of real search engine marketing comes in.