Search Engine Robots - How They Work, What They Do (Part I)

Written by Daria Goetsch

Automated search engine robots, sometimes called "spiders" or "crawlers", are repparttar seekers of web pages. How do they work? What is it they really do? Why are they important?

You'd think with allrepparttar 128141 fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can't do certain things. Robots don't understand frames, Flash movies, images or JavaScript. They can't enter password protected areas and they can't click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation.

How Do Search Engine Robots Work? Think of search engine robots as automated data retrieval programs, traveling repparttar 128142 web to find information and links.

When you submit a web page to a search engine atrepparttar 128143 "Submit a URL" page,repparttar 128144 new URL is added torepparttar 128145 robot's queue of websites to visit on its next foray out ontorepparttar 128146 web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one ofrepparttar 128147 reasons why it is important to build your link popularity and to get links from other topical sites back to yours.

When arriving at your website,repparttar 128148 automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other filesrepparttar 128149 robot doesn't need to concern itself with.

Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially followrepparttar 128150 links from one page to another. The entire World Wide Web is made up of links,repparttar 128151 original idea being that you could follow links from one place to another. This is how robots get around.

The "smarts" about indexing pages online comes fromrepparttar 128152 search engine engineers, who deviserepparttar 128153 methods used to evaluaterepparttar 128154 informationrepparttar 128155 search engine robots retrieve. When introduced intorepparttar 128156 search engine database,repparttar 128157 information is available for searchers queryingrepparttar 128158 search engine. When a search engine user enters their query intorepparttar 128159 search engine, there are a number of quick calculations done to make sure thatrepparttar 128160 search engine presents justrepparttar 128161 right set of results to give their visitorrepparttar 128162 most relevant response to their query.

You can see which pages on your siterepparttar 128163 search engine robots have visited by looking at your server logs orrepparttar 128164 results from your log statistics program. Identifyingrepparttar 128165 robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google's "Googlebot"; others are bit more obscure, like Inktomi's "Slurp". Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers.

The Myth of "Guaranteed #1 Ranking" in Search Engine Marketing

Written by Dale Goetsch

You've seenrepparttar ads: Guaranteed #1 Ranking! There are no guarantees in search engine marketing and website promotion. If anyone tells you different, you should check quickly to make sure they don't have their hand in your wallet.

Suppose you sell widgets. You want to sell more widgets, andrepparttar 128140 way to do that is to make sure that more people know about widgets, and that you arerepparttar 128141 place to buy their widgets. You might decide to buy a half-page ad in a national magazine to tell your story. When you place that ad, you are "guaranteed" your position.

With a magazine advertisement, you know whatrepparttar 128142 magazine's circulation is, who reads it, and which page will feature your ad. The magazine can guarantee all that, because they ownrepparttar 128143 medium.

Search engine marketing is qualitatively different. When you work with a search engine marketing firm to promote your website, they cannot guarantee where your listing will appear. Certainly there are types of online ads where there are guarantees in place: banner ads priced at "cost per thousand impressions", pop-up ads, and so forth. These are like traditional media buys, where you are working directly withrepparttar 128144 owner ofrepparttar 128145 medium whererepparttar 128146 ads appear, but this is not search engine marketing.

Even so-called pay-per-click search engines cannot guarantee your position. In Google AdWords, for example, it is not justrepparttar 128147 price you pay for a given keyword that determines where you will rank. They also bring in other factors, including how often your ad is clicked-on, to determine which ad will be listed first. Just throwing money at them will not necessarily get you intorepparttar 128148 #1 spot.

The bottom line is this: search engine marketing professionals do not ownrepparttar 128149 search engines. They can tell you that you will achieve #1 ranking on a given search engine, or they can tell you thatrepparttar 128150 moon is made of green cheese, but there is no way they can make either of those happen. When you tell Time magazine you want your ad to be onrepparttar 128151 back cover, and you pay them enough money, they will guarantee yourepparttar 128152 back cover. If you tell your search engine marketing people you want to be #1 in AllTheWeb, they cannot guarantee you that result. They can recommend changes to your site that will increaserepparttar 128153 likelihood of your ranking higher, but that is a long way from a guarantee. If you don't controlrepparttar 128154 medium, you can't guaranteerepparttar 128155 result. Since your search engine consultant doesn't controlrepparttar 128156 search engine, there is no way they can guarantee your position.

The ranking algorithms ofrepparttar 128157 search engines are a closely-guarded secret. The search engine wants to give top ranking torepparttar 128158 site that isrepparttar 128159 best match to an individual visitor's search query, not torepparttar 128160 site that was able to "beat"repparttar 128161 system. That is whererepparttar 128162 value of real search engine marketing comes in.

Cont'd on page 2 ==> © 2005
Terms of Use