Search Engine Robots - How They Work, What They Do (Part I)

Written by Daria Goetsch

Continued from page 1

Along with identifying individual robots and countingrepparttar number of their visits,repparttar 128141 statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. Inrepparttar 128142 resources section ofrepparttar 128143 end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them.

How Do They Read The Pages On Your Website? Whenrepparttar 128144 search engine robot visits your page, it looks atrepparttar 128145 visible text on repparttar 128146 page,repparttar 128147 content ofrepparttar 128148 various tags in your page's source code (title tag, meta tags, etc.), andrepparttar 128149 hyperlinks on your page. Fromrepparttar 128150 words andrepparttar 128151 links thatrepparttar 128152 robot finds,repparttar 128153 search engine decides what your page is about. There are many factors used to figure out what "matters" and each search engine has its own algorithm in order to evaluate and processrepparttar 128154 information. Depending on howrepparttar 128155 robot is set up throughrepparttar 128156 search engine,repparttar 128157 information is indexed and then delivered torepparttar 128158 search engine's database.

The information delivered torepparttar 128159 databases then becomes part ofrepparttar 128160 search engine and directory ranking process. Whenrepparttar 128161 search engine visitor submits their query,repparttar 128162 search engine digs through its database to giverepparttar 128163 final listing that is displayed onrepparttar 128164 results page.

The search engine databases update at varying times. Once you are inrepparttar 128165 search engine databases,repparttar 128166 robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they haverepparttar 128167 latest info. The number of times you are visited depends on howrepparttar 128168 search engine sets up its visits, which can vary per search engine.

Sometimes visiting robots are unable to accessrepparttar 128169 website they are visiting. If your site is down, or you are experiencing huge amounts of traffic,repparttar 128170 robot may not be able to access your site. When this happens,repparttar 128171 website may not be re-indexed, depending onrepparttar 128172 frequency ofrepparttar 128173 robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then.


SpiderSpotting - Search Engine Watch

List of robots and protocols for setting up a robots.txt file.


Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing.

Articles and resources about tracking Search Engine spiders.

Sim Spider Search Engine Robot Simulator

Search Engine World has a spider that simulates whatrepparttar 128174 Search Engine robots read from your website.

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing (, a Search Engine Promotion company serving small businesses. She has specialized in search engine optimization since 1998, including three years as the Search Engine Specialist for O'Reilly & Associates, a technical book publishing company.

The Myth of "Guaranteed #1 Ranking" in Search Engine Marketing

Written by Dale Goetsch

Continued from page 1

Whilerepparttar search engine marketing person cannot guarantee you a position, what they can do is to apply years of experience to tell you what has worked inrepparttar 128140 past, and to help you make it work today. In many ways, "organic" search engine optimization is really a matter of editing web pages or whole sites to make them repparttar 128141 most search engine friendly they can be. Making sure that a given page has justrepparttar 128142 right combination of keywords, title, links, and so on, is really at its base simply a matter of making that pagerepparttar 128143 best web page it can possibly be. The page that will rankrepparttar 128144 best inrepparttar 128145 search engines is alsorepparttar 128146 page that will makerepparttar 128147 most sense torepparttar 128148 human visitor. Rather than relying on tricks to try to makerepparttar 128149 page rank high, it is a matter of just makingrepparttar 128150 pagerepparttar 128151 most focused and on-message that it can be. The bad news is that this doesn't guarantee which position inrepparttar 128152 search engine rankings that page will occupy on a given day. The good news is thatrepparttar 128153 page will always rank well.

The search engines change their algorithms from time to time. If today's rule, for example, is that justrepparttar 128154 right combination of text inrepparttar 128155 title tag will make or break a site, and you know this is true, then all you have to do is to tweakrepparttar 128156 title tag to fit within that rule, and you will automatically rank very high. Today.

Suppose that tomorrow, however, that rule is changed. Suppose that nowrepparttar 128157 most important factor thatrepparttar 128158 search engines use to rank a site isrepparttar 128159 content of repparttar 128160 META Description tag. Allrepparttar 128161 work you went to yesterday to fixrepparttar 128162 title is now useless. All of your attention is now focused on fixing that description tag.

Clearly, over timerepparttar 128163 focus ofrepparttar 128164 search engines will vary. The best way to deal with this is to not deal with it! This means that rather than tweaking a site one way today and another way tomorrow,repparttar 128165 best way to approach optimizing a page or a whole site is to not try to beatrepparttar 128166 system. Instead of trying to "psych-out"repparttar 128167 search engines, why not add value torepparttar 128168 site? A "common sense" approach to search engine optimization, looking for long term results, isrepparttar 128169 way to go. When you try to help a site rank better by making itrepparttar 128170 best it can be, everybody wins.

Dale Goetsch is the Technical Consultant for Search Innovation Marketing (, a Search Engine Promotion company serving small businesses and non-profits. He has over twelve years experience in software development. Along with programming in Perl, JavaScript, ASP and VB, he is a technical writer and editor, with an emphasis on making technical subjects accessible to non-technical readers.

    <Back to Page 1 © 2005
Terms of Use