How To Control Search Engine Robots

Written by Michael Rock

Continued from page 1

ia_archiver isrepparttar crawler name forrepparttar 138222 wayback machine that you may have heard of, andrepparttar 138223 / after disallow tells ai_archiver not to index any of your site. The # allows you to write comments to yourself so you can keep track of what you typed.

Typerepparttar 138224 above three lines into notepad from your computer and save it torepparttar 138225 root directory of your web site as robots.txt. Web crawlers look for this document first at a web site before doing anything else. This helpsrepparttar 138226 crawler to do its job, and helpsrepparttar 138227 web site owner tellrepparttar 138228 spider what to do. Say for instance you have some data that you don't wantrepparttar 138229 crawlers to see. (Like duplicate content for other browser referrer pages)

You can deter crawlers from indexingrepparttar 138230 'duplicate' directory by typing this into your robots.txt file. Or if you would like to haverepparttar 138231 robots.txt file created for you, visit To validate your robots.txt file to make sure it works properly you can visit

User-agent: * Disallow: /duplicate/

The * after user-agent says that this action applies to all crawlers and /duplicate/ after disallow tells all crawlers to ignore this directory and not search it. For each user-agent and disallow line there must be a blank space between them in order for it to function correctly. So this is how you would createrepparttar 138232 above two commands into a robots.txt file:

# this identifiesrepparttar 138233 wayback machine User-agent: ia_archiver Disallow: /

User-agent: * Disallow: /duplicate/

One thing to note that is very important: Anyone can accessrepparttar 138234 robots.txt file of a site. So if you have information that you don't want anyone to see don't include it intorepparttar 138235 robots.txt file. Ifrepparttar 138236 directory that you don't want anyone to see is not linked to from your web siterepparttar 138237 crawlers won't index it anyway.

An alternative to blocking indexing of your site is to put a meta tag intorepparttar 138238 page. It looks like this:

You put this intorepparttar 138239 tag of your web page. This line tellsrepparttar 138240 robot crawlers not to index (search)repparttar 138241 page and not to follow any ofrepparttar 138242 hyperlinks onrepparttar 138243 page. So as an example tellsrepparttar 138244 robots crawlers to not indexrepparttar 138245 page, but followrepparttar 138246 hyperlinks on this page.

Did you know that Google has its own tag?

It looks like this: This tellsrepparttar 138247 Google robot crawler not to indexrepparttar 138248 page, not to follow any ofrepparttar 138249 links, and not to keep from storing cached versions of your web site. You will want this done if you updaterepparttar 138250 content on your site frequently. This preventsrepparttar 138251 web user from seeing outdated content that isn't refreshed because of storage inrepparttar 138252 cache. You can userepparttar 138253 tag to specifically talk to Google's robots to avoid complications or if you are optimizing your site for Google's search engine. This concludes this month's article.

Untilrepparttar 138254 next article have a great day!

Copyright Michael Rock Web development contractor (Web Design and Hosting) Internet Presence

The owner of this registered company has over twenty years experience with DOS, windows business applications, numerous programming languages, artistic development, and web design. Other areas of interest include web marketing, web promoting, and business marketing and development. After the persuasion of those praising his work, he decided to go into business himself and highly suggests everyone else to do the same.

Choosing The Right Keywords For Your Pay Per Click Advertising Campaign

Written by Stan Hauser

Continued from page 1

Fly Fishing Gear

Fishing Lures

Fishing Reels

Many ofrepparttar Pay Per Click Search Engines offer tools to help advertisers identify hard to find keywords related to their sites. Overture's isrepparttar 138145 most advanced in this area, but Google's works well too. The leading third party tool to help advertisers uncover profitable keywords is offered by a company called WordTracker. WordTracker not only helps identify good keywords for your Pay Per Click Advertising program, but it also tells you how many timesrepparttar 138146 words were searched on across allrepparttar 138147 major Search Engines. This is important, becauserepparttar 138148 more nicherepparttar 138149 keyword,repparttar 138150 less traffic it will generate. It doesn't do any good to find a perfectly targeted keyword if it only generates a few clicks a month.

Because each targeted, niche, keyword by definition will only produce a small amount of traffic, it's important to identify as many as possible so their combined traffic will have a meaningful impact on your Pay Per Click Advertising program.


The beauty of Pay Per Click Advertising isrepparttar 138151 more targetedrepparttar 138152 keyword,repparttar 138153 less you'll likely have to pay for each click. This is opposite of how every other form of advertising works. If you use direct mail, magazines, TV, or even other forms of online advertising to market your products,repparttar 138154 prices rise asrepparttar 138155 audience becomes more targeted.

To succeed you must take advantage of this unique dynamic of Pay Per Click Advertising by using every available tool to help you think "out ofrepparttar 138156 box" to create a comprehensive, targeted list of keywords to drive a steady volume of profitable traffic to your web site.

Stan Hauser is a leading expert on Pay-Per-Click Advertising strategies and the creator of

    <Back to Page 1 © 2005
Terms of Use