How to Control Search Engine Robots

Written by Michael Rock

Continued from page 1

Disallow: /duplicate/

The * after user-agent says that this action applies to all crawlers and /duplicate/ after disallow tells all crawlers to ignore this directory and not search it. For each user-agent and disallow line there must be a blank space between them in order for it to function correctly. So this is how you would createrepparttar above two commands into a robots.txt file:

# this identifiesrepparttar 138175 wayback machine

User-agent: ia_archiver

Disallow: /

User-agent: *

Disallow: /duplicate/

One thing to note that is very important: Anyone can accessrepparttar 138176 robots.txt file of a site. So if you have information that you don't want anyone to see don't include it intorepparttar 138177 robots.txt file. Ifrepparttar 138178 directory that you don't want anyone to see is not linked to from your web siterepparttar 138179 crawlers won't index it anyway.

An alternative to blocking indexing of your site is to put a meta tag into repparttar 138180 page. It looks like this:

You put this intorepparttar 138181 tag of your web page. This line tellsrepparttar 138182 robot crawlers not to index (search)repparttar 138183 page and not to follow any ofrepparttar 138184 hyperlinks onrepparttar 138185 page. So as an example tellsrepparttar 138186 robots crawlers to not indexrepparttar 138187 page, but followrepparttar 138188 hyperlinks on this page.

Did you know that Google has its own tag?

It looks like this: This tellsrepparttar 138189 Google robot crawler not to indexrepparttar 138190 page, not to follow any of repparttar 138191 links, and not to keep from storing cached versions of your web site. You will want this done if you updaterepparttar 138192 content on your site frequently. This preventsrepparttar 138193 web user from seeing outdated content that isn't refreshed because of storage inrepparttar 138194 cache.

You can userepparttar 138195 tag to specifically talk to Google's robots to avoid complications or if you are optimizing your site for Google's search engine. This concludes this month's article.

Untilrepparttar 138196 next article have a great day!

Copyright Michael Rock

(You have permission to copy this article as long as it remains intact withrepparttar 138197 author's byline)

Web development contractor (Web Design and Hosting)

Internet Presence

The owner of this registered company has over twenty years experience with DOS, windows business applications, numerous programming languages, artistic development, and web design. Other areas of interest include web marketing, web promoting, and business marketing and development. Afterrepparttar 138198 persuasion of those praising his work, he decided to go into business himself and highly suggests everyone else to dorepparttar 138199 same.

Internet Presence was founded in 2003 from a desire to become independent. Less than 1 year later Internet Presence has had accounts in three different states ranging from a locally owned auto collision repair shop to a glass packaging industry that sells its product worldwide.

Where To Look For Dirt-Cheap DVD Software. Part XVII

Written by David D. Deprice

Continued from page 1

Subtitle and audio language can be selected to user preference. The target framerate, audio volume and sampling frequency, AC3 or Dolby Surround output -- all can be adjusted withinrepparttar limitations ofrepparttar 138096 selected format. ImTOO DVD Ripper also adjustsrepparttar 138097 ripping and conversion settings according torepparttar 138098 detected CPU model.

Movies can be previewed before ripping, and also duringrepparttar 138099 ripping process.

Find out how David D. Deprice makes $58000 a week selling cheap DVD software at

    <Back to Page 1 © 2005
Terms of Use