These pages, although optimized for different search engines, often turn out to be pretty similar to each other. The search engines now have ability to detect when a site has created such similar looking pages and are penalizing or even banning such sites. In order to prevent your site from being penalized for spamming, you need to prevent search engine spiders from indexing pages which are not meant for it, i.e. you need to prevent AltaVista from indexing pages meant for Google and vice-versa. The best way to do that is to use a robots.txt file.

You should create a robots.txt file using a text editor like Windows Notepad. Don't use your word processor to create such a file.

Here is basic syntax of robots.txt file:

User-Agent: [Spider Name] Disallow: [File Name]

For instance, to tell AltaVista's spider, Scooter, not to spider file named myfile1.html residing in root directory of server, you would write

User-Agent: Scooter Disallow: /myfile1.html

To tell Google's spider, called Googlebot, not to spider files myfile2.html and myfile3.html, you would write

User-Agent: Googlebot Disallow: /myfile2.html Disallow: /myfile3.html

You can, of course, put multiple User-Agent statements in same robots.txt file. Hence, to tell AltaVista not to spider file named myfile1.html, and to tell Google not to spider files myfile2.html and myfile3.html, you would write

User-Agent: Scooter Disallow: /myfile1.html

User-Agent: Googlebot Disallow: /myfile2.html Disallow: /myfile3.html

If you want to prevent all robots from spidering file named myfile4.html, you can use * wildcard character in User-Agent line, i.e. you would write

User-Agent: * Disallow: /myfile4.html

However, you cannot use wildcard character in Disallow line.

Once you have created robots.txt file, you should upload it to root directory of your domain. Uploading it to any sub-directory won't work - robots.txt file needs to be in root directory.

I won't discuss syntax and structure of robots.txt file any further - you can get complete specifications from here.

Google boasts more than eight billion web pages. How will yours show up? When will sales or hits start rolling in?

Recently I’ve come across a few people who are interested in joining Internet and want some assistance in building their first web site. If you want to be online, start looking for information there, especially at a site such as http://www.webproworld.com as well as in your local bookstore and library. Read as much as you can and don’t allow yourself to become overwhelmed. The best web sites are built one page at a time.

Creating It Whether you’re adding one page to World Wide Web or thousands, you should first decide not only what you want to show to people but also which program will you use? Microsoft FrontPage and Macromedia’s Dreamweaver MX are two of most popular software programs, but there are many others and some of them are even free when you sign up with a web host. Make sure you’re comfortable using software and that enough information is provided such as reference books and/or articles.

To determine how you will capture your audience, what will you put on your home page? A welcome sign? Lots of text? Lots of pictures? Will it be a catalogue, scrapbook, library, photo album, mall, menu or museum? The best way to begin is most old-fashioned: use index cards and a pen. This will be your site map and this way you can physically create your site. Begin with home page and add other cards for your additional pages. Hopefully you’ve decided on your site’s name. If not, now is time.

Navigation How you get around your site is determined here, and on your home page, [also referred to as index page], as this is where you decide how you will link each page. A row of well labeled navigation buttons or text links along top or side? On bottom? Will you need any?

For example, someone needs to create a site where they will be selling several used cars. The home page will have links to each car. Don’t try cramming all that information on your main page, as you’ll lose viewers. It’s just too much information, too much scrolling down and far too many pictures and megabytes. Each car should have its own page. Suppose that bright red Ford Mustang is on your home/index page for its visual appeal, but create a link to a Mustang page where you have all relevant information such as price, mileage, other statistics and more photos of car. Same goes for a Honda Prelude, and/or all other cars/products you are selling. Each item should have its own page as that creates additional pages and more chances of your site being found by search engines which translates into more people visiting your site.

Enticing Your Online Audience What will drive people to your site and keep them there? Whatever you have to offer, whether free or involving a fee, you must make your site as legitimate looking as possible.