Playing in Googlebot's Sandbox with Slurp, Teoma & MSNbot

Written by Mike Banks Valentine

There has been endless webmaster speculation and worry aboutrepparttar so-called "Google Sandbox" -repparttar 147807 indexing time delay for new domain names - rumored to last for at least 45 days fromrepparttar 147808 date of first "discovery" by Googlebot. This recognized listing delay came to be calledrepparttar 147809 "Google Sandbox effect."

Ruminations onrepparttar 147810 algorithmic elements of this sandbox time delay have ranged widely sincerepparttar 147811 indexing delay was first noticed in spring of 2004. Some believe it to be an issue of one single element of good search engine optimization such as linking campaigns. Link building has beenrepparttar 147812 focus of most discussion, but others have focused onrepparttar 147813 possibility of size of a new site or internal linking structure or just specific time delays as most relevant algorithmic elements.

Rather than contribute to this speculation and further muddyrepparttar 147814 Sandbox, we'll be looking at a case study of a site on a new domain name, established May 11, 2005 andrepparttar 147815 specific site structure, submissions activity, external and internal linking. We'll see how this plays out in search engine spider activity vs. indexing dates atrepparttar 147816 top four search engines.

Ready? We'll give dates and crawler action in daily lists and see how this all plays out on this single new site over time.

* May 11, 2005 Basic text on large site posted on newly purchased domain name and going live by days end. Search friendly structure implemented with text linking making full discovery of all content possible by robots. Home page updated with 10 new text content pages added daily. Submitted site at Google's "Add URL" submission page.

* May 12 - 14 - No visits by Slurp, MSNbot, Teoma or Google. (Slurp is Yahoo's spider and Teoma is from Ask Jeeves) Posted link on WebSite101 to new domain at

* May 15 - Googlebot arrives and eagerly crawls 245 pages on new domain after looking for, but not findingrepparttar 147817 robots.txt file. Oooops! Gotta add that robots.txt file!

* May 16 - Googlebot returns for 5 more pages and stops. Slurp greedily gobbles 1480 pages and 1892 bad links! Those bad links were caused by our email masking meant to keep out bad bots. How ironic slurp likes these. * May 17 - Slurp finds 1409 more masking links & only 209 new content pages. MSNbot visits forrepparttar 147818 first time and asks for robots.txt 75 times duringrepparttar 147819 day, but leaves when it finds that file missing! Finally get around to add robots.txt by days end & stop slurp crawling email masking links and let MSNbot know it's safe to come in!

* May 23 - Teoma spider shows up forrepparttar 147820 first time and crawls 93 pages. Site gets slammed by BecomeBot, a spider that hits a page every 5 to 7 seconds and strains our resources with 2409 rapid fire requests for pages. Added BecomeBot to robots.txt exclusion list to keep 'em out.

* May 24 - MSNbot has stopped showing up for a week since findingrepparttar 147821 robots.txt file missing. Slurp is showing up every few hours looking at robots.txt and leaving again without crawling anything now that it is excluded from repparttar 147822 email masking links. BecomeBot appears to be honoring repparttar 147823 robots.txt exclusion but asks for that file 109 times duringrepparttar 147824 day. Teoma crawls 139 more pages.

* May 25 - We realize that we need to re-allocate server resources and database design and this requires changes to URL's, which means all previously crawled pages are now bad links! Implement subdomains and wonder what now? Slurp shows up and finds thousands of new email masking links asrepparttar 147825 robots.txt was not moved to new directory structures. Spiders are getting errors pages upon new visits. Scampering to put out fires after wide-ranging changes to site, we miss this for a week. Spider action is spotty for 10 days until we fix robots.txt

The Fundamentals of Inbound Links

Written by Justine Curtis, Enable UK

Link development is one ofrepparttar most overlooked components of a successful search engine optimisation campaign. Web site owners can put allrepparttar 147533 key phrases they wish on a page, develop killer content and a search-engine friendly navigation scheme, but without a well-planned link-development campaign, crawler based search engine visibility is difficult to achieve and high listings are virtually impossible. This month we will be coveringrepparttar 147534 importance of inbound links, how to get them and what they mean for your web site.

We have all heard that adding quality content to your web site will giverepparttar 147535 search engines a good idea of how to index your web site, it’s a topic we covered in “Content, Content, Content” back in November 2004. Butrepparttar 147536 secret to luringrepparttar 147537 search engines to your web site, and in part to improving your position within those listings is your inbound links.

So what are inbound links? Inbound links are simplyrepparttar 147538 links from an external web site to your web site. You will also hearrepparttar 147539 phrase “link popularity” which is simply a search engines measure ofrepparttar 147540 number and quality of links pointing to a single URL. We will cover “quality” later in this article but a site's home page usually hasrepparttar 147541 highest number of inbound links, or highest link popularity, because site owners tend to link to home pages rather than any other content page.

What makes a quality link? Achieving good rankings using links used to be simple; today things have become more complex, with search engines using very complicated algorithms to measurerepparttar 147542 quality ofrepparttar 147543 links that involves: ·Number of links to your site? ·What is written in those links? ·What key phrases are used in those links? ·What isrepparttar 147544 quality ofrepparttar 147545 site that is linking to yours? ·How many other links does that site have? A link from a web site thatrepparttar 147546 search engine recognises as a leader in your industry carries a lot of clout and means that your web site is important. Two links from industry leaders means your site is even more important. But 100 links from random web sites, from industries you are not even related with, means almost nothing. Thus, getting links is onlyrepparttar 147547 start;repparttar 147548 important thing is getting good links from quality (high ranking and relevant) web sites.

How Do We Get Links? It all starts with content – yes we’re back to that again. No one will link to you unless you offer quality information about a particular subject. If you are an estate agent, why not offer information aboutrepparttar 147549 area you are selling in. People will be interested in local schools, clubs and associations, crime rates, etc. Once you have completed your information; find web sites whose visitors would benefit from your site's content; solicit links from nearby leisure facilities, local towns, builders, gardeners, removal companies, etc.

How Do You Solicit Links? To solicit links takes a lot of work. First you need to find web sites that are appropriate to your audience and also bear in mind that often, web site owners will requirerepparttar 147550 link to be reciprocal; i.e.; they want one back from you. As a general rule, if you think your target market may visit a web site, you would benefit from a link from it. Type your search phrases into a search engine like and see what sites are listed inrepparttar 147551 first page of results and if appropriate, ask them for a link. Also check which sites are linking to these well-listed web sites and ask those sites for a link. For a list of inbound links to a third parties web site type in "" in Google.

Send out personalised emails torepparttar 147552 web sites you have identified giving some evidence that you have actually visitedrepparttar 147553 web site. Don't waste time on approaching web sites that haven't been updated in years, many web sites are perfect but if they are rarely updated, your link wont be added. Be friendly and polite in your approach, and point outrepparttar 147554 benefits of linking to your web site. If you are lucky, around one third of your emails will get a response. It can be frustrating and discouraging when you get little response or a whole string of no’s, but it’s important to keep going.

It is also important to follow up. Until you get a definite no, keep in contact and keep it personal. The trick is to letrepparttar 147555 recipient of your emails think that they arerepparttar 147556 only web site that you are approaching for a link so make sure you keep track of who you have contacted and what you have written or said. If your linking prospects think that you are sending out mass emails, or that you are usingrepparttar 147557 exact same approach with other sites, you will probably lose their respect - and any chance of a link.

Directories and Links Do not forget to submit torepparttar 147558 directories, relevant business directories and local listings. Getting listed in some ofrepparttar 147559 larger directories such asrepparttar 147560 Open Directory Project counts for a lot inrepparttar 147561 major crawler based search engines. Take your time and chooserepparttar 147562 right category to submit to. Also, make sure you readrepparttar 147563 advice about how to writerepparttar 147564 descriptions and titles and write them inrepparttar 147565 way they ask, bear in mind that human beings edit these directories! Also consider submitting to Yahoo and, both charge annual subscription fees but both would provide high quality links. Ifrepparttar 147566 links turn out not to be cost effective, simply don’t renew your subscription.

Cont'd on page 2 ==> © 2005
Terms of Use