Sins Of The Internet: Email Spiders

Written by Richard Lowe

Warning: this article is not forrepparttar squeamish. It contains graphic descriptions of one ofrepparttar 109665 biggest evils onrepparttar 109666 internet. If you can face down this evil you can reduce your load of spam by several times. Hold onto your seats and try and keep down your lunch - you are about to learn one ofrepparttar 109667 secrets of how ruthless, unethical and, well, downright evil spammers steal your email address - and what you can do about it.

If you have access to your web site's log files, you will quickly find an interesting phenomenon. Your site is being visited a lot more often than you think it is. In fact, if you look closely you may be shocked to find that your HTML files are actually being used to harm you and others. In fact, you may be seeingrepparttar 109668 footprints left by some ofrepparttar 109669 tools used by unscrupulous spammers to steal your email addresses.

Oh wait, let me back up a bit and explain a few things. Each time you visit a web site a record is kept of every page, graphic, sound file, video or anything else that you access (look at or download). This record is called a log file. Each line withinrepparttar 109670 log file is one "hit" (other things are recorded also, but that is not important to this discussion). A "hit" is getting one "thing" from a web site. A "thing" can be an image, an HTML page, a video, a sound file or anything else. In fact, generally when you look at one HTML page you are actually "hitting"repparttar 109671 web site many times, once for each file onrepparttar 109672 page.

Each of these lines withinrepparttar 109673 log file records a number of pieces of information so that webmasters can later see what happened (don't worry, they are not generally interested in individuals - they want to know things like how many people are using Internet Explorer verses Netscape). One critical piece of information is calledrepparttar 109674 "user agent". Generally this containsrepparttar 109675 browser name (Internet Explorer for example) or spider name (googlebot, for example, isrepparttar 109676 spider forrepparttar 109677 Google search engine).

Examine these user agent fields and you will find out many interesting facts. You will see that your site is being visited a lot more often than you would think by lots of things with strange names:

- Googlebot - Slurp (used hundreds of search engines including Hotbot) - Scooter (Altavista robot) - Lycos Spider (used byrepparttar 109678 Lycos search engine) - and many others as well.

Most of these are innocent 'bots, used byrepparttar 109679 major search engines to keep their indexes up to date. These robots are very important, for they keep your pages listed so you will get traffic. Occasionally they have other uses, including checking your pages for changes, saving your pages for offline browsing and various statistical functions.

You will also find some other names buried in your log files. These go by names such as EmailSiphon and Cherry Picker. These robots are malignant and are used by spammers to harvest email addresses. What they do is scan every single page in your web site, as fast as they can, looking for email addresses. Specifically, they are usually looking for "mailto:" type links.

Many websites have these kind of links. They are convenient, simple and create a great way for visitors to send an email to someone. In fact, it's hard to find a website which does not have email addresses embedded somewhere withinrepparttar 109680 site.

In addition, people often leave their email addresses in guestbooks, message boards and other online communities which translate to web pages. Spam harvesters love these types of pages, as they can get dozens, hundreds or even thousands of different, valid and usable email addresses quickly and easily.

How do email harvesters work? Well, some scum spammer will install one of these programs on his system. He will tell it to begin scanning, which it will do rapidly and efficiently. In fact, these generally scan a web site so quickly thatrepparttar 109681 server cannot do anything inrepparttar 109682 meantime (most "good" spiders, onrepparttar 109683 other hand, limit their visits to one per second, minute or even hour in order to allow other people and spiders to userepparttar 109684 site while it is being scanned).

There Ain't No Such Thing As A Free Lunch: Listbot

Written by Richard Lowe

This column is about TANSTAAFL, which is a term from a book by Robert A. Heinlein (one ofrepparttar best Science Fiction authors that ever lived) called "The Moon Is A Harsh Mistress". The term means "There Ain't No Such Thing As A Free Lunch". This concept isrepparttar 109664 basis ofrepparttar 109665 plot ofrepparttar 109666 book, which is about a Lunar penal colony and it's attempt to free itself from Earth domination.


Some years ago I decided it was time to create an email newsletter. I had been doing quite a bit of research on internet promotion, and it was quite obvious that a newsletter is an essential part of any advertising campaign. I didn't know much about elists atrepparttar 109667 time, so I more or less usedrepparttar 109668 first service that I found. That service was Listbot.

Listbot was cool. It was easy (almost trivial) to create a list (or as many lists as I wanted). In fact, I soon had finished creating my list and went on to attempt to get people to join (and that'srepparttar 109669 hard part ofrepparttar 109670 job of any listmaster). Listbot seemed to berepparttar 109671 perfect service, andrepparttar 109672 price of a small advertisement atrepparttar 109673 end of each newsletter seemed very reasonable.

I kept my mailing list on Listbot for about a year until I more or less outgrew their service. Oh, it would handle my list easily (well under 500 subscribers), but I was findingrepparttar 109674 user interface awkward and difficult to use. So I went off to find a different service, and after some searching and experimenting, settled on Bravenet's mailing list.

Movingrepparttar 109675 list was a pain but fortunately it was fairly small so I managed to do it in a few hours. Forrepparttar 109676 next few months, I was happy on Bravenet's service, until I ran intorepparttar 109677 500 email address limit. At that time, Bravenet did not provide a solution for larger lists.

Thus, I had to move my list again, and this time I moved it to my own list server. Now I have none ofrepparttar 109678 limits forced upon my by other list services. With any luck, I will never have to moverepparttar 109679 list again.

I never did delete my Listbot account, and thus received occasional mailings from them. A few weeks ago, I received an email which I found interesting. Here isrepparttar 109680 first paragraph of that email.

"Dear ListBot User:We are sending this letter to inform you that effective Aug. 20, 2001, we are discontinuingrepparttar 109681 central ListBot service. ListBot has become a very popular free service, and we've been pleased to offer this valuable tool to both consumers and business users. As a provider of services for small and medium-sized businesses, our main goal isrepparttar 109682 continual improvement of services to meetrepparttar 109683 growing needs of business users. To achieve this goal, we have launchedrepparttar 109684 new List Builder service, an enhanced e-mail marketing tool that will takerepparttar 109685 place of ListBot."

Is there a foul smell inrepparttar 109686 air?

What'srepparttar 109687 problem with this? Well, I've been seeing this more and more lately and it really ticks me off. This is when an excellent service is picked up by a large company and ruined, destroyed or simply closed down. The company cannot makerepparttar 109688 service fit their business model so they simply discard it without a single care that human beings arerepparttar 109689 recipients of their services and products.

Almost as bad is whenrepparttar 109690 service is modified to fitrepparttar 109691 larger company's standards andrepparttar 109692 previously excellent tool or service is made more-or-less worthless. The best example of this behavior isrepparttar 109693 destruction ofrepparttar 109694 Webring system by Yahoo! in September of 2000.

These companies have totally forgotten (or don't care) that people, real flesh-and-blood human beings with feelings, use and depend upon their services.

Listbot is an old service which has always been free (advertising supported). A few years ago Microsoft purchasedrepparttar 109695 service, presumably to be able to offer this kind of service to their customers. They added Listbot torepparttar 109696 offerings on their Bcentral service and promoted it all overrepparttar 109697 internet.

Cont'd on page 2 ==> © 2005
Terms of Use