Expecting an unsupervised beta testing group to do anything meaningful - Beta testers need well defined goals, constant supervision and strong leadership to be successful. Without these things beta testing is simply a numbers game which does nothing useful at all.

Testing is not appropriate time to make design decisions - Something that I commonly see from Microsoft and other large companies is they create a product and send it out to their beta testers for feedback. Guys, come on. Beta testing is not place for this. Design decisions need to be made well before a product is sent out for testing. You want to find out if your users will like a feature? Create a prototype, send it to a statistically valid sample audience and ask them for their opinions. Clearly define it to audience as a prototype and survey them for their opinions afterwards. Don't send out a poorly defined "beta test" to a hundred thousand people and try and get their opinions on features. The only thing you are going to accomplish is to get yourself slammed in media. You also find yourself making design changes in a product at wrong stage of product life cycle. Design changes need to be made during analysis and design phases of project, not AFTER implementation.

So how is good testing done?

Analysis and design must be done first - No matter how large project, you will be much more likely to succeed if you do these two steps thoroughly and completely before implementation and testing. Many years ago I had a boss, name of Gary, who didn't understand this simple rule. He asked me to start implementing a warehousing system for a client without writing a specification (over my objections). His concept of design was to spend an hour or two asking customer what was needed, then to start coding, then to show customer, make changes, code some more, show customer, make changes and so on until customer said "it's fine". Needless to say, project took far longer than necessary and did not do a great job of meeting customers needs. It was also very buggy and required an immense amount of support during first couple of years of it's life cycle.

The only phase where marketing department should be involved is analysis - A well trained analyst understands that marketing department is a customer and must be included in analysis phase of project. This is only time (until product is through testing) that marketing should have any input. If you don't follow this rule you will wind up with a product which changes direction during testing, and thus invalidates your test.

Understand that a specification is a contract - The goal is to implement something that meets specification. This is only way that I know of to produce a software project that (a) gets finished at all and (b) meets customers expectations. Of course, this assumes that your analysis and design is top notch. What does this really mean? It simply means that changes to design are only allowed during analysis and design phase. Period. If your customer changes anything at all after analysis and design, you must reanalyze, redesign and renegotiate - always and without fail.

Let's say you are contractor who has been hired to create a new warehouse system. You do your analysis and design and it is approved by customer. You now have a contract and it is important that your customer understands this from beginning. Okay, you begin project and your customer decides he wants to add bar coding. This seems pretty simple so you say "sure". Wrong thing to do. You should say either "let's finish project as designed then add things" or "okay, we will need to stop, see how that effects project (at customers cost), then we will submit a cost and new delivery date".

Maintain standards - Testing measures implementation against specifications and standards. Standards should be made known to customer as part of entire package. These might include things like all fields will be validated in specific ways, all buffers will not overflow, screens will have a certain look and so on.

Remember purpose of testing - Testing should prove implementation meets everything included in specifications and standards. Testing does NOT mean product is measured against customer expectations (that's a marketing function which should have been nailed down during analysis and design phase). You see, specification MUST meet customer expectations before implementation beings. Then final product WILL meet customer expectations as specification is expectation.

Before I turned my attention to web and small business computing a few years ago, I was an automotive journalist and had pleasure of attending new car introductions put on at glitzy resorts exclusively for automotive press. Those of us that reported on new car introductions were wined, dined and entertained in first-class style by auto manufacturers and their PR firms before being given keys to gleaming new models as-yet-unseen by world for first drive impressions and photo ops in stunning locales.

It's a very different world when it comes to computers as they lined up several hundred journalists outside doors of Moscone convention center for an hour-long wait and admittance to hard chairs packed shoulder to shoulder as loudspeakers urged everyone to "please move to center to be sure all seats are taken so everyone can have a seat." I'd love for Porsche to try that approach and hope for rave reviews from automotive press.

Some technology columnists routinely gripe about lack of objectivity shown by adoring fans of any new Mac product. What is stunning is not adoration at introductory shows, but fact that sales figures of 6 million iMacs over last three years fails to impress. Inevitably comparisons are made to Microsoft and Windows operating system that powers 95% of PC market. One grumbling post at a tech site message board said, "You won't see iMacs dominating enterprise!" Darn, Dilbert! You mean post office won't be ordering three million units?

Thank goodness!

As long as Apple needs to produce stunning designs and fun software that makes you smile while you work to attract 5% of PC market, we won't see mediocre beige boxes sitting in every cubicle in corporate America. Darn, I guess Macs will have to remain in movie production, music mixing, print and online publishing, design and photo studios, education (the State of Maine just ordered 36,000 iBooks for public schools) and biotech firms (Genentech ordered 1000 of these new iMacs.)

Remember Apple byline is "Think Different." If we all wanted Toyotas, there would be no Ferrari's. If everyone ate at MacDonald's we wouldn't need Chez Panisse. I may drive a VW Beetle and eat at home most nights, but Damn, I'm gonna have an iMac on my desktop!