Back in 2003, a reader named Brad asked me for some advice. He'd conducted a test campaign that resulted in a huge discrepancy he couldn't identify. No doubt losing 90 percent of his sales, when his normal conversion rate is over 4.6%, distressed him. He wanted to know what caused his conversion rate to come in at just 0.47%. Together, we discovered the variable he'd ignored, and in his most recent test the conversion rate is back to normal.

Brad was so excited he insisted I share this case study with all of you. I hesitated to do so but promised him I'd share the data with you and let you reach your own conclusions. My concern is what is valid for Brad's business isn't valid for everyone's. Brad's product has a target market with very particular characteristics. Plus, his marketing is response-driven, not branding-driven. The entire buying process for his products is, though not entirely unique, not universally applicable, either.

Case studies can be very informative, but they can also do a lot of damage. Be very careful what you copy. Many companies try to emulate more successful competitors without really knowing what they're copying, because they're unaware of all the variables. What you need to do is uncover the key factors that influence your target audience to buy your particular products, then utilize this information to help you refine your sales process.

At Future Now, we constantly get requests for averages and benchmarks on conversion rates. We always advise against comparing your own conversion rates with "norms" or "averages." These are drawn from sites that don't have the same traffic or product as yours, and they may differ in any number of ways. In our work alone, we've identified over 1,100 variables that affect conversion. This is why I cringe when I hear the phrase "best practices" tossed around as if it were gospel.

To illustrate my point, I called a friend I knew could further confuse the issue. I told Sam Decker, then senior manager of Dell Consumer eBusiness (currently VP of Marketing & Products at Bazaarvoice), what Brad did. Dell remains one of the top e-commerce sites and is known for its innovative approach to measuring, testing, and optimizing. At Dell, Sam was responsible for sales on the consumer website. So, if anybody would know what to learn from Brad's case study, it would be Sam, right? Interestingly enough, Dell's own case studies proved that making Brad's "mistake" would actually help Dell's online sales.

How can it be that two case studies contradict each other so blatantly? The answer is no business is linear. There are many facets, or topological elements, to consider in designing an effective online strategy to maximize your conversion rate. Your conversion rate is only a reflection of the marketing and sales effectiveness and your customers' satisfaction. If you're looking for one canned, simple solution, you're bound to be either bankrupt or very disappointed.

I'm going to make this simpler on you than it was on me. Here are two screen shots: Test A and Test B. Now, see if you can figure out what the offending variable is–and, yes, there is more than one change on the page.

Test A

Test B

Can you find them all? Youll get to see the answers in a moment. But in the meantime, Id like to share some recent thoughts on A/B Testing from Future Nows CTO, John Quarto-vonTivadar:

Lately, everywhere you go analytics industry folks are talking about AB Testing. Thats a good sign, since it means the industry is focusing on an overlooked leverage point in their web analytics investment.

But as so often happens, achieving full buzzword compliance has become the goal rather than the means; what lies behind the words is often lost. In this case, "AB testing" the buzzword has become a euphemism for plain old "testing," which, like ordering liver on a first date, may be good for you, but is certainly not sexy. But throw some "AB" in front of "testing" and your dour liver is magically transformed into pat de foie gras.

This is a bit disturbing, especially when you hear people sprinkling the "AB" condiment to add flavor to anything from a focus group ("Hey, did you AB Test the response to the new company logo?") to the mundane ("Suzies lamp is out, can you AB Test the light bulb?") to the painfully comical ("Honey, lets AB test the Lord of the Rings Directors Cut with the Wide-Screen edition!").

Mixed in there, perhaps lost among the cacophony of buzzword hype, are the ingredients to some real AB testing and with it a future vision of how to achieve its true objective.

Lets say we want to determine whether Nolan Ryan is a better baseball player than Homer Simpson? How should we proceed? First, we might set a metric for what we mean by a "better" baseball player. We can measure evidence in concrete ways, noting the two subjects different batting averages or RBIs or the like. What were searching for is the right metric–a formula that would lead us to a correct decision. Such a formula is more precisely termed a "fitness function."

In virtually all such measures Nolan is the better candidate. If you were choosing a player for your team, youd certainly pick Nolan; you can be confident youve made the correct decision.

But lets think on that a moment: the reason you feel confidence in signing Nolan stems from your familiarity with the metric and fitness function that are implicitly applied when we speak of baseball. Your decision might be quite different if we want to pick an effective donut quality assurance taster. Suddenly, Homer Simpson is back in the running.

Even then, your confidence may be based on your understanding that "tastes better" is the donut metric and that Homer Simpson is an acknowledged expert in donut consumption. But what is the fitness function? That is, what does it mean to "taste better"? Are you relying solely on Homers reputation as an expert? But his expertise is based on consumption quantity so perhaps you suspect he enjoys all donuts equally and actually has little, uh, "taste" at all. In other words, its quite possible you dont have any knowledge at all of what we might call the "donut tastiness" fitness function.

Interestingly, marketers and business owners are asked every day to make more important decisions with less information with an undetermined fitness function.

More formally, AB Testing first requires a metric be identified (that is, "what will be contrasted?"). Second, a fitness function describing that metric is agreed upon ("how will we measure and contrast the differences?"). And third, an optimization step where the system is tweaked based on comparison of exactly two tested solutions, which differ in only one respect of how they meet the fitness function.

A typical problem for AB testing might be: "Do more people buy when I use a RED buy-it-now button or when I use a BLUE buy-it-now button?" The data for this test will come from your web analytics. Once you know which candidate converts better, you use the winner and discard the other. You might then test again, comparing the winner to some other variant you have in mind.

By examining incremental improvements to conversion at the page level you should be able to measurably move higher on your sites fitness profile (here, "converts more") versus what might be expected by random chance at least as compared to your pre-existing analytics and your non-optimized competitors. Thats just another way of saying you can use AB Testing to pick the low-hanging conversion fruit.4

Be careful, though, with low-hanging fruit; typically you dont know if youve picked a juicy apple or sour lemon until after its already in your basket.

Alright, class Pencils down!

When I posed this test to my readers, 85% did not correctly identify the offending variable in the case study. Now that youve played "Wheres Waldo," heres a list of all possible variables:

1. Closed space between top "Proceed to Checkout" button line and next line.
2. Removed top "Continue Shopping" button.
3. Removed "Update" button underneath the quantity box.
4. Moved "Total" box down a line. Text and amount appear in different boxes.
5. Above the "Total" box is a "Discount" box, with amount in a box next to it.
6. Above "Shipping Method" line is "Enter Coupon Code" with a box to enter it.
7. New "Recalculate" button left of "Continue Shopping."
8. Bottom tool bar now on two lines.
9. Shopping cart icon one space closer to the words "Shopping Cart."

Which of the above was the culprit?

Remember, the goal was to bring the conversion rate back to where it was before the "improvements" started. When nine variables change at once, how can you determine which were effective? Did only one variable cause the change, or combination? Did some of changes improve the conversion rate, only to be overwhelmed by others that made a negative impact? This is hardly a systematic method to measure, test and optimize.

The bigger question is, "How do you get the conversion rate back over 4.6 percent–fast?" Should time be spent testing each of the nine variables? Sure, that would work; if we had nine weeks.

This is when customer psychology insight and experience pay off. The following test increased the conversion rate back to 4.9%. The offending variable was Number 6, the "Enter Coupon Code" field on Test A.

One of the respondents, Jill Whalen, who got it right, had this to say: "When I've seen these on sites. I wonder how I can get that discount too. I don't like having to pay full price knowing there's some way for me to get a discount I don't know about."

Couldn't have said it better myself.

Don't jump to the conclusion that couponing on a site is a bad thing. It depends what sort of site you have and what business you're in. On this site, this one small change accounted for a 1,000% increase in volume.

There are thousands of such variables on your site. Some have a major impact; some are only incremental. Ones that works for your widget business may not work for your brother's whatzit business. This applies whether your site targets consumers (B2C) or businesses (B2B).

Identify these variables. Systematically plan, benchmark, test and optimize. Have a system in place to categorize and account for each test and its result. Sometimes, a change will move your conversion rate only a bit. That doesn't mean your choice of variable had no impact. It could mean your choice of solution had no impact. So, by all means, test! (But be careful.)

###