A/B and A/B/A Ads and Creatives Testing

by Brett Crosby

As you probably know by now, it's extremely important to track each of your ads and keywords to see how well it performs. But how do you design a highly effective ad in the first place? One of the most interesting approaches to ad optimization is the A/B split test and its sibling, the A/B/A test. Both of these involve testing different creative, such as multiple versions of an email newsletter or multiple versions of an ad, against each other.

So, what is the difference between A/B and A/B/A testing and when should you use each approach? How do you actually go about setting up a test? Let's start with an example of A/B testing in AdWords.

A/B Testing in AdWords

There are two ways to run an A/B test in AdWords. The first approach, and the approach we currently recommend if you rely on AdWords autotagging, is to create two different versions of your ad, making sure that each version has a different headline. Autotagging will then automatically allow you to compare conversion metrics for each version of the ad in the Ad Versions report (in the Traffic Sources section). For each keyword, you can drill down and compare the performance of each version of the ad, because each ad will be differentiated by a unique headline.

The second approach gives you a little more flexibility because you'll be able to have the same headline for both versions of your ad if you wish. However, you should not use this approach if you have been using autotagging.(Note: In order for to run this test, you will need to disable autotagging in your AdWords user preferences. To ensure equal delivery between destination page URLs, be sure to set up all of your test ads in a single Ad Group. Also, make sure that you turn off "Automatically optimize ad serving for my ads" from the AdWords Campaign Settings. This allows AdWords to deliver your test ads equally.)

To conduct the A/B test, set up multiple versions of ads and set your AdWords account to randomize their display. Let's consider the following case in which the marketer has set up two ads:

Text Ad #1 Designer Cookware Recommended by Seattle's top chefs. www.abccustomcookware.com Destination URL: http://www.abccustomcookware.com ?utm_source=google &utm_medium=cpc &utm_term=designer+cookware &utm_campaign=AdWords05 &utm_content=SeattleChefs

Text Ad #2 Designer Cookware Recommended by Seattle's top chefs. www.abccustomcookware.com Destination URL: http://www.abccustomcookware ?utm_source=google &utm_medium=cpc &utm_term=designer+cookware &utm_campaign=AdWords05 &utm_content=InteriorDesigners

The ads have different copy in the descriptions and are tagged with a different utm_content tag. The different tags allow you to view conversion metrics for each version of the ad in the Ad Versions report (in the Traffic Sources section). For each keyword, you can drill down and compare the performance of each version of the ad.

You can use the same approach with email campaigns.

The same tagging approach is easily used for emails. Create one or more versions of your emails, and tag the links in the email with a utm_content tag that identifies the particular version of the email. Your results will be visible in the Ad Versions report.

For example, to conduct an email A/B test, you send the "A" version of your email to one half of your group, and the "B" version to the other half. If you have a large number of recipients, you can even test three or more versions of your email. Your results will be most useful, however, if each group contains at least 5,000 recipients.

The case for A/B/A Testing

Although A/B testing is adequate for many situations, the results will be misleading if you have introduced any bias into the experiment. In other words, you might see a difference in the performance of your "A" and "B" versions not because of the emails themselves, but because your target audience groups differ in some important way.

You are unlikely to encounter bias in a randomized AdWords campaign such as the one outlined in the example above, but it is possible to unknowingly introduce bias in email campaigns. Even more common is the bias introduced by different versions of an ad on a single page. How can you be sure that differences in how the ads perform is actually due to the different creative, and not to the differences in location on the page?

To guard against this kind of bias, you can use an A/B/A test.

An A/B/A test splits your target audience into three groups. One third receives the "A" creative, one third receives the "B" creative, and one third receives the same "A" creative. This approach allows you to compare results from the two "A" groups against each other to determine how much bias may be introduced by other factors. Theoretically, clickthroughs and conversions from the two "A" groups will be virtually identical. If they are not, this means some factor other than the ads themselves is at work.

Start testing.

Using A/B and A/B/A testing, you can test ad copy, images, design and layout, colors, and many other factors. The key to doing it successfully is to vary one thing at a time. If the ads you test are completely different, you won't know which factors contribute to your success. Remember to tag each of your links within your ads and use a unique utm_content tag within each link.

As with other kinds of tracking, expect to learn a great deal about what motivates your audience. The more you test, the more you'll learn.