An Introduction to A/B Testing

1.1 Why You Should A/B Test

What is an A/B test

An A/B test is the comparison of one original (control) landing page to another page or set of pages. When you’re getting started it’s usually easiest to compare individual items on your landing pages so you can see what produced the outcomes you measure in your test.

After running an A/B test for a set amount of time (usually in 7 day increments) and with an adequate volume of traffic, you can compare conversion rates to see how changes in a single page element affect your results positively, negatively, and to what degree.

Terms like A/B/n or Split Testing are all synonyms for A/B testing, and can be thought of as a more specific expression of the testing process. You may use the term A/B/n to clarify that you are running an A/B test with multiple variants. In this case, though you could be comparing 5 pages, each of them is still testing different versions of the same change. For example, you could A/B/n test the headline of your landing page by comparing 5 different headlines, but no other part of the landing page will vary.

Some other common ‘categories’ of A/B testing include layout choices (placement of page elements) and functionality choices (interactive elements such as error messages).

A/B Testing: What you’ll need:

Traffic is a common concern for marketers wanting to run valid A/B tests. In order to make sound statistical decisions by A/B testing, a good ballpark is to aim for at least 100 conversions per variation before looking at statistical confidence. If you have high traffic volume, the more conversions that you have per page variant, the more confident you can be in your test results. RJ Metrics offers a useful online tool for calculating sample sizes: http://www.testsignificance.com

It’s wise to wait for about 1,000 total visitors in your test, and to run your test for at least 7 days. This way you can account for day to day fluctuations.

What decisions can you impact through A/B testing?

At their core, A/B tests are the most useful in evaluating the sensitivity of your audience to really detailed elements of your landing pages. With the knowledge you gain through A/B testing, you can make incremental improvements to the performance of your page, and you can anticipate which changes will be more noticeable (for better or for worse) to your customers.

You can use A/B tests to cushion drastic changes like redesigning a homepage by only serving it to a portion of your visitors at a time. Companies like Twitter and Facebook use this strategy to test major interface changes by only rolling out the new version to a segment of their visitors and measuring how that group reacts.

In the above example, A/B testing can work as a change management tool that helps you keep control over customer response to a set of changes on your site.

When done right, A/B testing can be a fast and simple way to get see significant changes in your conversion rates, and will teach you which strings to pull to connect with your audience.

Advantages of A/B testing in Unbounce

  • A/B tests are fast to set up and run. No fuss no muss
  • While the process is simple, the data is broad. You can use advanced analytics for each variation (e.g. click tracking, heatmaps, etc.)
  • You can achieve more dramatic conversion rate lift results
  • A/B testing requires loads less traffic than a multivariate test, so you can get actionable insights faster