Testing: One, Two, Three
by Rafe VanDenBerg
03 Aug, 2007
The concept of testing has burst onto the Internet scene in a big way. The push for improved ROI and the greater availability of tools and technology to aid the testing process has fueled the excitement. But can this "next great thing" be the solution to all of our problems online?
Testing has actually been around for a long time. Nearly a century ago, direct-marketers were conducting fairly sophisticated in-market tests using coded coupons in newspaper advertisements and advertisers have been leveraging testing methods such as "split-runs" in print and broadcast advertising for decades.
On the Internet, however, testing can be taken to a whole new level of depth, sophistication, and speed that just isn't practical or cost-effective in the offline world. In an online environment, nearly anything can be tested with enough thoughtful planning, the right methodology and the right tools.
Sequential Testing
Sequential testing is the most common form utilized on the Web today - maybe because it matches what we humans tend to do naturally or as a matter of course. In sequential testing, you simply measure ratios, trends or performance-bands over time, make changes to your approach then see whether those measurements go up or down.
For example, you might track your checkout abandonment ratios on a weekly basis and discover that the performance-band consistently hovers around 30-40 percent. You might then decide to make some improvements to your checkout process to see if you can positively impact performance. After enhancing the usability of your checkout forms, adding another payment method and including some reassuring messaging throughout the process, you monitor the ratios and see that you've reduced checkout abandonment to 20-25 percent.
This type of testing is fairly easy and intuitive for most people. You generally don't need any additional tools or vendors beyond your current analytics solutions and maybe a spreadsheet or two. On the downside, however, sequential testing takes time. It can take weeks to establish baseline trends and even more time to see conclusive changes to those trends. And that linear expanse of time introduces another variable that can call into question the validity of your results.
A lot can change in the marketplace over a short period of time. With sequential testing, this makes it very difficult to know whether changes in trend-performance are due to your specific actions or external marketplace dynamics. In certain fast-moving markets, the results of a sequential test could be completely unreliable if market conditions are changing dramatically throughout the testing period.
Split-Testing or A/B Testing
Split-testing - or A/B testing, as it's often referred to - addresses the deficiencies in sequential testing by eliminating the time variable. With split-testing, two approaches are tested at the same time, in parallel. In theory, both approaches will then be operating under the exact same market conditions, producing a more-reliable result.
For example, you may want to test whether a new call-to-action on your shopping-cart page will have a positive influence on the ratio of visitors proceeding to checkout. With split-testing, you can rotate your current call-to-action and the new call-to-action on a visitor-byvisitor basis. In effect, half of your visitors get the current messaging and the other half get the new messaging. By tracking the behavior of the visitors to each treatment, you can see which call-to-action produces the better result. Because the treatments are being rotated between every other visitor in the same time period, the influence of external factors is less of a concern.
In addition to reliable results, split-testing also tends to be highly time effective. Whereas a sequential test might require a month to complete, a split-test might only require a week.
However, there's generally a little more back-end work involved with split-testing. While the concept of testing "A" versus "B, or even "A" versus "B" versus "C" on a rotational basis is pretty straightforward and easy to grasp, actually executing the tests usually requires some additional tools. These specialized tools handle the rotational serving of whatever you're trying to test while also facilitating the proper tracking of the end-results. Once you've engaged a vendor or implemented a split-testing tool, conducting split-tests is a relatively simple matter.
Of course, the downside to split-testing - if there really is a downside - is that only limited variations can be tested with accuracy and in a timely manner. In most cases, you're only able to split-test two or three variations of a single element at one time and produce a reliable result.
You can, of course, split-test wholly different pages with different messaging, layouts and advertisements and ultimately understand which page performs best. But you won't know which specific elements within those treatments are actually influencing the result. Was it the different headline? Or maybe it was the different layout? Or maybe it was the offer?
Multivariate Testing
Multivariate testing is a relative newcomer to the Internet. Like split-testing, multivariate testing allows for the testing of different variations in the same timeframe. But multivariate testing allows you to test dozens, even hundreds or thousands of different variables all at the same time. Using sophisticated mathematical algorithms, multivariate testing can isolate the influence of individual variables, as well as measure the variables' influences on each other.
Through multivariate testing, it's possible to determine the bestperforming combination of multiple variables.
For example, you might want to develop an improved landing page for a major online advertising initiative. You know that a number of different page elements could impact performance - the headline, the imagery, the layout, the call-to-action, the components of the offer, etc. In fact, you might have two or three variations of each of these elements to test. Of course, split-testing each variation of each individual element would take forever. Through multivariate testing, you could test them all at the same time, and ultimately understand not only the influence of the individual variations, but also which combination of all the different variables produces the best results.
As you might have guessed, there's a lot going on behind-thescenes with multivariate testing. The variations you want to test must be defined and combined into a smaller number of "test recipes" to run. After performance data is gathered on the test recipes, the real fun can begin - applying the statistical algorithms to determine the isolated and combined influences of the elements and variables.
While multivariate testing can be extremely powerful and efficient, it can also be a fairly complex practice. Proper experimental design and execution on a large scale usually requires specialized expertise and tools. Rest assured, however, that there are a number of vendors and tools available to help. In fact, the tools have advanced at such a rapid pace that multivariate testing is now within reach for smaller sites and adventurous do-it-yourselfers.
Testing is not a Panacea
With all of the hype surrounding testing, it's easy to see it as the solution to everything. But, just like anything else in online business, it's just another piece of the puzzle. And doing your homework makes testing easier and far more productive.
You only have a limited amount of time and resources, so you have to prioritize. You shouldn't just test for the sake of testing. Identify where testing will produce the greatest impact. Will it be more profitable to spend time and money testing your landing pages or aspects of your checkout process? Should you test changes to your homepage or product pages? Where is the greatest opportunity for profitable improvement? Even the most sophisticated testing methods will yield only marginal results when applied to the wrong areas.
Understand your target prospects and competitors. Without a solid foundation of marketing research and competitive analysis, you might test 100 different things that your prospects fail to find particularly important or differentiating. Of course, you'll be able to determine the best of the lot - but the whole lot might be lacking.
Often, you will need to look well beyond the test results to understand the true impact to your business. For example, while a certain offer might produce a higher conversion rate and more orders, it might produce less profit once all variable costs are considered. Another combination of elements might produce more short-term sales, but customers with lower lifetime values, higher support costs or higher return rates. You should always try to expand on your test findings to assess the full impact to your firm's profitability.
The bottom line is that testing-in all of its various forms-can be very powerful when applied properly. But testing can't solve all of our problems and we can't abdicate our responsibilities to the testing process. We still have to do our homework, prioritize, and ensure that our long-term business objectives are being met.
Testing has actually been around for a long time. Nearly a century ago, direct-marketers were conducting fairly sophisticated in-market tests using coded coupons in newspaper advertisements and advertisers have been leveraging testing methods such as "split-runs" in print and broadcast advertising for decades.
On the Internet, however, testing can be taken to a whole new level of depth, sophistication, and speed that just isn't practical or cost-effective in the offline world. In an online environment, nearly anything can be tested with enough thoughtful planning, the right methodology and the right tools.
Sequential Testing
Sequential testing is the most common form utilized on the Web today - maybe because it matches what we humans tend to do naturally or as a matter of course. In sequential testing, you simply measure ratios, trends or performance-bands over time, make changes to your approach then see whether those measurements go up or down.
For example, you might track your checkout abandonment ratios on a weekly basis and discover that the performance-band consistently hovers around 30-40 percent. You might then decide to make some improvements to your checkout process to see if you can positively impact performance. After enhancing the usability of your checkout forms, adding another payment method and including some reassuring messaging throughout the process, you monitor the ratios and see that you've reduced checkout abandonment to 20-25 percent.
This type of testing is fairly easy and intuitive for most people. You generally don't need any additional tools or vendors beyond your current analytics solutions and maybe a spreadsheet or two. On the downside, however, sequential testing takes time. It can take weeks to establish baseline trends and even more time to see conclusive changes to those trends. And that linear expanse of time introduces another variable that can call into question the validity of your results.
A lot can change in the marketplace over a short period of time. With sequential testing, this makes it very difficult to know whether changes in trend-performance are due to your specific actions or external marketplace dynamics. In certain fast-moving markets, the results of a sequential test could be completely unreliable if market conditions are changing dramatically throughout the testing period.
Split-Testing or A/B Testing
Split-testing - or A/B testing, as it's often referred to - addresses the deficiencies in sequential testing by eliminating the time variable. With split-testing, two approaches are tested at the same time, in parallel. In theory, both approaches will then be operating under the exact same market conditions, producing a more-reliable result.
For example, you may want to test whether a new call-to-action on your shopping-cart page will have a positive influence on the ratio of visitors proceeding to checkout. With split-testing, you can rotate your current call-to-action and the new call-to-action on a visitor-byvisitor basis. In effect, half of your visitors get the current messaging and the other half get the new messaging. By tracking the behavior of the visitors to each treatment, you can see which call-to-action produces the better result. Because the treatments are being rotated between every other visitor in the same time period, the influence of external factors is less of a concern.
In addition to reliable results, split-testing also tends to be highly time effective. Whereas a sequential test might require a month to complete, a split-test might only require a week.
However, there's generally a little more back-end work involved with split-testing. While the concept of testing "A" versus "B, or even "A" versus "B" versus "C" on a rotational basis is pretty straightforward and easy to grasp, actually executing the tests usually requires some additional tools. These specialized tools handle the rotational serving of whatever you're trying to test while also facilitating the proper tracking of the end-results. Once you've engaged a vendor or implemented a split-testing tool, conducting split-tests is a relatively simple matter.
Of course, the downside to split-testing - if there really is a downside - is that only limited variations can be tested with accuracy and in a timely manner. In most cases, you're only able to split-test two or three variations of a single element at one time and produce a reliable result.
You can, of course, split-test wholly different pages with different messaging, layouts and advertisements and ultimately understand which page performs best. But you won't know which specific elements within those treatments are actually influencing the result. Was it the different headline? Or maybe it was the different layout? Or maybe it was the offer?
Multivariate Testing
Multivariate testing is a relative newcomer to the Internet. Like split-testing, multivariate testing allows for the testing of different variations in the same timeframe. But multivariate testing allows you to test dozens, even hundreds or thousands of different variables all at the same time. Using sophisticated mathematical algorithms, multivariate testing can isolate the influence of individual variables, as well as measure the variables' influences on each other.
Through multivariate testing, it's possible to determine the bestperforming combination of multiple variables.
For example, you might want to develop an improved landing page for a major online advertising initiative. You know that a number of different page elements could impact performance - the headline, the imagery, the layout, the call-to-action, the components of the offer, etc. In fact, you might have two or three variations of each of these elements to test. Of course, split-testing each variation of each individual element would take forever. Through multivariate testing, you could test them all at the same time, and ultimately understand not only the influence of the individual variations, but also which combination of all the different variables produces the best results.
As you might have guessed, there's a lot going on behind-thescenes with multivariate testing. The variations you want to test must be defined and combined into a smaller number of "test recipes" to run. After performance data is gathered on the test recipes, the real fun can begin - applying the statistical algorithms to determine the isolated and combined influences of the elements and variables.
While multivariate testing can be extremely powerful and efficient, it can also be a fairly complex practice. Proper experimental design and execution on a large scale usually requires specialized expertise and tools. Rest assured, however, that there are a number of vendors and tools available to help. In fact, the tools have advanced at such a rapid pace that multivariate testing is now within reach for smaller sites and adventurous do-it-yourselfers.
Testing is not a Panacea
With all of the hype surrounding testing, it's easy to see it as the solution to everything. But, just like anything else in online business, it's just another piece of the puzzle. And doing your homework makes testing easier and far more productive.
You only have a limited amount of time and resources, so you have to prioritize. You shouldn't just test for the sake of testing. Identify where testing will produce the greatest impact. Will it be more profitable to spend time and money testing your landing pages or aspects of your checkout process? Should you test changes to your homepage or product pages? Where is the greatest opportunity for profitable improvement? Even the most sophisticated testing methods will yield only marginal results when applied to the wrong areas.
Understand your target prospects and competitors. Without a solid foundation of marketing research and competitive analysis, you might test 100 different things that your prospects fail to find particularly important or differentiating. Of course, you'll be able to determine the best of the lot - but the whole lot might be lacking.
Often, you will need to look well beyond the test results to understand the true impact to your business. For example, while a certain offer might produce a higher conversion rate and more orders, it might produce less profit once all variable costs are considered. Another combination of elements might produce more short-term sales, but customers with lower lifetime values, higher support costs or higher return rates. You should always try to expand on your test findings to assess the full impact to your firm's profitability.
The bottom line is that testing-in all of its various forms-can be very powerful when applied properly. But testing can't solve all of our problems and we can't abdicate our responsibilities to the testing process. We still have to do our homework, prioritize, and ensure that our long-term business objectives are being met.
Rafe VanDenBerg
Rafe VanDenBerg is the founder and president of Business Development Xcellerator, Inc. The creator and architect of the Xcelleration Process, he helps companies large and small to optimize and improve their online sales and marketing-ultimately producing more growth and profits, with greater leverage and scalability.
