Landing Page Testing - Translating the Terminology
Before you can start doing any testing on your website landing pages, you need to understand some common concepts and definitions used in landing page testing. But be forewarned: while members of the testing community refer to the same concepts some may use different language to describe them.
The primary objective of landing page testing is to predict the behavior of your audience given the specific content on the landing page that they see. You will collect a limited sample of data during your test, summarize and describe it (descriptive statistics), and predict how people from the same traffic sources will act when interacting with the page (inferential statistics). In other words, the ultimate goal is to find the best possible version of the landing page among all of the variations that you are testing.
Input and Output Variables
A landing page test has two basic components: a set of input variables (also called "independent variables") that you can control and manipulate, and one or more output variables (or "dependent variables") that you measure and observe. Independent variables as discussed here are simply the tuning elements that you have chosen for your test.
Variable
The word variable (when used by itself) means a tuning element that you have selected. Variables can be of any granularity or coarseness. For example, a variable might be the headline of your landing page, or a whole-page redesign. In multivariate testing, a variable is also commonly referred to as a factor.
In a multivariate test, you will have more than one variable. To distinguish among them I use the following notation: a capital "V" followed by a unique number that has been assigned to a particular variable. For example, let's assume that you have a simple landing page with a headline, some sales copy test, and a call-to-action on a button, You might decide to test alternatives to each of these page elements and name them as follows:
- V1 = Headline
- V2 = Sales copy
- V3 = Button text
- V4 = Button color
Note that the variables do not necessarily define a unique physical location on the page. In fact, V3 (the button text) and V4 (the button color) actually occupy the same space. Nor do they have to be localized. For example, I can choose a variable to test a larger font size (for improved readability) versus an existing smaller one. In this case, the font size change would take effect throughout my whole landing page and would overlap with other variables (such as the actual text on the page) that I might also be testing.
Value
A value is a particular state that a variable can take on. When traditional multivariate testing is used in other fields, variable values are often continuous (which means they can vary smoothly across a range). This allows you to predict the behavior at interpolated values of the variable (in between the places where you actually sample). For example, if I know that the output of a car engine at 1000 RPM (revolutions-per-minute) is 100 horsepower, and at 2000 RPM is 200
horsepower, I can interpolate between these two values to estimate that the output should be 150 horsepower at 1500 RPM.
In landing page tuning, variable values are almost always discrete (distinct from each other, and countable). For example, a button color might be green, blue, or red. I will number the possible choices by successive lowercase letters. By convention, the letter a represents the original version of the variable (as seen on your baseline pretest landing page). The letter is combined with the variable name to exactly specify the value of a particular variable. If V4 is our button color, an example assignment might look as follows:
- V4a = green button (the original)
- V4b = blue button
- V4c = red button
Unlike continuous variables, measuring the effect of discrete variable values does not give us any information about the other possible values. Continuing our example from earlier, even if we had measured the average conversion rates with the green and blue buttons, we would not have any information about the performance of the red one.
Branching Factor
The total number of possible values for a discrete variable is called its branching factor. For discrete variables, the branching factor must be at least 2 (the original version and one alternative). Some experimental designs require that the branching factor be the same for all variables in the test.
In the button color example, V4 has a branching factor of 3 (because it can take on the values signified by a-green, b-blue, and c-red). In traditional multivariate testing, the number of values for a variable is called the level of the factor. Each value is also called a level because historically it was drawn from continuous variables. For example, if your variable only has two values, they might be signified by "low" and "high" (or "-1" and "+1").
Recipe
A recipe is a unique combination of variable values in your test. It is a sequential listing of the specific values that each variable takes on in the specific version of the landing page. For example, let's assume that you had set the following variable values from my previous example for a particular landing page in your test: V1b, V2c, V3a, V4a. The recipe could be abbreviated as bcaa.
Each recipe is unique. By convention, the recipe with all a's is the original or baseline recipe to which all others will be compared.
Search Space Size
The number of unique recipes in your test is your search space size. It can generally be computed by multiplying together all of the branching factors of the variables in your test (for a possible exception to this rule, please see the next section, "Test Construction").
In my earlier example, let's assume that there are three headlines, four versions of the sales copy, four calls-to-action, and three button colors (and "BF" stands for the branching factor for a particular variable).
Search space size
= BFV1 X BFV2 X BFV3 X BFV4
= 3 X 4 X 4 X 3
= 144
This example is a small one. As you can see, if you have more variables and higher branching factors for each one, the search space size will grow very rapidly. If the search space size is large, it can quickly exceed the practical limits of common tuning methods such as A-B split testing, fractional factorial parametric, and full factorial parametric testing.
Now that you understand the basic terminology of landing page testing you can begin to think about which items to test and how to construct your test.
