Skip to main content
Asked a question last year

What is Multivariate Testing ?

Where am I?

In SaaSworthy you can ask and answer questions and share your experience with others!

To understand MultiVariate Testing (MVT), one needs to understand A/B and A/B/n testing.

A/B Testing is an experimentation process, where you pick one topic that you're looking to experiment on (e.g. Button Colour, Headline, Layout, Page, etc.) and test your current version vs. one alternative. If the topic of change was Button Colour, and your default version (known as the Control group) was Blue, an ABn Test may see you test that against a Green button.

A/B/n Testing is similar to A/B, except that you're testing more than one alternative. If the topic of change was Button Colour, and your default version/Control group was Blue, an ABn Test may see you test that against a Green, Purple and Black button.

MVT is looking not just at one topic, but considering multiple topics within a single Test. In the above example, your Topics could be Button Colour, Headline and Layout. An MVT would see you trial one or more alternatives of each of these topics. You do not need the same number of variations in each topic, if you're running a Full Factorial test (see below). An example structure of this MVT could be:

  • Button Colour:
    • Control: Blue
    • Variation 1: Green
    • Variation 2: Purple
  • Headline:
    • Control: Denim Jeans
    • Variation 1: Next-generation denim jeans
    • Variation 2: Comfortable, stretchy denim jeans
  • Layout:
    • Control: Current page layout
    • Variation 1: Some different page layout
    • Variation 2: Another different page layout

There are also two types of MVTs you can run - Full Factorial and Fractional Factorial.

Full Factorials test every possible combination of the options you've constructed. The principle behind this is to be thorough. In the above example, you'd be running 3x3x3 = 27 experiments.

Fractional Factorial looks to test a balanced subset of these, where every option is given an even chance of exposure to users. The principle behind this is efficiency and indicative results. In the above example, the subset would be just 9 experiments, which is considerably more traffic efficient.

Hope that explains things in detail! 

Sandeep from the Webtrends Optimize7 team.