Build a Multivariate Test Experience

You can use Multivariate Test (MVT) experiences to test all the possible permutations of multiple elements on your page. For example, a standard A/B test may contain three actions, but a site visitor only ever sees a version of the page with all the actions turned on or all the actions turned off.

An MVT test, on the other hand, will test all possible combinations of those three actions to help you determine the best combination. With that said, multivariate experiences require a longer period of time to run before they provide usable insights because they require more traffic to declare significance than a standard A/B/n test.

Multivariate Test experiences should not be viewed as a standalone element, but more like the first step in the experimentation process that should be followed up with additional A/B/n testing.

Follow these steps to build an MVT experience.

1. Click EXPERIENCES in the top navigation bar, and then click Web.

   

2. Click the arrow to the right of CREATE EXPERIENCE, and then select Multivariate Test (MVT).

   

3. Name the experience.

   

4. Select the page type on which the experience should run.

   The options in Page Type are determined by the page types passed from your site to Monetate in the setPageType method.

   

5. Select the goal metric.

   You cannot include custom events in MVT experiences.

   

6. Click CREATE.

   

7. Optionally, narrow the target audience for the experience. Click WHO, then click ADD TARGET, select a target type, and then select and configure an option. See the Targets section of the User Guide for more information about the types of targets and configuring them.

   When adding targets to an MVT experience, consider how long it may take to obtain a large enough audience sample to declare significance. As a result, a larger target audience allows you to obtain significance in less time.

   

8. Click WHAT, then click ADD ACTION, select an action type, and then select and configure an action template.

   Only the action types that are relevant to the page type you selected in step 4 appear after you click ADD ACTION.

   You cannot add action conditions to individual actions in an MVT experience.

   

9. Optionally, click ADD VARIANT and then configure a variant of the action you configured in the previous step.

   

10. Repeat the previous step as many times as necessary to configure the action variants that you want to test.
    • The original version of a site element is included and tested as a variant for any action that you add to the experience. Therefore, an experience with two sets of variants for a single action actually has three when the original is considered. To determine the total variants in an MVT experience, multiply the number of variants including the original for each action.
    • The more permutations that are present in your experience, the higher the risk of getting false positives. As a best practice, remain agile: Complete a multivariate experience with the fewest number of variants, review the results, and then create an A/B/n experience that focuses on the highest or the lowest performing segment. As a rule of thumb, you should limit the number of permutations to 25 or less.
11. Optionally, repeat steps 8 through 10 to add an additional action and multiple variants of that action to the experience.
12. If the experience doesn't target all pages or if you want to add any conditions, then configure the page-targeting conditions.
    1. Click EDIT PAGE TARGETING CONDITIONS.

       

    2. Click ADD CONDITION.

       

    3. Select an action condition category and then an option from that category.

       

    4. Complete the action condition equation, and then either click ADD CONDITION to create another page-targeting condition or click Save.

       If you create multiple conditions, then all actions must meet all the conditions. You cannot use OR logic with multiple conditions.

       

13. Optionally, calculate the experience's minimum duration.
    1. Input an estimate of your site's average number of sessions in a day.

       

    2. Input an estimate of your site's current conversion rate percentage.

       

    3. Input an estimate of the conversion rate percentage you anticipate as a result of the experience.

       

    4. Optionally, adjust any of the estimates you input in the previous steps to recalculate the estimated minimum experience duration.

       

14. Optionally, configure the start and stop times for the experience. Click WHEN and then click START AND STOP TIME. See Configure Experience Timing for more information.

    

Ensure that you preview the experience before you activate it. When you click PREVIEW and select the first option in the menu, you can select which variant of each action that you want to preview.

See Activate an Experience for Preview and Testing for more information.