Build a Multivariate Test Experience

You can use Multivariate Test (MVT) experiences to test all the possible permutations of multiple elements on your page. For example, a standard A/B test may contain three actions, but a site visitor only ever sees a version of the page with all the actions turned on or all the actions turned off.

An MVT test, on the other hand, will test all possible combinations of those three actions to help you determine the best combination. With that said, multivariate experiences require a longer period of time to run before they provide usable insights because they require more traffic to declare significance than a standard A/B/n test.

If you're unsure what to test or are just beginning to build experiences, running an MVT experience on all pages can provide the most flexibility and the largest amount of data.

Multivariate Test experiences should not be viewed as a standalone element, but more like the first step in the experimentation process that should be followed up with additional A/B/n testing.

MVT experiences may require at least a month to declare significance unless your site has a high level of traffic.

Follow these steps to build an MVT experience.

  1. Click EXPERIENCES in the top navigation bar, and then click Web.

    Callout of the Web option in the COMPONENTS menu

  2. Click the arrow to the right of CREATE EXPERIENCE, and then select Multivariate Test (MVT).

    Callout of the 'Multivariate Test (MVT)' option in the menu of the 'CREATE EXPERIENCES' button on the Web Experiences list page

  3. Name the experience.

    Callout of the Name field on the 'Multivariate Test (MVT)' modal

  4. Select the page type on which the experience should run.

    The options in Page Type are determined by the page types passed from your site to Monetate in the setPageType method.

    Callout of the 'Page Type' selector on the 'Multivariate Test (MVT)' modal

  5. Select the goal metric.

    You cannot include custom events in MVT experiences.

    Callout of the 'Goal Metric' selector on the 'Multivariate Test (MVT)' modal

  6. Click CREATE.

    Callout of the CREATE button on the 'Multivariate Test (MVT)' modal

  7. Optionally, narrow the target audience for the experience. Click WHO, then click ADD TARGET, select a target type, and then select and configure an option. See the Targets section of the User Guide for more information about the types of targets and configuring them.

    When adding targets to an MVT experience, consider how long it may take to obtain a large enough audience sample to declare significance. As a result, a larger target audience allows you to obtain significance in less time.

    Callout of the WHO settings and the ADD TARGET button on the Experience Editor page

  8. Click WHAT, then click ADD ACTION, select an action type, and then select and configure an action template.

    Only the action types that are relevant to the page type you selected in step 4 appear after you click ADD ACTION.

    You cannot add action conditions to individual actions in an MVT experience.

    Animated demonstration of a user clicking the WHAT settings, clicking the ADD ACTION button, and then viewing the action types available for the page type

  9. Optionally, click ADD VARIANT and then configure a variant of the action you configured in the previous step.

    Callout of the 'ADD VARIANT' button for the first action on the 'Page Targeting Conditions' panel of the WHAT settings

  10. Repeat the previous step as many times as necessary to configure the action variants that you want to test.
    • The original version of a site element is included and tested as a variant for any action that you add to the experience. Therefore, an experience with two sets of variants for a single action actually has three when the original is considered. To determine the total variants in an MVT experience, multiply the number of variants including the original for each action.
    • The more permutations that are present in your experience, the higher the risk of getting false positives. As a best practice, remain agile: Complete a multivariate experience with the fewest number of variants, review the results, and then create an A/B/n experience that focuses on the highest or the lowest performing segment. As a rule of thumb, you should limit the number of permutations to 25 or less.
  11. Optionally, repeat steps 8 through 10 to add an additional action and multiple variants of that action to the experience.
  12. If the experience doesn't target all pages or if you want to add any conditions, then configure the page-targeting conditions.
    1. Click EDIT PAGE TARGETING CONDITIONS.

      Callout of the 'EDIT PAGE TARGETING CONDITIONS' button on the 'Page Targeting Conditions' panel of the WHAT settings

    2. Click ADD CONDITION.

      Callout of the ADD ACTION button on the Experience Page Targeting' panel of the WHAT settings

    3. Select an action condition category and then an option from that category.

      View of the 'Landing page' category and its options in the ADD CONDITION selector

    4. Complete the action condition equation, and then either click ADD CONDITION to create another page-targeting condition or click Save.

      If you create multiple conditions, then all actions must meet all the conditions. You cannot use OR logic with multiple conditions.

      Callout of the 'ADD CONDITION' selector and the Save button

  13. Optionally, calculate the experience's minimum duration.
    1. Input an estimate of your site's average number of sessions in a day.

      Callout of the 'Daily sessions' field in the 'Minimum test duration' portion of the WHAT settings

    2. Input an estimate of your site's current conversion rate percentage.

      Callout of the 'Current conversion rate percent' field in the 'Minimum test duration' portion of the WHAT settings

    3. Input an estimate of the conversion rate percentage you anticipate as a result of the experience.

      Callout of the 'Expected conversion rate percent' field in the 'Minimum test duration' portion of the WHAT settings

    4. Optionally, adjust any of the estimates you input in the previous steps to recalculate the estimated minimum experience duration.

      Callout of the calculated minimum duration for the experience

  14. Optionally, configure the start and stop times for the experience. Click WHEN and then click START AND STOP TIME. See Configure Experience Timing for more information.

    Callout of the WHEN settings and the 'START AND STOP TIME' link

Ensure that you preview the experience before you activate it. When you click PREVIEW and select the first option in the menu, you can select which variant of each action that you want to preview.

Animated demonstration of a user clicking the PREVIEW button, clicking the 'Preview...' option, and then reviewing the variant options for each action in the 'Select variants for preview' modal

See Activate an Experience for Preview and Testing for more information.