In an Automated Personalization experience, Monetate maximizes the performance of each individual. To accomplish this measurement, it separates experience traffic into two groups: Automated Personalization and Holdout. The average performance of all variants in the Automated Personalization experience is compared to the holdout traffic performance to measure the value of personalization.
For information about configuring a holdout for an Automated Personalization experience, see the Configuring Experience-Level Context in Engine Context.
Holdout Types
There are two types of holdouts. Each provides a different base for measurement and should be used in different situations.
Random Assignment
A Random Assignment holdout takes the designated traffic percentage and equally distributes it among the variants within the Automated Personalization experience. Monetate takes the average goal performance for each variant in the holdout to calculate the holdout's performance.
Single Variant
A Single Variant holdout takes the designated traffic percentage and sends it all to one variant. You can select a variant from the Automated Personalization experience to use as the holdout. Monetate then takes the performance of the holdout traffic to calculate the holdout's performance.
Determining Which Holdout to Use
| Without Automated Personalization, Would You Have... | Appropriate Holdout | Explanation |
|---|---|---|
| Given everyone in the audience the same experience? | Single Variant | This type measures the value of introducing content different from what you would have served everyone while making intelligent decisions. |
| Run a Standard Test experience? | Random Assignment | This type accounts for the introduction of new content using intelligent decisions while minimizing risk. |
| Run a Dynamic Testing experience? | Random Assignment | This type accounts for the introduction of new content using intelligent decisions |
| Given everyone in the audience the winner of a test? | Random Assignment | This type reduces the risk of having to run a test while accelerating time to ROI. |
Traffic Allocation in Automated Personalization Experiences
Traffic allocation is a trade-off between ease of lift measurement and potential ROI. Therefore, traffic allocation in an experience should be based on how much you value each factor. This diagram can help inform your decision.
