Standard Test experiences and Dynamic Testing experiences are both valuable tools, albeit for different goals. Consider the following points before you which test to build.
Standard Test Experiences
Standard Test experiences are A/B/n tests that offer a controlled learning environment. Select this experience type if one or more of the following statements best describe your goal:
- Impact multiple metrics with a test. A Standard Test experience provides control over exposure by keeping the traffic distribution fixed throughout the life of the test.
- Differentiate and understand true performance of all splits in a test.
Dynamic Testing Experiences
Dynamic Testing experiences offer faster ROI. You should select this experience type if one or more of the following statements best describe your goal.
- Drive maximum ROI. Derive value regardless of which variant is deemed the best at any given point in time.
- Test content within a small window of time. Determine quickly which split performs best while simultaneously driving ROI.
Testing Comparison
| Standard Test Experience | Dynamic Testing Experience |
|---|---|
| One decision | Many decisions |
| Static — assumes nothing changes | Adaptive — reacts to changes in customer behavior |
| Thoughtful experiment design and tactics — sample size calculator, goals, and conditions | Just tactics — goals and conditions |
| Significance — "measure of surprise" | Probability — "chance the leader is actually the best at driving ROI given everything we know" |
| Reach significance, stop, promote | Continuous testing and learning drives more ROI |
| Manual monitoring | Fully automated |
| High resolution of performance on all variations | Focused on exploiting the leader |
| Measured understanding of all metrics | Observation of metrics beyond goal for the leader |