Standard Test vs. Dynamic Testing Experiences

Standard Test experiences and Dynamic Testing experiences are both valuable tools, albeit for different goals. Consider the following points before you which test to build.

Standard Test Experiences

Standard Test experiences are A/B/n tests that offer a controlled learning environment. Select this experience type if one or more of the following statements best describe your goal:

  • Impact multiple metrics with a test. A Standard Test experience provides control over exposure by keeping the traffic distribution fixed throughout the life of the test.
  • Differentiate and understand true performance of all splits in a test.

Dynamic Testing Experiences

Dynamic Testing experiences offer faster ROI. You should select this experience type if one or more of the following statements best describe your goal.

  • Drive maximum ROI. Derive value regardless of which variant is deemed the best at any given point in time.
  • Test content within a small window of time. Determine quickly which split performs best while simultaneously driving ROI.

Testing Comparison

Standard Test ExperienceDynamic Testing Experience
One decisionMany decisions
Static — assumes nothing changesAdaptive — reacts to changes in customer behavior
Thoughtful experiment design and tactics — sample size calculator, goals, and conditionsJust tactics — goals and conditions
Significance — "measure of surprise"Probability — "chance the leader is actually the best at driving ROI given everything we know"
Reach significance, stop, promoteContinuous testing and learning drives more ROI
Manual monitoringFully automated
High resolution of performance on all variationsFocused on exploiting the leader
Measured understanding of all metricsObservation of metrics beyond goal for the leader