Data interpretation begins with statistical confidence and understanding what it means in the context of experience results. Confidence is the measurement of the likelihood that the observed change in a KPI did not occur by random chance. Monetate reports confidence at three confidence levels: 90%, 95%, and 99%. The higher the confidence, the less likely that a test's results occurred by chance.
For example, if an experience achieves 95% confidence, then Monetate estimates that the difference between experiment and control is not due to random variation in 95% of possible outcomes.
Changes in Confidence
Unlike a laboratory experiment or a clinical trial, website testing takes place under real world conditions. Confidence is influenced by a number of factors, including the difference in lift between experiment and control groups and the sample size.
Visitor behavior can change with time. For example, if you test a new experience on your site, you may see an initial performance lift. However, the novelty of what you test may diminish over time. When this occurs, the behavior of the experiment begins to trend towards that of the control group and statistical confidence drops.
Acting on the Data
Statistical confidence is influenced by a number of factors. The time to reach statistical confidence is based on a number of variables including sample size and the magnitude of difference between the performance of the experiment and control groups. For all metrics and KPIs, Monetate declares confidence at 90%, 95%, and 99%.
If you're running a split in an experience, then confidence is a great mathematical indicator of when to act on the results data. However, it's not the only factor that you should consider. Keep these other considerations in mind:
- How long has the test been running? Confidence increases with more data (and time) and with less variance.
- How big is the sample size?
- How prominent is the personalization that the experience is driving on your site?
- On what page(s) is this change being rendered?
- Is there a time constraint under which this experience must have a declared winner?
After you take these factors as well as your confidence level into account, you can accurately declare a winning split for your experience.