Sentient has developed a Bayesian-based statistical model uniquely suited for Ascend's evolutionary AI methodology to improve customers' understanding and robustness of experiment results.
Bayesian vs Frequentist Statistics
Imagine you are trying to find a old shipwreck. You have some kind of sonar technology that will provide you data about where it thinks the shipwreck is. After a pass over the search space, the sonar provides the following data:
It thinks it has found the sunken vessel in the yellow area. This is the view of frequentist statistics – the only information available to make the decision is the observations that have been made to indicate the craft's position. Armed with only this information, you're reasonably confident this is where the shipwreck is.
Now imagine you also have a map of old shipping routes commonly used in the search space, (in orange). If these routes are no where near the location you think you have found the wreck, then suddenly you are far less confident in your conclusion about the ship's position. If the evidence goes against what is expected to be true, you need more evidence to reach the same level of confidence.
On the other hand, if the data suggest the wreck is on a known route, confidence in the conclusion is increased.
This is the Bayesian approach. There is a set of preexisting assumptions (the prior) that is combined with the evidence you have seen to give you your belief of what is happening (the posterior). After combining the prior and recorded data, the posterior ends up somewhere between the two data sets.
Sticking with the same example – having extra knowledge to guide a conclusion is clearly beneficial. It allows one to weigh the likelihood of unexpected claims, and more confidently trust claims that are consistent with how things are known to work.
But, what if your maps are wrong? In that case we'll be overly skeptical of the evidence presented to us. This makes the prior calculation one of the most important aspects of Bayesian statistics – get it wrong and it's worse than knowing nothing.
It's important to note that as more and more evidence accumulates, our prior becomes less and less important – if we had data showing all discovered shipwrecks of the same era in places that the maps claim they should never be, then we would throw away the maps.
The benefit of using Bayesian statistics in comparing performance is that we do know something beyond just the data we have collected for the candidate – not all performance rates are equally likely. No ecommerce store has 100% conversion, so why should we treat our analysis as
if 100% and 5% conversion rates are equally likely? (Frequentist statistics are actually equivalent to Bayesian statistics if our prior is assuming that all conversion rates are equally likely). Using this methodology for computation of a prior, Ascend is able to benefit from the unique way experiment candidates evolve. As many, many candidates are tested within a single experiment, a very clear view of the underlying spread of performance rates as a whole. This is be used to construct an accurate prior.