PROS Resources

This blog post is a recap of the presentation “Lies, Damned Lies, and Search Marketing Statistics” given by Vistaprint SEO Manager (North America, Australia, New Zealand) Adria Kyne at the SMX Advanced Conference in Seattle, June 2016. During this presentation, Adria touches on the use of Bayesian methods to infer the results of an A/B test.

Her point was that the sample size for A/B testing is often too small and experiments are ended too early to be statistically significant. So, before prematurely acting on results we must understand the likelihood that any performance differences we observe are due only to chance rather to the change we are testing.

To this end, Bayesian methods help avoid common pitfalls of statistical testing and make analysis and results easier to understand while enabling better decision-making.

Thomas Bayes

Bayesian Inference

Bayesian inference is used during statistical modeling to update the probability of a hypothesis based upon ongoing data collection.  In short, using Bayesian methods allows for communicating that there is a “90% probability that campaign B performs better than campaign A”.

It starts with identifying prior beliefs – or “prior” – about what results are likely, and then updating those according to the data collected. For example, if the current conversion rate is 5% it is very unlikely to achieve a conversion rate higher than 20%:

As the data begin compiling, the beliefs are updated. If the incoming data points signal an improvement in the conversion rate, the estimate of the effect can be moved from the prior upwards; the more data that is collected, the more confident one can be in the results moving away from the prior – then called a “posterior”, or the likely distribution of the effect.

Bayes’ Theorem actually allows for stopping a test at any time if there is a clear winner or run it for longer if more samples are needed. In Bayesian testing, outputs are easily interpreted quantities – for example, the probability that version A is better than version B.

The more popular alternative to Bayesian inference is classical significance testing, which draws conclusions based on the data sets’ comparison to a null hypothesis

There is a great article on Lyst’s engineering blog that goes more in depth on the subject of Bayesian A/B testing.

Case Study

The objective is to test PROS’s proprietary Dynamic Price Insertion in ad copy and analyze the results using Bayesian Inference.

Set Up

Dynamic Price Insertion is a new product that PROS has built which allows for inserting the lowest fare in an airline’s ad copy for every route in real-time.  To test the tool’s efficacy, an experiment was necessarily set up. The goal of the ongoing experiment is to determine if dynamically surfaced data performs better in ads than a static ad.

The ad copy tested was exactly the same for control and experiment, except that a price was added to the headline of the ad copy of the experiment.

Control:

Static ad for flights from Athens to London

Experiment:

Dynamic Price Insertion of €62 in same ad for flights from Athens to London

Analysis of Results using Bayesian Calculator

The test is still very young as the experiment has been running for four days only. However, we can use Bayesian Inference to determine what the probability of the experiment performing better at this moment than the control.

Using a Bayesian calculator can provide insight as to the outcomes of A/B testing, even when signals are weak.  There are different online Bayesian calculators, but here is the one used for this analysis and an accompanying description of the underlying principles of the calculator.

The results show that after running the experiment for four days only that the probability of the ad copy with dynamic price insertion performing better than the static ad copy is 54%.

Version Description Trials (Impressions) Successes (Conversions) ~ Prob.of being best
Control Static-Low Price 82,710 9 46%
Experiment Dynamic Price Insertion 52,974 6 54%

Conclusion

Using Bayesian Methods is a great communication and A/B testing analysis tool to better understand marketing results. If we would stop our experiment right now, the probability of the experiment performing better than the original static ad copy is 54%. Bayesian methods allow for getting results and drawing inferences at any time. The experiment has only run for four days, but we are already able to draw conclusions using these methods. However, by continuing to collect more data, the level of confidence in the results will grow meaning we can move further away from our prior.

Other content in this Stream

AI Search Engines Are Here—Is Your Airline Ready?

AI is reshaping flight search. Discover how airlines can adapt SEO for ChatGPT, Gemini & more to capture direct traffic and high-intent bookings.

Blog

Building Materials Industry at a Crossroads: Achieving Consistent Margins with Smart Pricing

Discover how PROS pricing solutions protect margins, drive value, and optimize operations for building materials manufacturers and distributors.

Blog

Collaborative Quoting: Minimizing Sales Friction for Better Results

Learn how collaborative quoting minimizes sales friction, boosts efficiency, and creates seamless digital buyer-seller experiences.

Blog

Rewiring MedTech for the Value Era: Why Digital Pricing Is the Next Frontier

Discover how PROS pricing and quoting solutions empower MedTech companies to thrive in a value-driven, AI-powered landscape.

Blog

PROS Platform Travel Summer Release 2025

Discover how PROS Summer 2025 release transforms airline retailing with AI agents, dynamic pricing, NDC automation, and optimized offers for revenue growth.

BlogPlatform Releases

PROS Platform B2B Summer Release 2025

Discover the PROS Summer Platform Release 2025 with AI, CPQ, pricing, and rebate innovations to help B2B businesses outperform and exceed expectations.

BlogPlatform Releases