NPR’s Planet Money recently did a podcast on A/B testing. It is worth a listen if you are new to the topic. A/B testing is a key part of the current analytics movement.
The podcast did a nice job in highlighting one of the key case studies on how Barack Obama used it in his 2008 campaign. (For more information on that case and A/B testing in general, click here for an hour long lecture from one of the software companies leading A/B testing).
It also had a good case study of how a shoe retailer uses A/B testing and data to better lay out and improve her store.
The podcast also did a nice job highlighting the “big idea” of A/B testing. The big idea of A/B testing is that managers no longer have to argue about what idea is better, you can just test them. For example, with Obama’s website, if Obama wanted the sign-up button to be green and his aide wanted it to be red, they no longer had to argue about who was right– they could just test it.
This idea is part of a bigger trend in analytics, the idea that we can test our ideas with data and use models to help make better decisions.
I only had one minor and one bigger quibble with the podcast.
First, the minor point, when testing idea “A” vs “B” you don’t have to have a 50/50 split for the test. But, Planet Money isn’t a math podcast so that is OK.
Second, the bigger quibble, show tries to make a point that A/B testing takes away the creativity and judgement of people. This is not the case. Instead, what A/B testing does is help you make a decision between two good ideas. A/B testing won’t uncover the good ideas– you are still on the hook for coming up with those!