Small Experiments That Lead to Big Results: The Value of A/B Testing

The use of randomized experiments to determine the most effective marketing or communications approach – known as A/B testing – is an extremely valuable tool for companies aiming to make the biggest impact on key stakeholders. However, according to Pete Koomen, president of website optimization software company Optimizely, the method is not implemented nearly enough. At the BRITE ’13 conference, Koomen shared personal experiences to demonstrate the very real value that A/B testing can contribute to developing results-driven communications programs. The most compelling of his examples included the 2008 Obama presidential campaign and the countless tests its analytics team ran on www.barackobama.com website content and email subject lines. Koomen noted that even the slightest word phrasing could drive visitors to action (e.g., donating funds, signing up to volunteer). “It was an extremely powerful technique for [influencing] decisions,” he said. However, given the deep investment in time and resources needed for A/B testing, Koomen observed over time that companies tended to avoid using the technique – a fact that he and his business partner Dan Siroker quickly recognized as a major business opportunity.

The success of the 2008 campaign spoke for itself. Koomen estimated that methodical experimentation accounted for roughly $75 million more in donations to President Obama’s campaign and 4 million new website registrants. These results motivated Koomen and Siroker, both former Google product managers, to found Optimizely in 2009. They created a simple program that even small and medium-size businesses could utilize without having to depend on specialized in-house talent to run experiments. Organizations with limited resources could take advantage of marketing tactics that Amazon and other major blue-chip companies have been using for years to increase traffic and user conversion.

At BRITE ‘13, Koomen shared some best practices and lessons learned from running over 100,000 tests for clients and identifying the most effective approaches for achieving business objectives. For the Obama campaign, this entailed what Bloomberg BusinessWeek called “strange, incessant, and weirdly over familiar e-mails” due to the unusual, extremely casual tone Obama’s team usedjourney to office in 2008. The fundraising team found that the most successful subject heading “Hey” alone brought in millions of dollars in funding.

A few things that Koomen recommends businesses keep in mind as they take stock of their websites’ performance are:

• Define quantifiable success metrics. One of the most important parts of testing. As exemplified by the Obama campaign, Koomen states that the campaign staffers did a good job of attracting people to the official website, but turning the site’s visitors into subscribers had proved more challenging and converting email signups to paying donators even more so.  By tweaking the website to optimize those two KPIs – subscribers and payers – the new website outperformed the old version by about 40%.

• Explore before you refine. Koomen cautions against refining and optimizing in favor of exploring first to ensure you are aware of all potential solutions before selecting one to improve.  Otherwise, there is a chance the best solution will be missed.

• Less is more. Reducing optionality can have a major impact on a website’s effectiveness. Koomen cites a client which removed a series of links related to its product portfolio and company background from its shopping cart page and saw a 16% improvement in the dollars per visitor.

Watch Pete Koomen’s BRITE ’13 talk to learn more about how A/B Testing can drive greater communications effectiveness.

BY NANDITA RAY