Small Experiments That Lead to Big Results: The Value of A/B Testing

The use of randomized experiments to determine the most effective marketing or communications approach – known as A/B testing – is an extremely valuable tool for companies aiming to make the biggest impact on key stakeholders. However, according to Pete Koomen, president of website optimization software company Optimizely, the method is not implemented nearly enough. At the BRITE ’13 conference, Koomen shared personal experiences to demonstrate the very real value that A/B testing can contribute to developing results-driven communications programs. The most compelling of his examples included the 2008 Obama presidential campaign and the countless tests its analytics team ran on www.barackobama.com website content and email subject lines. Koomen noted that even the slightest word phrasing could drive visitors to action (e.g., donating funds, signing up to volunteer). “It was an extremely powerful technique for [influencing] decisions,” he said. However, given the deep investment in time and resources needed for A/B testing, Koomen observed over time that companies tended to avoid using the technique – a fact that he and his business partner Dan Siroker quickly recognized as a major business opportunity.

The success of the 2008 campaign spoke for itself. Koomen estimated that methodical experimentation accounted for roughly $75 million more in donations to President Obama’s campaign and 4 million new website registrants. These results motivated Koomen and Siroker, both former Google product managers, to found Optimizely in 2009. They created a simple program that even small and medium-size businesses could utilize without having to depend on specialized in-house talent to run experiments. Organizations with limited resources could take advantage of marketing tactics that Amazon and other major blue-chip companies have been using for years to increase traffic and user conversion.

At BRITE ‘13, Koomen shared some best practices and lessons learned from running over 100,000 tests for clients and identifying the most effective approaches for achieving business objectives. For the Obama campaign, this entailed what Bloomberg BusinessWeek called “strange, incessant, and weirdly over familiar e-mails” due to the unusual, extremely casual tone Obama’s team usedjourney to office in 2008. The fundraising team found that the most successful subject heading “Hey” alone brought in millions of dollars in funding.

A few things that Koomen recommends businesses keep in mind as they take stock of their websites’ performance are:

• Define quantifiable success metrics. One of the most important parts of testing. As exemplified by the Obama campaign, Koomen states that the campaign staffers did a good job of attracting people to the official website, but turning the site’s visitors into subscribers had proved more challenging and converting email signups to paying donators even more so.  By tweaking the website to optimize those two KPIs – subscribers and payers – the new website outperformed the old version by about 40%.

• Explore before you refine. Koomen cautions against refining and optimizing in favor of exploring first to ensure you are aware of all potential solutions before selecting one to improve.  Otherwise, there is a chance the best solution will be missed.

• Less is more. Reducing optionality can have a major impact on a website’s effectiveness. Koomen cites a client which removed a series of links related to its product portfolio and company background from its shopping cart page and saw a 16% improvement in the dollars per visitor.

Watch Pete Koomen’s BRITE ’13 talk to learn more about how A/B Testing can drive greater communications effectiveness.

BY NANDITA RAY

Brand Value: Measuring All the Angles

Ah, the L.A. Dodgers, a team whose recent woes under the leadership of Frank McCourt resulted in a purchase that rocked the baseball world. In 2011, Forbes estimated the Dodgers to be worth $800 million, more than the average value of a major league baseball team team. Thanks to a TV deal in March 2012, Forbes upped that number to $1.4 billion. One month later, news erupted that Guggenheim Baseball Management, a consortium lead by Magic Johnson, would acquire the team for a whopping $2.15 billon.

Many questioned the rationale behind this dollar amount, exclaiming the consortium vastly overpaid for a franchise with such a comparatively low net worth. David Carter, executive director of the USC Sports Business Institute, tells Fox Business, that it’s not solely about buying an organization at face value. “It’s purchasing a baseball team that is an anchor that allows you to make money off other revenue-rich opportunities.”

Chief executives that agree with this see the value of a brand. However, many CEOs, CFOs, and accounting teams still see marketing and communications as cost line items rather than buttresses that sustain brands and lend book value to companies.

Columbia Business School recently published two studies related to varying perspectives organization’s have about the value of marketing. The first, “Accounting for Marketing Activities,” found that part of the problem is internal communications. Experts feel that marketers don’t clearly articulate the impact of their expenditures on bottom lines to finance officers. The second, “Marketing ROI in the Era of Big Data,” adds that there’s a need to develop a better understanding of how marketing creates financial returns for companies. Attributing value to marketing is a common struggle.

Marketing’s role, in its most basic form, is to get the word out to the right people through a range of channels. But an integral part of this is communicating a brand’s core message and relevance to its audience, demonstrating value for consumers, which in turn adds value to a company. The foundation of a strong brand is built by delivering on a promise—providing quality and dependable products and/or services. But intangible items like brand persona, experience, and reputation are also largely influential when it comes to actual purchase and loyalty, generating sales and revenue. Much of this process can be attributed to the emotional connection formed between brand and consumer.

Developing meaningful relationships with a brand’s audience is a vital part of path-to-purchase, advocacy, financial growth and longevity. Humanizing brands leverages such relationships. In addition to products and services, they offer distinct images, feelings, experiences, thoughts and perceptions creating differentiated value in the mind of its audience. In fact, a brand is, in essence, an extension of one’s self. Consumers are more apt to buy products from a particular brand that reflects their personality and aligns with their interests and personal values, subsequently raising the net worth of a brand.

Apply this notion to the Dodgers. Another factor contributing to the high-priced offer is that they’re the… L.A. Dodgers, a well-known brand that officially acquired its name over 70 years ago. There’s history behind these “boys of summer,” a term that’s not only in a Don Henley song, but the title of a book based on this very team. More than ticket sales, sponsorship, and TV deals, the bid amount stemmed from relationships with devoted fans and the iconic value of the Dodgers brand in and of itself.

Coca-Cola ad from 1938
Click to enlarge

Another prime example is Coca-Cola. In CNBC’s documentary, “Coca-Cola: The Real Story Behind the Real Thing,” Donald R. Keough, former president of Coca-Cola, discussed the implications of its 1985 debacle, the launch of the now-defunct “New Coke.” He received a tearful call from a stranger, an elderly woman. She was heartbroken by the reformulation of its trademark beverage. Interestingly enough, she hadn’t sipped the soda since the 1940s, and therefore not tasted this new iteration. However, the Coca-Cola brand held memories for her, ones that she strongly associated with her childhood. She told Keough that they were playing with her youth. When Coca-Cola decided to lay the original Coke to rest, it took a piece of her past with it.

A CFO may say this is a nice story, but she hadn’t been a consumer in years. Though she may not have spent a nickel on Coca-Cola in decades, the brand clearly resonated with her. And there were plenty of present-day consumers who also protested New Coke. Value to her and the millions of others was emotive. For those who did not like the taste, had the emotional connection not existed, they wouldn’t have expressed such outrage over the new formula. Instead, they simply would have ceased buying it. Keough noted to CNBC, “We did not understand the deep emotions of so many of our customers for Coca-Cola.”

BY ALLIE ABODEELY