Small Experiments That Lead to Big Results: The Value of A/B Testing

The use of randomized experiments to determine the most effective marketing or communications approach – known as A/B testing – is an extremely valuable tool for companies aiming to make the biggest impact on key stakeholders. However, according to Pete Koomen, president of website optimization software company Optimizely, the method is not implemented nearly enough. At the BRITE ’13 conference, Koomen shared personal experiences to demonstrate the very real value that A/B testing can contribute to developing results-driven communications programs. The most compelling of his examples included the 2008 Obama presidential campaign and the countless tests its analytics team ran on www.barackobama.com website content and email subject lines. Koomen noted that even the slightest word phrasing could drive visitors to action (e.g., donating funds, signing up to volunteer). “It was an extremely powerful technique for [influencing] decisions,” he said. However, given the deep investment in time and resources needed for A/B testing, Koomen observed over time that companies tended to avoid using the technique – a fact that he and his business partner Dan Siroker quickly recognized as a major business opportunity.

The success of the 2008 campaign spoke for itself. Koomen estimated that methodical experimentation accounted for roughly $75 million more in donations to President Obama’s campaign and 4 million new website registrants. These results motivated Koomen and Siroker, both former Google product managers, to found Optimizely in 2009. They created a simple program that even small and medium-size businesses could utilize without having to depend on specialized in-house talent to run experiments. Organizations with limited resources could take advantage of marketing tactics that Amazon and other major blue-chip companies have been using for years to increase traffic and user conversion.

At BRITE ‘13, Koomen shared some best practices and lessons learned from running over 100,000 tests for clients and identifying the most effective approaches for achieving business objectives. For the Obama campaign, this entailed what Bloomberg BusinessWeek called “strange, incessant, and weirdly over familiar e-mails” due to the unusual, extremely casual tone Obama’s team usedjourney to office in 2008. The fundraising team found that the most successful subject heading “Hey” alone brought in millions of dollars in funding.

A few things that Koomen recommends businesses keep in mind as they take stock of their websites’ performance are:

• Define quantifiable success metrics. One of the most important parts of testing. As exemplified by the Obama campaign, Koomen states that the campaign staffers did a good job of attracting people to the official website, but turning the site’s visitors into subscribers had proved more challenging and converting email signups to paying donators even more so.  By tweaking the website to optimize those two KPIs – subscribers and payers – the new website outperformed the old version by about 40%.

• Explore before you refine. Koomen cautions against refining and optimizing in favor of exploring first to ensure you are aware of all potential solutions before selecting one to improve.  Otherwise, there is a chance the best solution will be missed.

• Less is more. Reducing optionality can have a major impact on a website’s effectiveness. Koomen cites a client which removed a series of links related to its product portfolio and company background from its shopping cart page and saw a 16% improvement in the dollars per visitor.

Watch Pete Koomen’s BRITE ’13 talk to learn more about how A/B Testing can drive greater communications effectiveness.

BY NANDITA RAY

Case Study: Developing a Culture to Run Marketing As a Business

As 2010 approached, SAP found itself in a critical position. The competition was evolving to be leaner and more targeted. Customers were becoming more knowledgeable, demanding and price sensitive. SAP’s image was quickly becoming outdated, and there was a growing rift between employees and upper management that threatened to pull the organization apart. A new case study by Matthew Quint of the Center on Global Brand Leadership, Run Marketing as a Business: The Transformation of SAP Marketing, is a two-part study that delves into how SAP’s leadership worked to reinvigorate the company and how SAP Marketing evolved into a department focused on culture and ROI-driven results.

The financial crisis of 2008 impacted SAP and other enterprise resource planning providers in three major ways: it sharply contracted IT investments, it changed the ways that companies evaluated and purchased ERP services, and spawned a powerful new competitor in Software-as-a-Service (SaaS). Spencer Osborn from Ogilvy notes, “…There is an increasing trend of ‘prosumer’ purchasing behavior in the business IT sector. Google, Apple, and others brought simplicity to the IT interface and professionals now expect the same for business software.” The business was becoming more and more complex, and traditional methods of product management and marketing were no longer applicable.

Amidst these business and consumer challenges, SAP’s Board brought in new co-CEO’s, Jim Hagemann Snabe and Bill McDermott, who quickly set ambitious goals for SAP: generate a revenue target of €20B, create an operating margin of 35%, and reach 1 billion people with SAP technology and services.

The CEO’s broke from tradition and focused on expanded SAP’s portfolio of offerings through mergers and acquisitions. Simultaneously, SAP Marketing developed a strategy to transform SAP’s image to a more innovative, dynamic and approachable company. In 2011, Jonathan Becher was appointed CMO, and immediately began driving a platform effectively combining both the art and science of marketing. He elaborated, “[I]… think ‘business first, marketing second.’ From that comes a mantra that marketing is a business, not just a division that supports a business.”

Becher recognized that SAP Marketing was excellent at communicating clear messages that grew brand awareness, but he wanted to institute an approach that would align his team to build a culture that supported a measurable SAP Marketing strategy with overall company goals. Thus, SAP Marketing developed 5 key “transformation pillars” to drive all future marketing activities. By using these pillars as guide posts, Becher and SAP Marketing set to update SAP’s image to match its new and expanded product portfolio. Key performance indicators (KPIs) were created to measure marketing outcomes, rather than marketing activities. To further align incentives and encourage staff members to work together, bonuses were tied to achieving the 10 collective KPI’s.

SAP Marketing’s changes led Paul Greenberg, a customer relationship management and technology author, to declare, “SAP has transformed their company from…a highly traditional, conservative, closed company, to an open innovative, accessible organization…”

Learn how SAP Marketing was able to revitalize SAP’s brand against target business metrics and read more about how they plan to continue moving forward in today’s dynamic business environment in the new case study.

Download Run Marketing as a Business Part 1 and Part II.

By Matthew Quint

What You Don’t Know About One-Night Stands

If you’re reading this, clearly I’ve caught your attention. I’m sorry to say this isn’t an article about one-night stands.

At some point in recent years, many of us have likely clicked on what we thought would be an interesting article only to discover that it was a paid advertisement in editorial guise. Content marketing is not a new concept, but it’s becoming an increasingly popular strategy for media companies and brands to team up on new ways to drive revenue. According to Pew Research Center, sponsored content increased by 56% in 2011 and is still on the rise.

Edelman’s Chief Content Officer Steve Rubel stresses that sponsored “content is no longer optional. It’s imperative.” At BRITE ’13 Rubel explains, “It’s hard now to amass large audiences the way you used to. And that means money problems for everyone.” He notes, however, that “out of economic disruption come great opportunities.” Rubel says that display advertising has become less lucrative in recent years, and can even drive down CPM. Content marketing, on the other hand, is a fraction of the cost with the potential for greater results.

Consider Wine Enthusiast magazine. Sure it’s a media company, but it’s also a brand. By incorporating custom content, Wine Enthusiast successfully increased site traffic by 154% and boosted monthly email opt-ins by 50%. Director of Internet Marketing Erika Strum tells MarketingSherpa:

We put time into creating… content that helps people either make a buying decision or entertains them. Even if they aren’t making that purchase in the moment, we feel that they will come back to us as a… source of information.

Rubel has identified three ways that brands are partnering with media companies—syndication, integration, and co-creation. These partnerships borrow from traditional marketing models like paid media and product placement, but they now overlap with owned and earned media as an additional driver of revenue.

• Syndication: Rubel describes this method as “advertorial reinvented.” Sometimes the sponsor scripts the content, sometimes the publisher assumes this role, and sometimes they work together to design content.
• Integration: Similar to syndication, integration stems from product placement. But rather than placing a product within eyeline (think Wayne’s World) the brand becomes part of the narrative (think Mad Men).
• Co-creation: The primary difference with co-creation is that the sponsor provides the funding, but the media company takes responsibility for the content. Rubel likens this to a sports stadium. Gillette bought the naming rights to the home stadium of the New England Patriots, but Kraft Sports Group, which owns and operates the venue, is responsible for the action on the field. Okay, okay, “action” may not be what non-New Englanders would call it. But you get the point.

While many media companies have embraced sponsored content, some are still resistant. Google for one refers to this as “commerce journalism” and explicitly states on its website:

Stick to the news–we mean it! Google News is not a marketing service…. [If] we find non-news content mixed with news content, we may exclude your entire publication from Google News.

As with anything, there are associated risks. It can offer control of content, data and measurement, and opportunities for innovation. But there is the potential for backlash. You may recall this past January The Atlantic issued an apology for posting a content piece from the Church of Scientology. Readers complained that it resembled a traditional editorial, not clearly identifying that it was a sponsored article. “We screwed up,” were the words of The Atlantic‘s media relations team.

Rubel emphasizes, though, that sponsored content isn’t going away, at least not any time soon. He advises businesses to adapt to this marketing model. “You have to put a content engine inside your company. If it’s not there already, you have to think about how to get it in there.”

What do you think?

Watch Rubel’s BRITE ’13 talk to learn more about the benefits, and the risks, of these new media-brand relationships.

By ALLIE ABODEELY