Hunting HiPPOs: Optimizely’s Testing Tools Bring Data-Driven Web Design to the Masses
Optimizely doesn’t put its most effective sales pitch on its website; it doesn’t have to. It boils down to this: If it was good enough for Barack Obama, it’s good enough for you.
“It,” in this case, is online A/B testing: the practice of altering live websites in small, controlled ways to see whether the alterations lead to higher response rates in the form of donations, purchases, newsletter signups, and the like. After Dan Siroker gave up a product manager position at Google to be director of analytics for the Obama presidential campaign during two stretches in 2007 and 2008, he became one of the first people to try A/B testing in the world of online campaign fundraising. His team identified design tweaks for the campaign website that ultimately helped Obama raise nearly half a billion dollars, much of it in the form of online donations of $200 or less.
After the campaign, Siroker left politics and joined up with another former Google product manager named Pete Koomen to build an online math game for elementary school kids. But the pair ultimately drifted back to A/B testing, convinced they could build better commercial testing tools than ones that had been available to the Obama campaign. That’s the concept they developed during the Winter 2010 term at the Y Combinator venture incubator in Mountain View, CA, and that’s the product that Optimizely is now providing to more than 5,000 registered users.
The fundamental philosophy behind A/B testing is that decisions should be based on data, not just instinct. Koomen jokes that when it comes to crucial choices about Web design, many organizations seem to guided by HiPPO—the Highest Paid Person’s Opinion. “Our overall mission is to make it much easier for companies to make data-driven decisions, because they are almost always better than HiPPO,” he says.
If that sounds like a Google-esque statement, that’s because Koomen and Siroker both learned their stuff at the search and advertising giant, working on products like Chrome, AdWords, and App Engine. “Google has this philosophy that no matter whether you are an engineer or a vice president, at the end of the day a decision comes down to ‘What does the data say?'” Siroker says. “When I came to the [Obama] campaign as an outsider, I was able to say, ‘Maybe I don’t know the best tagline or strategy to convince somebody to vote for him, but I can show you the data we collected in an experiment we did, and here is the proof that this is the decision we should make.'”
Of course, the stakes for most of Optimizely’s customers are somewhat lower than winning the White House. Most publishers and e-retailers use Optimizely’s Web-based A/B testing system to change elements on their Web pages—moving a “buy” button, for example, or changing its size or color—and then divert part of their incoming Web traffic to the alternative page. Optimizely’s back end tracks user behavior such as clicks or purchases and can measure whether the changes are producing the desired effect.
One attraction of Optimizely’s technology is that almost anyone in an organization can use it. As with Bo.lt, another marketing technology company I profiled recently, changing the look of a page on Optimizely and publishing an alternate version requires absolutely no programming, design, or database skills. “We spoke to a lot of companies and the majority of them are not doing A/B testing, even though it’s a tremendously valuable thing to be doing,” says Koomen. “They are not testing because it’s so difficult to do. Our idea was to remove all the unnecessary steps that we possibly could.”
Simplifying A/B testing software was an idea born on the campaign trail. “I spent the summer of 2007 on leave from Google, sleeping on the floor in the office in Chicago while everyone else was in Iowa,” Siroker recounts. “When I showed up they had a guy doing blogging and a guy doing e-mail, and a ton of people coming to the website. But they didn’t know what they should be looking at. Joe Rospars, the director of new media, thought the biggest opportunity was to tap into that data. We started with one simple experiment on the splash page that generated a substantial increase in e-mail signups, and in the end it meant big dollars for the campaign.”
But those tests employed two commercial, off-the-shelf A/B testing tools, Google Website Optimizer and Omniture’s Test and Target (now Adobe Test & Target—originally the creation of Jamie and Matthew Roche, who went on to start Bo.lt). Neither was developed with non-programmers in mind. “Both had huge up-front costs in understanding how to use the product,” Siroker says. “Both required you to tag different parts of a page [before testing it], and put a block of script above and below those parts. To run the experiment you had to commit code and go through QA [quality assurance]. The process was what hampered us. We had a hundred brilliant copywriters on the campaign, but we were bottlenecked on engineers to run the experiments.”
That pain provided the initial spark for Optimizely. But Siroker says that the campaign left him feeling a little burned out on Web analytics—so the spark lay dormant for a while.
After the campaign and the transition, Siroker returned to San Francisco, where he and Koomen built an education startup called CarrotSticks. The site offers games where first- through fifth-graders can compete with peers to finish math problems. It’s still online today, but more as a hobby than a business; Koomen says he and Siroker quickly realized that “it’s tremendously difficult to make money in education. Schools are broke, and parents are difficult to reach. The product ended up spreading quite well, and it pays its own server bills, but we decided that there were other ideas that were a lot bigger.”
The story doesn’t come back to A/B testing quite yet. When Koomen and Siroker decided to apply to the Winter 2010 term at Y Combinator, they pitched an idea for a social commerce startup they called Spreadly. The idea was to give people incentives to tweet or post Facebook status updates about purchases they’d just made—tickets to see Justin Bieber, for example—as a tool for marketers. But after getting into YC and spending a month putting together a prototype, the pair quickly saw that “it’s really hard to pay a person to broadcast something to their friends,” Koomen says. “If you think of it in terms of social capital, the amount they are spending by sending something out to their friends is almost never going to match up to the amount you are offering them.”
That’s when Spreadly finally gave way to Optimizely. “At the end of the day, I enjoy data analysis, and that’s something I only realized once I got out of it,” Siroker says. “I was more burned out from the sedentary lifestyle [of the campaign], and no sleep, and sitting in a room for months eating deep dish. But now for the first time, we were working on something that I would want—I was scratching my own itch. Neither of us were parents”—or Justin Bieber fans, for that matter—“but I was the guy who could have used this technology back in the campaign.”
If you’re having trouble visualizing all that, I recommend watching Optimizely’s 4-minute introductory video. Suffice it to say that there’s enough money at stake in the worlds of e-commerce and online marketing to make a subscription to Optimizely worthwhile for many customers. The service ranges from $17 per month to $359 per month or more, depending on the number of visitors being “bucketed” into Optimizely’s tests.
The startup’s first big client, though, was actually a pro bono account: the Clinton Bush Haiti Fund, an initiative to support rebuilding in Haiti after the devastating January 2010 earthquake. “Their goal was about getting people who come to the site to donate more money,” says Siroker. “We did several variations. At the bottom of the donation page, instead of saying ‘Submit,’ we changed the button to say ‘Support Haiti,’ or we increased the font size or the layout or included images of the victims. The net impact was an improvement of about 10 percent on the dollars per page view. By the time the experiments were done, they had raised $11 million, about $1 million of which could pretty directly be tied to these improvements.”
That’s the power of data for you. Of course, there are limits to A/B testing. For one thing, the end result of a test will only be as good as the elements you were testing in the first place. “I don’t think you can start with a blank page and A/B-test your way to success,” says Koomen. “It requires creativity and initiative.”
Google itself has often been criticized for taking data-driven decision making to extremes; one visual designer for the company famously quit in 2009 after tiring of a culture where it was standard procedure to test 41 shades of blue for the Google toolbar to see which performed best. And here’s a telling fact—the Obama White House isn’t using A/B testing tools on the whitehouse.gov website. “Using technology to run a campaign is very different from using technology to govern,” says Siroker. “One challenge [with whitehouse.gov] is figuring out what to optimize for. Do you want people to watch this video, or share something on Facebook?” (Or maybe just learn something about how their government works?)
Online businesses and Web designers are closer to Optimizely’s sweet spot, and Koomen says the company has found enough users that it’s been profitable (or at least ramen profitable) since before it exited Y Combinator. The startup has raised $1.2 million in angel backing from the usual suspects, including Sam Altman, Paul Buchheit, Ron Conway, Chris Dixon, Steve Huffman, Nils Johnson, Mitch Kapor, Ashton Kutcher, Ariel Poler, Naval Ravikant, Aydin Senkut, Ram Shriram, Joshua Schachter, and Brian Sugar. It’s now got nine employees, all working from the startup’s stylish loft adjacent to San Francisco’s South Park startup haven, and there’s enough revenue coming in the door that the company doesn’t need to think about raising more investment, Koomen says. The startup’s focus right now is in “customer service and retention, learning our sales cycle, and building out the product,” he says.
Then there’s the 2012 presidential election coming up, when, it’s safe to say, online fundraising and A/B testing will be in even wider use. “I noticed that for a lot of campaigns [in 2008], their optimization strategy was just to copy what Obama did,” says Siroker. “Candidates always like to do the thing that worked last time. Plus, if you really want to push the limit and improve on what’s there [on a website], the only way you can do that is optimization.” So don’t be surprised if the Donate Now button on your favorite candidate’s website is in a different spot, or a different shade of blue, every time you visit. It just means you’ve been bucketed.
Here’s Optimizely’s introductory video.