TopCoder—Crowdsourcing Software Long Before Crowdsourcing Got Cool

4/23/09Follow @wroush

Can competitions and prizes get you to the Moon? Google thinks so—it’s backing the $30 million Lunar X Prize, which will be awarded to the first privately funded team that sends a remote-controlled robot to the Moon, drives it 500 meters, and collects video of the trip. Back here on Earth, the $10 million Archon X Prize is being offered to the first team that can build a device that sequences 100 human genomes in 10 days or less, and the Wellpoint Foundation is proposing a $10 million Healthcare X Prize for the first organization that figures out how to deliver a 50 percent improvement in the cost-effectiveness of community healthcare over a three-year period. Right here in the Boston area, Waltham, MA-based InnoCentive is using the prize model to attract solutions for dozens of problems, ranging from improving the fire resistance of polyurethane foam to accelerating the growth of soybean shoots.

But in Glastonbury, CT, there’s a company called TopCoder with a prize-based business model that predates all of these efforts. It’s using the model to create products that are arguably more relevant to our economy in the short term—better software applications. And it’s doing it for far less money; first-place winners rarely take home more than $3,000.

Companies like AOL, ESPN, Ameriprise, Ferguson, Geico, and LendingTree have outsourced thousands of software development projects to TopCoder’s worldwide freelance community—”from something as simple as a Web page all the way up to full-blown enterprise resource planning (ERP) systems,” in the words of company founder and chairman Jack Hughes.

The model is so successful that it’s attracting the attention of business scholars from Harvard, MIT, and other academic centers. “They’re creating enterprise-class software projects in a highly distributed setting, and for me that was an order of complexity that I wasn’t expecting,” says Karim Lakhani, an assistant professor at Harvard Business School who has written two papers about TopCoder.

You might wonder why the competition model attracts any willing participants, considering that competing takes a lot of work, and that the big money is usually reserved for the top-placing teams, while everyone else gets zilch. Indeed, I’ve always wondered whether giant prize programs like the X Prizes are an efficient way for a society to array its resources—just look at the 26 organizations that collectively invested more than $100 million competing for the $10 million Ansari X Prize. All but one of the teams walked away with nothing after Burt Rutan’s SpaceShipOne won the prize in 2004.

But TopCoder has invented a system that seems to work, both for its clients and for its community members. For one thing, the company never bought into the winner-take-all concept. For each competition, the company sets the dollar amount of the first-prize award based on the size of the job and the amount the client is willing to pay. TopCoder sets aside about half as much for the second-prize winner. The third-, fourth-, and fifth-place winners don’t get cash, but they do get points—and everyone gets paid later out of a general prize pot based on how many points they’ve accumulated.

“We wanted to avoid the problem of people submitting and always coming in third and never getting paid,” says Hughes. While it’s standard practice elsewhere in the software business to offshore software development to the cheapest available labor pool, “We just weren’t interested in that,” says Hughes. “We were more interested in finding really good people, and to do that you have to enable them to make a decent amount of money.”

TopCoder’s story goes back to the late 1990s, when Hughes was leading Tallán, a Connecticut-based developer of e-business software. “Most of the people at Tallán were very competitive, but in a joy-of-programming sense,” says Hughes. “We’d be working on a long project, and just to blow off steam, we would try to code something unrelated and see who could do it better or faster.” CMGI acquired Tallán in 2000, and after Hughes left, he decided to start a new company to see whether the competition model could energize programmers on a broader scale.

For the first couple of years—in observance of a non-compete agreement with CMGI—TopCoder restricted itself to running non-commercial, college-based software competitions. “But at the end of the two years, we said, ‘Let’s figure out if we can actually build real software using this model,’” Hughes says. “That launched us down a long path of figuring out how the software development life cycle could be boiled down into a series of competitions.”

The privately funded company is capable of handing over a large, multi-layered, enterprise-class software system to a client—but it never asks its community of 190,000 programmers and counting to take on a giant problem like this all at once. Instead, it breaks down each project into phases and holds separate competitions for each phase, using the winning submissions from the last phase as the starting material for the next.

Because many clients come to TopCoder without a clear idea of what they want, according to Hughes, the first step in the process is often a “conceptualization” and “specification” contest that helps clients organize their ideas and produce high-level requirements documents. (If clients need help roughing out basic designs or visualizing the look and feel of their applications, there’s a TopCoder Studio area where graphical and user-interface designers can join contests.)

Once the specifications are fixed, the basic components a software system needs are identified through architecture competitions. Then there are more competitions to design and develop each component.

“The modularity of the problems is a hugely important thing,” Harvard’s Lakhani says. “Only an IBM or an Accenture or a WIPRO could come in and try to create an entire ERP system. But if you break it down into small chunks, and create mini-contests around all of these problems, then you can lower the barrier to participating, so individuals or small teams can do it.”

Another part of the “secret sauce” at TopCoder, Lakhani says, is in the way the company integrates the chunks into finished systems. Not surprisingly for a company that takes competition so seriously, this part is largely outsourced as well, through software assembly contests. The company even handles the quality-assurance phase of software development, through application testing contests and so-called “bug races” (literal races to fix small software flaws, with the winners getting $25 to $100).

The general philosophy at TopCoder, according to Hughes, is that “the community is a lot smarter than we are. So what we tend to do is say, ‘Look, here’s what we’re trying to do. What’s the best way to do this?’ What works for them is what works for us.”

That doesn’t mean TopCoder can evade responsibility for software that doesn’t work. Every client project has a platform manager who’s responsible for shepherding the work along, and “ultimately, if a project doesn’t work, it’s going to come down to TopCoder to fix it,” says Hughes. But the fixes, too, are farmed out to the community.

Just who are these contestants? They’re mostly individuals—although occasionally, the company receives submissions from teams, and there are even a few small software companies that make a living using the TopCoder platform, Hughes says. About half of the people who join Top Coder are from North America, while Indian, Chinese, and Eastern European programmers make up the other half. Lately, the community has been growing at about 100 members per day.

Hughes says community interest in TopCoder always picks up in the spring, as the company’s annual TopCoder Open tournament builds to a head—the finals are coming up on June 4 in Las Vegas. Last year’s first-prize winner took home $60,000; the third-prize winner, Polish programmer Tomek Czajka, has earned more than $130,000 from TopCoder over the last five years, and parlayed his record into a job at Google. Indeed, the TopCoder Open attracts so many smart programmers from around the world that this year’s tournament is being sponsored by the National Security Agency.

Lakhani says that from an economist’s perspective, contests like TopCoder’s contests and the X Prize competitions shouldn’t really work—they’re just not efficient, from the competitors’ point of view. “In most competitions, people over-invest, and spend way more money than they can make back,” he says.

But what makes TopCoder and the other prize-driven communities function, he says, is something larger. “Human beings are not just pecuniary animals,” he says. “They care about a lot of other things,” such as status, the opportunity to test themselves against others and learn from their competitors, and just plain fun.

And from a client’s perspective, says Lakhani, the TopCoder model is extremely efficient. “Oftentimes we don’t know who the best solvers are. We can’t a priori decide who has the best answer. So the competition approach is often better, because you can just broadcast your problem to everybody, and see who thinks they can solve it for you. And you, as the problem holder, don’t have to make a bet—you pay only on completion, which takes all the risk out.”

There may be certain areas of engineering where it would be inadvisable to employ the modular, competition-based approach; I’m not sure I’d want to drive a car or fly on a jumbo jet that was designed through a TopCoder-like competition, for example. But many of us are already browsing websites and using enterprise and e-commerce software that were built this way—and not noticing the difference. Of course, it helps that TopCoder has elaborate source-code control procedures and online review systems to make sure its software performs as designed.

Recently the company introduced a beta version of a service called TopCoder Direct that allows anyone to tap directly into the company’s Web-based platform for mounting software competitions (which was itself designed and built through—you guessed it—-competitions). In other words, businesses can now launch competitions and interact with TopCoder community members directly, without become formal clients of TopCoder.

So could TopCoder eventually find itself outcompeted by its own community? “To a certain extent, we may be putting ourselves out of a job,” Hughes acknowledges. “But that probably won’t be a bad thing.”

Wade Roush is a contributing editor at Xconomy. Follow @wroush

By posting a comment, you agree to our terms and conditions.