UTest Tests Its Testers in Payday Snafu

5/25/10Follow @wroush

Creating Web-based marketplaces for “crowdsourcing”—farming out digital piece-work to freelancers around the world—has been a hot business idea for the last half-decade. IStockphoto does it for stock images, TopCoder does it for enterprise software, Innocentive does it as a supplement to corporate R&D, and uTest does it for software quality assurance testing. But when push comes to shove, how well do these companies know and trust their networks of contributors, most of whom they’ve never met? A couple of weeks ago Boston-based uTest found itself at the mercy of its own crowd—and emerged with an encouraging story to tell.

At uTest, 25,000 software testers in more than 160 countries log on over the Web to do on-demand testing of software applications, helping the applications’ makers catch bugs and glitches, troubleshoot usability problems, and simulate performance under realistic loads. Twice a month, uTest pays its active testers via Paypal or Payoneer (a New York-based network that delivers payments using prepaid Mastercard cards).

According to Matt Johnston, uTest’s vice president of marketing and community, uTest ran into a glitch of its own the evening of Saturday, May 15, as it disbursed payments for the first half of May. It transferred funds to Paypal and Payoneer twice—meaning that all testers who had done any work in early May received double their usual fee.

“I won’t go into specific numbers, but it was a non-trivial amount of money,” Johnston tells Xconomy. (He also wrote about the episode on uTest’s blog last week.) “We’re talking well into the five figures”—way more than the venture-funded startup could afford to lose on a bookkeeping error.

But right away, uTest started to benefit from an unexpected side effect of being in the software-testing business. It wasn’t PayPal or Payoneer who detected the problem, but the testers themselves. And being testers, they sent in bug reports.

“If you saw an extra $200 sitting in your bank statement, you might go, ‘Huh,’” says Johnston. “But software testers, they say ‘Oh, this is an interesting defect.’ They are wired to point out flaws. The first reports we got were not just ‘Hey, you paid me twice’—they were, ‘It looks like I’ve got two separate transaction IDs for the exact same amount, seven seconds apart.’ They were diagnosing it for us, like it was just another software defect.”

This was late Saturday night, Johnston says. The company had to decide quickly what to do about the problem. PayPal and Payoneer politely informed uTest that there was nothing they could do—there’s no such thing as an “undo” button for electronic payment (for good reason—the idea of PayPal debiting your bank account without your consent is more than a little scary). “We ultimately came to the conclusion that we had to be really transparent about it, and tell our community exactly what happened and what we would like them to do,” says Johnston.

Returning the overpayments turned out to involve a tricky six-step process, and Johnston says he wasn’t sure how many testers would go to the trouble. “I have a great deal of trust and respect for our community, but this took it out of the theoretical realm,” he says. “I knew we were going to find out really quickly, when it comes to communities and crowdsourcing, whether this bucket holds water or not.”

But Johnston was surprised to find that testers pursued the refund process as zealously as if they’d discovered a bug in their own software. Within 24 hours of sending out an e-mail notice to the affected testers, Johnston says, uTest had gotten back two-thirds of its money. Within three days, 90 percent of the money had filtered back. And as of my conversation with Johnston Monday morning, only 0.5 percent of the funds remained to be recovered. “We don’t even look at that and say that the last half a percent are the dishonest ones,” he says. “They’re probably just less engaged and haven’t opened their e-mail yet.”

Of course, uTest’s testers weren’t acting out of pure altruism. It wasn’t hard to reason that if they tried to keep the overpayments—more than $2,000, in some cases—uTest would know it, and that there would eventually be repercussions.

“I’m not saying that we’ve found the last 25,000 honest people on the planet,” says Johnston. “But all too often, you hear about online community members bullying each other or committing fraud. This was a prime opportunity to do one of those things, since the mistake had already been made on our part. But many of them made a financial choice that staying in good graces with uTest was more financially rewarding in the long run than a quick buck.”

In the end, Johnston says, the payment problem turned out to be a brilliant, if unplanned, test of the company’s efforts to build a coherent community around its testing platform. “We like to talk internally about the difference between communities versus crowds versus mobs, and we like to think we’re building a community here,” he says. “That’s all great as a bumper sticker—but this put it to the test in an unintended way.”

And maybe, in the end, nice guys don’t finish last. Johnston says plans are in the works to reward the testers who jumped on the problem fastest. “The specifics are to be determined, but I would like to be able to do something for the community at large, and specifically for the people who first pointed the problem out to us or refunded the money most promptly,” he says. “There will be some form of thanks.”

Wade Roush is a contributing editor at Xconomy. Follow @wroush

By posting a comment, you agree to our terms and conditions.