Bing Is the Buggiest (But Second-Best) Search Engine, Say Software Testers in uTest Report

9/14/09Follow @wroush

Tomorrow, Boston-based uTest, a startup that crowdsources software quality-assurance projects to a global community of 19,000 freelance testers, is expected to announce the official results of its first “Search Engine Bug Battle.” Xconomy got an early look at the data today, and the folks in Redmond and Bellevue, WA, may not be happy about the results: uTest community members found 321 distinct bugs in Bing, Microsoft’s new search engine, more than the three other tested engines combined.

On the up side for Microsoft, part of the reason Bing fared so poorly compared to the other three engines that testers examined—Google, Yahoo, and Google’s experimental new search engine, Caffeine—may be that Bing is new and high-profile, meaning that testers probed it with extra zeal. Some 85 percent of the 1,100 uTest members who entered the contest chose to make Bing one of the engines they tested. (The same fraction tested Google.) And after testing Bing, 10 percent of the contestants told uTest they liked Bing so much they would make it their default search engine.

Another mitigating factor: Bing “has the youngest code base, compared to Google or Yahoo, so you’d expect to see more bugs,” Matt Johnston, uTest’s vice president of marketing and community, points out. “They’re also introducing features that are new to the search space, including multimedia features, visual search, interactive roll-over features.”

In other words, Bing has more stuff for aggressive software testers to break. Just today, for example, Microsoft unveiled a new “visual search” feature at Bing: users can now initiate and refine their searches by clicking on images, at least in specific categories such as “U.S. politicians.”

uTest’s bug battles are quarterly contests in which the company’s freelance members compete to discover bugs in popular software applications. uTest hands out cash prizes to the testers who find the most. It’s the fourth such battle for uTest, and the first one to focus on search engines. (Previous battles focused on Web browsers, social networking sites, and Twitter applications.)

uTest members not only found more bugs in Bing—321, compared to 130 for Google—but those they did find were more urgent. Contestants described 60 percent of the Bing bugs as either “high severity” or “showstoppers”—those in need of immediate attention—whereas only 8 percent of the Google bugs were showstoppers. A similar percentage of Yahoo’s 70 bugs were classified as showstoppers (10 percent).

The company followed up on the bug competition by surveying members about their search engine experiences. Alas, Bing suffered here too. Asked what quality is most important in a search engine, 71 percent of the uTest contestants answered “overall accuracy of search results.” In this category, Google again came out on top, with 90 percent of testers rating it excellent or good. Google Caffeine came in second, with 83 percent rating it this highly, followed by Yahoo with 53 percent and Bing with 42 percent.

But when asked to rank the three major search engines in terms of their overall accuracy, the testers put Bing in second place, ahead of Yahoo. And when it came to “best real-time relevance” and “fastest page load speed,” the testers ranked the three engines in the same order (Google, Bing, Yahoo).

I asked Johnston exactly what a uTest community member might consider a “bug” when it comes to a public-facing Web application like a search engine. He explained that uTest testers classify bugs as technical, functional, or user-interface-related. A user-interface bug might simply be a link that was supposed to be blue that came out purple. A functional bug might be an unexpected behavior—like being sent to the wrong page. A technical bug would be a more serious problem like a browser crash.

Interestingly, Johnston said that while the search engine testers found more bugs than in past battles, the bugs “skewed toward the lower end of the severity scale,” meaning that for the most part, all of the search engines work well. Johnston says the testers did not find a single bug relating to security or privacy—for example, mixed-up accounts or preferences settings.

And it’s important to remember, Johnston said, that uTest testers aren’t your average Web surfers. “When the average person is using a search engine, you’re trying to get it to work,” he points out. “When you’re testing it, you’re trying to get it to break.”

The bug battles are “an excellent community engagement tool” for uTest, says Johnston. “Certainly, the prize money matters, but I think the bragging rights are almost as important as the money. It’s an extension of the uTest reputation system. And it’s a great chance for new testers to break in and show us what they can do.”

Not incidentally, Johnston says the bug battles demonstrate the power of uTest’s crowdsourcing model. “It’s a real-world demonstration of what the community can do. We think it’s fascinating to see how quickly you can mobilize a community of professional testers and point them at an application of any type and bring that much coverage to bear.”

After past contests, Johnston says, the vendors whose applications were tested— Microsoft, Mozilla, and Google in the case of the browser battle, and LinkedIn, MySpace, and Facebook, in the case of the social-networking battle—have often reached out to uTest to ask for the detailed results, so that they could address the bugs directly.

“They took it in the spirit in which it was intended,” Johnston says. “We’re not picking on these companies or their products. We are vendor-neutral, and we think this is as honest a real-world comparison as we can set up.” He says uTest would be happy to share the results of the search engine battle with Microsoft, Yahoo, and Google.

Wade Roush is a contributing editor at Xconomy. Follow @wroush

By posting a comment, you agree to our terms and conditions.