Speak & Spell: New Apps Turn Phones into Multimedia Search Appliances

About five years ago, in a previous life at another technology publication, I wrote that I wished I could “Google my sock drawer.” I was being facetious, but my point was that searching the Web had become so easy that it left me yearning for equally convenient ways to search other things, like the books in my local library, the stores in my neighborhood, the recordings in my CD or DVD collection, even the everyday stuff in my house.

Well, the idea of searching your sock drawer isn’t so tongue-in-cheek anymore. You still can’t ask Google to find the missing half of your favorite argyles—but you can use the new Amazon Mobile app to take a picture of your sock drawer, then have Amazon send you a link to a page where you can buy a matching pair online.

You can also use the popular Shazam app on the iPhone to capture a few seconds of a song playing on the radio, and find out instantly what it’s called, who recorded it, and where to buy it. You can use the Street View feature of the new-and-improved Google Maps application on the iPhone to take a virtual stroll down Boston’s Newbury Street and decide which stores you want to visit. Once you get there, you can use a location-aware app like Urbanspoon or Yelp to find interesting restaurants. And you don’t even have to type in your search terms anymore: Vlingo’s new iPhone app and the latest version of the Google Mobile app can work with spoken-word input just as easily.

My point is that the newest search-related applications, especially those for advanced wireless devices like the iPhone, are lending new meaning to the very concept of search. Finding entertaining media, useful products, and interesting places no longer requires a PC, a keyboard, a Web browser, or even a traditional search engine. On the query side, devices like the iPhone 3G have built-in cameras and microphones that let them capture unconventional types of input for a search, such as photos, spoken instructions, or snippets of music. They can also fill in key pieces of context on their own—for example, by grabbing your current location from the built-in GPS chip. On the output side, the devices can supply links, reviews, videos, maps, even walking directions. The end result—a new level of connectivity to the people, things, ideas, and places around you—is, to my mind, one of the best reasons to invest in a broadband-capable smartphone. (I admit to being an iPhone chauvinist, but similar experiences are available on other gadgets, such as the high-end Blackberry devices and the T-Mobile G1 phone.)

I’ve been playing around lately with three mobile search applications in particular. Each one illustrates different strengths of the mobile platform. And together, they’ve brought me full circle, to the point where I wish that conventional desktop or laptop-based search tools had some of the same capabilities as these mobile marvels.

Google Mobile App on the iPhoneThe first is the new Google Mobile app on the iPhone, released November 14. The app has two functions. It’s a convenient portal to the browser-based versions of many of Google’s other Web services—Gmail, Google Calendar, Google Docs, Google Talk, Google Reader, et cetera. But it’s also a freestanding search engine, with a compelling new keyboard-free “voice search” option. All you have to do is lift the phone to your ear, as if you were making a phone call; the iPhone’s accelerometer takes that as the cue to start listening for a spoken query, like “Quantum of Solace Boston showtimes.” Take the phone away from your ear, and the software sends your voice snippet to Google for processing; within seconds, the search results show up on screen. There’s a fun, Star Trek quality to the whole operation, except that the phone doesn’t talk back. (Maybe that’s the next improvement Google will roll out.)

Second, the new iPhone app from Cambridge, MA-based Vlingo, which came out December 3, also lets you initiate Google searches by speaking. With Vlingo, you have to tap the “Press + Speak” button to start the process, rather than holding the phone up to your ear, which is an annoyance, once you’ve gotten used to the Google method. But the Vlingo app does do several cool things that the Google app doesn’t. For example, you can speak an address or business name and see the location on a Google map, automatically call anyone in your contact list by speaking their name, or dictate a status update for your Facebook or Twitter account. The Blackberry version of Vlingo’s speech-recognition app, which has been out since June, goes even further, letting users dictate e-mail and text messages. I fully expect to see Vlingo’s engineers add such features to their iPhone app.

Vlingo iPhone AppBoth the Google Mobile app and the Vlingo app put an end to typing out search queries. But despite my general enthusiasm for these new voice-driven mobile search tools, I have to say that their speech-recognition algorithms still need work. Converting speech to text seems to be one of those problems, like building a foolproof A/V system for lecture halls, that experts are still going to be working on 30 years from now.

When I tried to speak the search term “Rahm Emanuel” into the Google Mobile app, the application came back with the transcriptions “roman manual,” then “robin manual,” then “brahmin manual.” (Thankfully, however, it got “Barack Obama” right the first time.) It fared a little better with “Xconomy,” one of the words I always like to use to torture speech-recognition systems, coming back first with “astronomy” and “taxonomy” and finally getting “Xconomy” on the third try. The Vlingo app got “Xconomy” right the first time—but I have a hunch that’s because the folks there knew I was evaluating the software.

The third mobile search tool I’ve been enjoying recently involves videos rather than voice. It’s WikiTap, an iPhone app released in September by Veveo, an Andover, MA-based startup I’ve covered several times. The app is a mobile-friendly mashup of Wikipedia and YouTube. Those two information sources might, at first blush, seem to blend about as seamlessly as Charlie Rose and Paris Hilton. But as it turns out, they go together remarkably well.

Veveo\'s WikiTap iPhone AppVeveo’s first mobile product was an “incremental search” tool called vTap, designed to make it easier to find the video you want on a mobile phone by narrowing down the list of possible matches as you type. WikiTap works the same way. To find the Wikipedia listing for my favorite film-score composer, Bernard Herrmann, I only had to enter “bernard h” and Herrmann popped up as the top match. But here’s the really cool thing about WikiTap: as soon as you click on a search result, the program brings up both the Wikipedia listing and related videos culled from YouTube and other sources. The videos presented alongside the Bernard Herrmann article, for example, included a YouTube slide show featuring Kim Novak, star of Vertigo, one of the many Hitchcock films Herrmann scored, and was followed by a video on the top 15 horror film themes of all time (Herrman’s music for Psycho, of course, topped the list).

I’ve found that leaping back and forth between Wikipedia text articles and related YouTube videos is a surprisingly fun way to kill a few hours. The pairing seems so natural that I now feel like there’s something missing when I visit Wikipedia on the conventional Web. There’s nothing new about the concept of multimedia reference works, of course—encyclopedia publishers like Britannica have been publishing CD-ROM and DVD-ROM versions of their content, spiced up with a few QuickTime videos, since the mid-1990s. But in these older works, the videos (which usually turned out to be clips from recycled 1960s educational documentaries) always felt to me like an afterthought—they were there more because the platform could support them than because anyone thought they were essential. WikiTap, like Wikipedia itself, is a gleefully crowdsourced hodgepodge where you never know quite what you’re going to find. The whole point is to make unexpected connections, and to see and hear things that you can’t understand just by reading about them.

And that’s the fun of the new mobile search applications in general: they lead you into experiences you never would have had otherwise. Which is part of the reason I’m still feeling like the money I put down for my iPhone 3G is the best $299 I ever spent.

Wade Roush is the producer and host of the podcast Soonish and a contributing editor at Xconomy. Follow @soonishpodcast

Trending on Xconomy