Qualcomm Claims Leadership in Augmented Reality, Sees Huge Potential on Its View Screen

6/21/11Follow @bvbigelow

I’ve never been particularly fond of the term “killer app,” which has thankfully receded from the lexicon of tech writers—no doubt from heavy overuse.

But during Qualcomm’s Uplinq conference for mobile app developers earlier this month, I was struck by the potential “killerness” of the wireless giant’s initiative in mobile augmented reality, or AR. What initially seemed like an amusing kind of virtual curiosity last year (when Qualcomm and Mattel demonstrated an AR version of the game “Rock ‘Em Sock ‘Em Robots”) struck me this year as a far more pragmatic and relevant technology—with a broad range of potential applications. What Augmented Reality really represents is a potential revolution in the mobile user interface—by simply aiming a camera-equipped mobile device towards an object (or anything, really) and seeing a layer of relevant data, images, or apps superimposed over the real world.

The combination of AR software and hardware that Qualcomm (NASDAQ: QCOM) has developed makes it possible to overlay 3-D virtual images and video content on top of the real world, as viewed through the camera of a smartphone or tablet computer. While there are some AR mapping technologies that use GPS and internal compass inputs to provide virtual labels (visible in the field of view) for shops along a street, say, Qualcomm has focused its R&D efforts on a different approach, called vision-based AR.

Jay Wright

“It’s a different set of technologies that require computer vision technologies that are capable of recognizing things in the field of view,” says Jay Wright, Qualcomm’s senior director for AR business development. While Qualcomm’s technology lies mostly in its proprietary software, Wright says the company is also integrating its AR software closely with its mobile chipsets.

Wright says the AR field traces its roots back to the aircraft maintenance business, as part of an effort to make the complexity of aircraft maintenance and repairs easier for mechanics. The idea was to devise a see-through display that would be head-mounted, so mechanics could superimpose the manufacturer’s schematic for a wiring harness while peering inside a panel at the real thing. “It makes a lot of sense,” Wright says, “because wires need to go in the right place inside of airplanes.”

Qualcomm is now working to make its AR technology the preferred software worldwide, in the tradition of Facebook, Google, and Microsoft.  “What we are doing commercially is we are taking this software technology for vision-based augmented reality and we are making it broadly available [for free] to all Android developers, and we also just announced that we’ll be making it available to iOS developers as well,” Wright says.

Because vision-based AR apps are very computationally intensive, Wright says, they take a lot of processing power. So Qualcomm has optimized its AR software to perform especially well on the company’s higher-end Snapdragon chipsets. By making its software widely available to app developers (Wright says more than 7,000 so far), Qualcomm is both driving demand for cool new AR apps and differentiating the performance of its chips in the marketplace, he says.

In the global competition for making wireless chips, which includes Nvidia, Texas Instruments, Samsung, Apple, and Intel, Wright says, “We are not aware of any other silicon provider that has a comparable offering. This is really an area where Qualcomm invested early.”

Of course, Google, Microsoft, and a number of other tech companies have related efforts in AR software. A number of others such as metaio, based in Munich, Germany, and San Francisco; Amsterdam-based Layar, and Salzburg, Austria’s Wikitude also have been developing mobile AR browsers.

“This is really at the leading edge of innovation for us,” Wright adds. “We’re taking a leadership position by developing a new technology that we’re bringing to market, and that app developers can take and develop a whole new set of applications.”

What kind of applications? Wright listed five categories where Qualcomm has been encouraging development:

Gaming and Play: As a vision-based technology, AR needs to recognize an image with fixed grid markers so it can orient the virtual imagery in space. As a result, Wright says developers can take Qualcomm’s AR technology and make just about any 3-D game developed for computers or video consoles play virtually on an ordinary game board, wall, cereal box, or other printed surface. The technology, Wright explains, “knows about a bunch of tiny feature points that are defined by high variance in contrast, and it’s the collection and distribution of those points that uniquely identifies this image. So then, when the application starts looking for images, it uses that same process to try to find all those little points, and it compares them to all the little points that the application knows about, and says ‘Aha!’ I’ve found it.”

Qualcomm sees opportunities for a variety of games, from tactile manipulations such as a virtual Rubik’s Cubic to AR versions of FarmVille and other popular Facebook games.

To stimulate the market, Qualcomm launched an Augmented Reality Developer Challenge last year, offering a $125,000 first prize, $50,000 second prize, and $25,000 third prize for third-party app developers to create new games, educational applications, and other concepts. When the deadline arrived three months later, Wright said Qualcomm had received 54 submissions from 22 countries. (The top prize went to “Paparazzi,” a game in which players use their smartphone camera to try to snap photos of an animated virtual celebrity, who bears a resemblance to actor Sean Penn. As the game proceeds, the character gets increasingly irate, and tries to “break” the player’s camera.)

By using reference points taken from any image, Wright says AR also can be used to enhance the play of game pieces on a conventional game board, like Monopoly, or to create virtual residents inside a dollhouse or a model built of Legos.

Media and Advertising: Some of the most powerful applications of AR appear to lie in creating an entirely new form of interactive advertising, in which magazine ads, store displays, and product packaging comes to life for users looking through their smartphone cameras. Wright demonstrated a women’s clothing ad that enables the user to change the colors of the model’s ensemble, zoom in for close-ups, and get a 360-degree-view of the product. In another example, aiming an AR-capable smartphone at a DVD package for the film “Kung Fu Panda” enables the viewer to watch a 3-D trailer for the sequel “Kung Fu Panda 2″ in theaters now.

“What it’s doing is turning a physical surface, which can be product packaging or print advertising, into a stage,” Wright says. “When we show this to marketing agencies, they get extremely excited by the potential.”

At the AR Summit in London last week, AcrossAir, an AR developer based in London and New York, showed off its AR-based ads for Condé Nast Traveler, Europe’s UEFA soccer organization, and beermaker Stella Artois, according to The Guardian.

Educational and Instructional: Qualcomm also sees opportunities for AR applications that revisit the idea of virtual schematics for aircraft maintenance—making the smartphone a see-through user manual for how to use something, how to assemble a new product, or how to fix it. Such applications could become a godsend for the technophobic by overlaying simple user instructions on the viewfinder image on top of the often complex, confusing, or intimidating controls of consumer electronics products, from a cable TV set top box to the dashboard of a contemporary luxury sedan.

By aiming an AR-enabled smartphone at the workplace speakerphone, for example, Wright says, “Up comes the user manual and I say ‘conference call,’ and it draws arrows on the buttons I need to push, and the order that I need to push them. That makes a ton of sense.”

Facial Recognition and Social Networking: While AR technology can now recognize certain images on printed materials, Qualcomm sees expanded opportunities as the technology improves. “The more that our computer vision can see, the more interactive experiences we can put into the environment,” Wright says. “So we want to detect and recognize faces for example—not so that you can point your phone and see [someone's] criminal history. But maybe so we can do a kid’s game to recognize mom and dad and put funny costumes on them, or something like that.”

Facial recognition in a smartphone also would be a useful way for politicians and corporate executives to remember the names of people whose names don’t spring immediately to mind—or for security guards to identify someone who is violating their access privileges.

Navigation and Discovery: One concept that Qualcomm is exploring is similar to Google Goggles, technology that enables users to snap a photo and then ask “what is it?” or “where can I buy it?”

AR applications in navigation would enable users to literally follow a yellow brick road from one location to another, or enable tourists in a foreign country to point their cell phones at street signs, restaurant menus, or grocery stores and see a translation on the view screen.

“There are things like text recognition that we are actively looking for doing things like sign recognition or translation,” Wright says. “There are a lot of things you can do when you can recognize text.”

The potential opportunities for augmented reality might appear endless. But more importantly many of the applications seem really practical and useful. Wright wouldn’t say how much Qualcomm is spending on AR, or how many employees are working in AR research and development. But he says, “As far as I know, we have the largest R&D effort in augmented reality in the world. We are actually doing this out of our San Diego operation, we’re doing it in our Vienna office, and we have efforts in Cambridge, China, India, and Korea. We have quite the R&D organization.”

Bruce V. Bigelow is the editor of Xconomy San Diego. You can e-mail him at bbigelow@xconomy.com or call (619) 669-8788 Follow @bvbigelow

By posting a comment, you agree to our terms and conditions.

  • Jeff Crusey

    This is why I miss San Diego! Forget the beaches, sun, beautiful people….fml

  • http://www.previewlabs.com Bernard François

    Interesting article!

    I’m also waiting for a first killer app (or game) to make everyone enjoy this technology. I feel there’s a huge potential in there, but someone will need to come up with the right idea for a game or app to be really successful.

    With PreviewLabs – a company specialized in rapid prototyping for video games, we’re also following this (and any other cutting edge technology) very closely.

    We had a look at Qualcomm’s AR technology for Android in Unity3D a few months ago, and posted a summary of our findings about it today on our blog:
    http://www.previewlabs.com/augmented-reality-our-findings/

  • Pingback: Coax Interactive Media Services » Interactive Media Solutions & Services Partner » Weekly Linkfest Plus