Craig Mundie of Microsoft on the Future of Software: Digital Assistants, Natural User Interfaces, and Room Computing

7/13/09Follow @gthuang

When you’ve been at Microsoft for 17 years, you’ve seen a few things. This morning, Craig Mundie, Microsoft’s chief research and strategy officer, shared some of his hard-earned wisdom in a keynote talk at company headquarters in Redmond, WA—helping to kick off Microsoft Research’s 10th annual faculty summit.

Mundie outlined his broad thoughts on the future of computing and Microsoft’s role in it—and went into pretty good depth in spots. Nothing too earth-shattering, but it was cool to hear the technologist discuss the roadmap of where his company may be headed with more of a research and innovation hat on, and not much marketing slickness. The backdrop of his talk was that computing devices are becoming more and more pervasive (at least in wealthy countries), so what are the emerging opportunities? “Computing everywhere will be something we take for granted,” Mundie said. “But it was not common knowledge as little as 15 years ago.”

“Today, computers work at your command,” Mundie continued. His vision, he said, is that they will transition to “working on your behalf.”

Here are my top five takeaways from his talk:

1. It’s all about the natural user interface.

Today’s devices are able to understand voice, handwriting, and touch commands better than ever before, but nobody has really put it all together yet. “All the things we talk about as natural user interfaces have been largely used one at a time as enhancements to [graphical user interfaces],” Mundie said. Gesture recognition, expressive responses, immersive 3-D virtual environments, and understanding of context—these advances in computing algorithms will lead to software that is “better at anticipating what you might want.” (Mundie said that Nathan Myhrvold, who hired him, pointed out that early movie cameras were used to film plays before people experimented with them to create new kinds of film experiences. In the same way, better user interfaces will lead to new ideas about how to use software, Mundie implied.)

2. It’s time for the digital assistant—but fear not, real assistants.

Mundie showed a demo of Microsoft researcher Eric Horvitz talking to a “robotic receptionist” (on a screen) to schedule a meeting. The software used machine vision to track Horvitz’s movements, gaze, and orientation to the screen, speech recognition to understand what he was saying, and speech synthesis to communicate back to him—all in real time. Mundie admitted each element was still rough, but said it could be practical in just a few years. “My dream in the future is a robotic teaching assistant, and a robotic physician’s assistant,” Mundie said. (And fear not, consumers—the digital assistants look nothing like Clippy.)

3. It’s not all about the cloud.

When it comes to computing tasks like real-time interactions with a digital assistant, even tiny delays in responses or understanding nuances in speech or gestures can make the whole thing break down. “As we move to continuous, contextual awareness, the idea that we can time-share these things is not practical,” Mundie said. He emphasized that the processing, communication, and integration with any other software needs to be done locally, not by a remote server in the Internet cloud. Otherwise, he said, “the latencies are too much.” (Of course, the company that makes most of the world’s client-side operating systems and a good bit of its desktop-based software would say that.)

4. It’s four devices, not three.

Back in May, Microsoft’s chief software architect Ray Ozzie talked about “three screens and a cloud” in discussing the company’s vision of cloud computing. The devices Ozzie was referring to were phones, laptops, and large monitors. But Mundie outlined four types of devices in his broader view of computing: phones, laptops, desktops, and something he called “specialty computers.” These are handhelds specialized to do certains kinds of computationally intensive tasks, like environmental forecasting, medical imaging analysis, or even delivering prenatal care in rural regions of a developing country. Mundie added that such devices could help “solve some of society’s biggest problems. All of us need to raise our sights a little bit.”

5. It’s the room, stupid.

Think of desktops as representing computers that don’t move. But laptops have become more dominant, at least for business use. “It’s actually a failure of the industry, and even Microsoft itself, that we haven’t asked the question, ‘Is there more you can do if you don’t have to move it?’” Mundie said.

With that, he gave a slick demo of the office of the future: high-definition displays on walls, a keyboard projected onto a desk surface, gesture recognition software that lets you move things around on a digital whiteboard, a digital assistant on the screen, a video conference with a colleague with interactive graphics within the video screen, and so forth. Admittedly, many of the demo features were “faked,” but most of the interface parts are doable in principle, using technology like Natal, the Xbox gesture-recognition tool, and surface computing. “If you can change people’s visions of business productivity, they’ll pay money for that,” Mundie said.

“There will be a successor to the desktop,” he concluded. “It will be the room.”

Gregory T. Huang is Xconomy's Deputy Editor, National IT Editor, and the Editor of Xconomy Boston. You can e-mail him at gthuang@xconomy.com. Follow @gthuang

By posting a comment, you agree to our terms and conditions.