Photographing Spaces, Not Scenes, with Microsoft’s Photosynth

8/29/08Follow @wroush

Up to now, software giant Microsoft has largely missed out on the digital photography revolution. The most popular photo editing tools come from Microsoft competitors like Adobe and Apple. Flickr, every geek’s favorite photo-sharing site, was invented in Microsoft’s backyard in Vancouver, BC, but went on to become part of Yahoo. And Corbis, Bill Gates’ bold early-90s experiment in licensing digital images for high-resolution displays in consumers’ homes, devolved into an online stock image house.

But the hottest new twist on digital photography is, unexpectedly, a Microsoft product. It’s a powerful Web service called Photosynth that can analyze multiple photos of a common object or space—say, Michelangelo’s David, or Times Square in New York—and intuit a 3-D model of the depicted subject, which then acts as the scaffolding for an interactive photo tour. The creation of a small Redmond-based product group called Live Labs, Photosynth is more than cool enough to earn Microsoft greater mindshare among photographers, both serious and amateur.

I’ve been playing around with the tool since Microsoft started allowing members of the public to create their own “synths” on August 21. I would call Photosynth almost post-photographic, in the sense that it abandons any allegiance to the idea of a single, definitive image (goodbye, Doisneau’s “Kiss by the Hôtel de Ville” or Adams’s “Moonrise, Hernandez, New Mexico”) in order to exploit the abundance of images on the Web, or, these days, on any digital photographer’s hard drive. It assembles related images into interactive montages that can be navigated almost as if the user were walking through or around the photographed space or object.

Wade's Copley Square SynthFor example, in early demos of Photosynth, Microsoft showed how hundreds of images pulled from Flickr could be assembled into a massive 3-D montages of Notre Dame cathedral in Paris or the Trevi Fountain in Rome. Words don’t really suffice to explain Photosynth: to understand it, you should just go to the Photosynth website and explore a few synths. I especially invite you to check out three synths I created last weekend around Boston, representing my South End apartment, Copley Square, and the Christian Science complex.

Embedded versions of these synths can also be found on the following pages of this article. Sadly, the special Photosynth viewer runs on Windows machines only, inside the Internet Explorer and Firefox browsers; this being a Microsoft project, there isn’t yet a version of the program that works on Macs. But at least the Live Labs people are apologetic about that: when you try going to the Photosynth site on a Mac, you get a message that says “Unfortunately, we’re not cool enough to run on your OS yet.”

When you start exploring a synth, you’ll notice that mousing over an individual image brings up ghostly white outlines, indicating that the synth contains other images presenting the same objects from different angles. Clicking on one of those outlines (or on the arrows around the screen) will take you to those other images, but not instantly: Photosynth provides a smooth, animated transition, as if you were merely turning your head or approaching an object for a closer view.

The best synths—that is, those with the most convincing transitions between images and the most complete sense of spatial unity—are those constructed from hundreds of photos taken with Photosynth in mind, like my three synths. The software’s matching algorithms have more to work with when there’s a lot of overlap between adjacent photos. So if you want to make a synth of an object like a sculpture, for example, you have to move gradually around the object, taking a picture every 15 degrees or so. (Conversely, to make a 360-degree panorama, you should take at least 24 separate photos as you spin in place.) If you’re interested in making your own synths, the Photosynth website has an entertaining video with more pointers.

Apparently I absorbed the video’s lessons well, because when I finished uploading the 300 photos I took of the Christian Science complex last Saturday, Photosynth declared them to be “99% Synthy.” My apartment photos were “95% Synthy” and my Copley Square photos were “92% Synthy.” Before you create your own synths, however, be aware that the program is already proving incredibly popular, and that as result, Live Labs’ servers have been largely overwhelmed by uploads. Each of my synths took about 8 hours to upload and process—and each failed at least once along the way, requiring me to start over. The Photosynth team says it’s busy optimizing the system to cope with the unexpected onslaught.

When you’re exploring your finished synths in the Photosynth viewer, there’s a hidden shortcut—the “P” button on your keyboard—that reveals something truly novel and jarring: the “point cloud” that makes up the 3-D scaffolding behind the individual images in a synth. The points represent tiny patches of color or texture that, in Photosynth’s judgment, are shared across multiple images. (This is actually the key to producing a 3-D effect: by comparing images in which the same details are depicted from multiple angles, the software is able to infer the existence of 3-D structures and render them in space. It’s not so different from the way our brains create stereo, 3-D views from the binocular images captured by our eyes. Technology Review‘s March/April issue includes a good explanation of the whole process behind Photosynth, which was conceived by University of Washington graduate student Noah Snavely and built into a workable Web-based system by Snavely and Microsoft programmer Blaise Agüera y Arcas.)

Coffee table point cloudThe more times an object appears in a synth, the denser that object’s point cloud will be. For example, on the coffee table in my apartment, I have two handmade ceramic bowls sitting on a Southwestern-style mat. I took a bunch of pictures of the table while I was preparing my synth, and in the finished point cloud, the mat and the bowls appear like a bright little galaxy of orange and blue stars.

The limitation of Photosynth is that once you’ve spent some time viewing your synths or others’ and exploring all the pretty point clouds, it’s not clear what to do next. (As the Technology Review article’s headline asked, “It is dazzling, but what is it for?”) It would certainly be a useful tool for learning your way around places that you intend to visit—why not tour one of the 200 synths already available for Paris, for example, before your next trip to the City of Light? In fact, Live Labs leader Gary Flake told TR that Microsoft may attempt to integrate Photosynth into the company’s virtual-globe program, Microsoft Virtual Earth, as a kind of shortcut to creating a 3-D metaverse.

The actual coffee tableBut as a tool for individual photographers, Photosynth is still in its earliest stages. For now, you can’t edit or add to existing synths. You can’t assist the program by manually placing the orphaned photos it couldn’t recognize into the jigsaw. You can’t export the 3-D data, as architects who use CAD drawings or virtual-world builders might wish to do.

I’m sure that there are a few artist-geeks out there dreaming up surprising and informative ways to use their allotted 300 photos (uploading more images than that tends to stymie the program). But I think the real significance of Photosynth is that it’s helping to touch off the next major shift in photography: from 2-D to 3-D. In short, it’s now possible to document not just scenes but spaces. And no matter what kind of image-processing software is out there on the Web, most photographers will need more time to get their heads around that idea.

For a full list of my columns, check out the World Wide Wade Archive. You can also subscribe to the column via RSS or e-mail.

Pages 3, 4, and 5 of this article contain embedded versions of the synths of my apartment, Copley Square, and the Christian Science complex. Please note that you can only view these synths on a Windows computer. If you have not installed the Photosynth browser plugin, you will be prompted to do so; please follow the onscreen instructions. Mac users—sorry!

A Photosynth view of my apartment in the South End. (Windows only.)

A Photosynth view of Copley Square, Boston. (Windows only.)

A Photosynth view of the Christian Science world headquarters in Boston. (Windows only.)

Wade Roush is a contributing editor at Xconomy. Follow @wroush

By posting a comment, you agree to our terms and conditions.

  • DisneySynth Project

    It will be very interesting as more synths appear where the photos were obviously taken with the intent of being a synth, as opposed to just whatever photos people happened to have handy. It’s an exciting technology in that people haven’t really fully exercised it yet.

    The Disney Synth Project