Combating the “CSI Effect”: Boston’s Salient Stills Extracts Evidence from Grainy Surveillance Video
Generally, I’m a big fan of the CSI shows (CSI: Miami, CSI: NY, and of course, the original CSI: Crime Scene Investigation), because the heroes are scientists. Who would have expected a trio of dramas about a bunch of wonky forensics experts to stay atop the ratings for years? But I do have a pet peeve about the shows: they routinely exaggerate the power of the image-enhancement software available to crime labs. CSI: Miami took this to a ridiculous and irresponsible extreme a couple of seasons ago, with an episode in which the Miami-Dade PD’s super-sleuths isolated a frame in a surveillance video, then blew up the image to such huge proportions that they were able to read a car’s license plate number in a reflection on a passerby’s eyeball.
What a lot of TV viewers probably don’t understand is that simply zooming in on a tiny feature in a digital image doesn’t reveal hidden information. It just makes the existing pixels bigger and fuzzier. But when influential TV shows mix science fiction with science facts, it can give average citizens—including jury members—unrealistic expectations about the capabilities of law-enforcement agencies. Indeed, according to USA Today, the “CSI effect” has made it harder for prosecutors to win convictions in the absence of spectacular physical evidence.
But there’s a company right here in Boston that’s correcting misperceptions about video enhancement technology—and at the same time pushing its real capabilities forward. It’s called Salient Stills, and it’s helping police departments and other government agencies around the world wring as much evidence as they can from surveillance recordings—which are usually far more blurry, jerky, and jittery than the videos portrayed in detective shows.
“Unfortunately, not only are juries and judges under these misapprehensions, but a lot of our customers need to be educated as well, so we spend a fair amount of time talking about the difference between what happens in Hollywood and what happens on your desktop,” says Salient Stills founder and CEO Laura Teodosio, a former Apple researcher who was one of the first people to get a degree from the MIT Media Lab in the early 1990s. “But once people begin to understand what can be done, they really start to appreciate it.”
Last month, the company rolled out the latest version of its flagship product, VideoFocus Pro, a Windows program that can take in video from almost any source—say, an old VCR-based surveillance system in a convenience store, a modern DVR-based system at a bank, or even a crime victim’s cell phone camera—and help users clean it up and sift through it, frame by frame, for key details.
In the past, says Teodosio, many police personnel have tried to process crime-scene videos by kludging together separate tools such as Adobe Premiere (for video editing) and Adobe Photoshop (for cleaning up still images). But VideoFocus Pro handles most of the same tasks as both of those programs, and adds specialized features that become crucial when surveillance video is being used to identify criminal suspects—and when it’s likely to end up as legal evidence in court cases. That includes software filters that correct common video artifacts such as interlacing (more on that below) and highlight hard-to-see features in nighttime shots. VideoFocus Pro also creates audit trails documenting exactly how a video file has been modified at each stage of processing, which helps to maintain the chain of evidence. “All of the things most law enforcement customers would need to do have been brought into one application, without the overhead and the complexity of things they don’t need to do,” says Teodosio.
Salient Stills’ 10 programmers, who recently escaped rising rents in Boston’s Fort Point Channel area to occupy a fourth-floor loft in Brighton, work with customers at dozens of city, county, and state police agencies, as well as national agencies like the FBI, CIA, DEA, and NSA and quite a few foreign organizations, such as Japan’s Public Security Intelligence Agency. But the company didn’t start out catering to law enforcement. When Teodosio founded the company in 1998, the main idea was to help big media companies create print-quality still images from video footage.
Salient Stills’ very first customer was The New York Times, which put the software to a dramatic test the night of November 7, 2000, when George W. Bush’s razor-thin margin over Al Gore in Florida appeared to have brought him the electoral votes needed to win the Presidency. The Times wanted to juxtapose pictures of Gore and Bush on the next morning’s front page. They had photographs from Gore’s concession speech, but Bush didn’t make a public appearance until minutes before the paper’s deadline, meaning that videotaped TV coverage was all the paper had to work with. “They didn’t want to seem like a liberal, Democratic-leaning paper, with a good image of Gore next to a blurry video of Bush,” recounts Teodosio. “But they were able to run the video of Bush through our software and clean it up and run the two images side by side, above the fold.”
The Times is still a customer today. But it didn’t take long after the 2000 election for the startup to sell its system to nearly every other media company big enough to need it. So the company started looking for new users—including the Boston Police Department, which turned to Salient Stills for pro bono help analyzing drug-store surveillance tapes during a wave of Oxycontin robberies in the summer of 2001. “We always knew there was potential in the law enforcement market, where you’ve got video and you need to make it look better so that you can ascertain something,” says Teodosio. “But in the Boston Police Department, we had a our first real live case study. In exchange for doing this video work for them, we picked their brains about what they wanted to do and what they needed. They effectively became our first law-enforcement customer.”
And then came September 11, 2001. Two of the four planes involved in that day’s suicide attacks departed from Boston’s Logan Airport, and conspirators Mohamed Atta and Abdul-Azzia Al-Omari had flown into Logan earlier that morning from the Portland Jetport in Maine. Both airports fell under the jurisdiction of the FBI’s Boston field office. “We brought a system over to their office and they used it to create some of the images that you know of Mohamed Atta from the Portland ATM video and the airport security tapes,” says Teodosio. “That was really the moment we changed focus.”
Since 2001, there’s been endless discussion (and considerable federal R&D spending) around face recognition software, biometrics, and other advanced surveillance and security technologies. But even now, very little of this technology is ready for daily use. Salient Stills, meanwhile, has concentrated on making its software as flexible and user-friendly as possible for the people in the trenches—police investigators whose agencies are often too small to have dedicated video forensics facilities and who have to deal with video evidence from diverse real-world sources, much of it very low in quality.
And that’s where Teodosio’s Media Lab training and her work at Apple come in. At the Media Lab, Teodosio was part of the “News in the Future” group led by Walter Bender (recently of XO Laptop and Sugar fame). The group believed that for digital newspapers, which might be viewed on handheld devices that lacked bandwidth or processing power to show video, TV news images would need to be condensed in some way. So for her 1992 master’s thesis, Teodosio developed software that could translate a moving image—say, video from a panning or zooming camera—into a single high-resolution frame, the “salient still,” that synthesized all of the key information over some period of time.
After MIT, Teodosio joined Apple’s advanced technology group in Cupertino, CA, where video processing was also high on the agenda. “It was a great place to be, because there was a very prominent culture of being concerned about user experience,” says Teodosio. “That work was as highly regarded as the underlying scientific work. And I think I learned a lot about the value of putting very hard work into the user interface.”
Ever since Teodosio returned to Boston in 1998 to turn her master’s thesis work into a commercial product, she says, her goal has been “keeping it simple”—for example, arraying multiple frames on a computer display in such a way that crucial moments jump out at the user instantly; enabling users to zoom in on portions of a video using a mouse’s scroll wheel; allowing quick drag-and-drop assembly of new video presentations from accumulated clips; and showing filtered or newly assembled video nearly instantaneously, without long waits for rendering.
Teodosio says such design simplifications have been “critical to our success,” especially among investigators confronted with a jumble of video technologies and formats. While many commercial premises have video surveillance systems, they’re often poorly maintained, and many businesses install them merely as deterrents without paying much attention to the resolution of the video cameras, the format in which video files are stored, or other factors that will determine whether the images are actually usable as evidence.
Interlacing is one common problem: older video systems scan and record every odd-numbered line in a video, then every even-numbered line, then start over. If a camera, VCR, or DVR is out of calibration, the lines can get mixed up, making images fuzzy. But using VideoFocus Pro, investigators can run a “de-interlacing” algorithm that removes fugitive lines with a single click. Teodosio showed me video from a bank heist where de-interlacing helped to make it clear that the perpetrator had a beard.
The software also helps investigators deal with video stored in proprietary formats that can be played as executable files but can’t be opened using standard video editing software. According to Teodosio, the surveillance video industry, unlike makers of consumer still and video cameras, has never bothered to adopt common formatting standards for digital files. For these cases, VideoFocus Pro offers a video-capture feature that, in essence, makes a new recording of whatever is playing on a computer.
Salient Stills’ software isn’t going to help investigators make out a terrorist’s face from his reflection in a droplet of sweat on a hostage’s cheek. But it is helping law enforcement agencies catch and convict everyday crooks. And while the company’s turn toward crime-fighting after 9/11 was “not something I expected,” says Teodosio, she thinks the company’s products may gradually shift back toward the mass-media world. For example, she points to the growing phenomenon of police agencies turning to the public for help by posting surveillance and crime-witness videos on YouTube.
“So much video now is coming in from handheld devices, and there is so much connectivity between computers, that I think our next directions are going to be very exciting,” she says. “We have so much expertise in creating image-processing applications that are easy to use. Now we want to take it to the next level of making it networked, getting the tools into even smaller agencies, and letting customers share information with each other and with the general public in a trusted way.”
That would take additional investment—and so far, the company has been making do on revenues from VideoFocus Pro (the software costs $5,000 to $30,000 per installation) and a $3 million venture pot collected shortly after its launch. “To properly do a networked or software-as-a-service version of our system we’ll need to find a new strategic partner, or go back out for funding,” Teodosio says. And that may happen sooner rather than later. “We want to take it to the next level.”