Superfish Aims to Dominate Visual Search, One Product at a Time
We’re awash in images. Human beings take one billion photos every day, and websites of all stripes have billions more. But most of those images vanish into the black hole of the Internet, untagged and unsearchable.
Sure, people can tag each photo with names, locations and other data, but very few take the time to do it. As a result, trying to find a dog that looks like Spot, or all of the pictures of you posted on the Internet, or where you can buy a chair you found on a blog, can be an exercise in frustration. It would be so much easier to start with an actual image—of yourself, of a friend, a suspected criminal, a product—and then quickly search vast numbers of images to find those that are identical or very similar.
Adi Pinhas has been working for years to make that possible. “With visual-search we all can be like kids, pointing to something and ask what is this? Are there any more like that?” he says. “As humans, this is more natural for us.” For the last eight years, he and Superfish, the Palo Alto, CA-based company he founded in 2006, have been toiling to create the combinations of algorithms needed to compare and match pictures.
In 2011, the company launched its first product, Window Shopper, a desktop and mobile app that allows buyers to click on a photo of, say, a pair of shoes on a blog, and then find similar shoes on sites like Amazon or eBay. “You don’t need key words or to explain why you like this shoe,” he says. “You click on an image and it happens. It’s easy for the machine to see what you mean,” Pinhas says.
Next came PetMatch, which lets people upload a photo of the kind of dog they like, then searches adoptable pets on the Internet to help users find a similar pup in need of a home.
Within the next six months, Pinhas expects Superfish to release at least two more apps that use visual search to solve a problem. Depending on user uptake, the company may eventually release a general visual search engine that can find pictures the way Google finds words, phrases, and images. “We will say to users, ‘Show me what you’re looking for, and we will find it,’” says Pinhas.
It’s a highly complex task, and though there are some products out there, no one has perfected image search—not even Google. In 2011, Google made it possible to drop images in their search bar in Google images. But that’s most helpful in finding images that already have related content on the web, like landmarks and paintings. It doesn’t work so well for puppies or photos from your wedding. Two years later, Google added similar capabilities to help Google+ users find their own photos without tagging them. But the company noted in a post announcing the capability that there’s still a long way to go to perfect image search. “Have we gotten computers to see the world as well as people do?” the note asked. “The answer is not yet, there’s still a lot of work to do, but we’re closer.”
So can Superfish do better than Google? The two companies have taken different approaches. For the most part, Google has used image search capabilities to enhance its existing products—Google+ and Search, while Superfish has used its algorithms to create dedicated products.
Aside from the tech giant, there aren’t a ton of companies working on the problem—though Pinhas says he has seen a lot of startups try and fail, and Superfish has managed to make serious strides with the technology and become profitable. Superfish’s general approach: The company’s algorithm splits the query image into thousands of small sections. Each section has distinct features, like a particular pattern, texture, edge, or dot. The algorithm then searches for other images with similar features in similar sections, with about the same distance between them.
When Pinhas first started the company, creating the right algorithms was the biggest challenge. “There wasn’t anything that we could copy, or something slow we had to make fast,” Pinhas says. “We just had to invent them.”
The company had a team of a dozen or so PhDs working for four years to find the algorithms—and combinations of algorithms—that would allow them to accurately search for photos. Though machine-vision technology was a huge piece of the puzzle, the team pulled algorithms from various sectors, including DNA search. “Today the visual search engine is a very complex machine,” Pinhas says. “We have many algorithms that are running in parallel to find similar content. It’s not just, ‘what is the big secret or the one algorithm?’”
Starting from scratch takes time, but it also takes a lot of money. Superfish didn’t release its first product— WindowShopper— until two and half years ago, five years after the company was founded. “It’s not the kind of thing where you create a company, and six months later you release a product,” he says. “Any company that thought, ‘In two years we will have something,’ two years later they understood it’s too complex. We weren’t even close to solving this two years in.” Spending five years developing a core technology requires a lot of money and very patient investors.
Luckily for Pinhas, he found some in Draper Fisher Jurvetson (DFJ), a VC firm that invests in early-stage technologies, and has funded companies like Skype, SpaceX, and Tesla. “We had a rough demo, and they understood it’s going to take time and require a lot of money, but it was interesting to them and they agreed something like this could be huge,” says Pinhas.
To date, Superfish has raised about $20 million, mostly from DFJ and Vintage Investment Partners, with some smaller investors participating in the seed round.
The company became profitable about a year and a half ago, thanks to WindowShopper, which currently has 100 million monthly users, and a high conversion to sale rate for soft goods, which include non-hardware items, like clothing jewelry and furniture. When consumers buy something through WindowShopper, Superfish gets paid affiliate fees, which vary based on type of good and country of origin, somewhere between two and 20 percent. On media items, like DVDs, for example, the rate is around two percent, whereas luxury items like watches will score the company around 20 percent.
Superfish isn’t Pinhas’s first company. In 1998, he founded Vigilant Technology, which developed servers that collected video from surveillance cameras. Though the company used some similar technology, it solved a very different problem—storing video for two to eight weeks and helping the monitoring team find information like license plate numbers —and counted big institutions like airports, city centers, casinos, and the New York City Police Department among its customers. For Vigilant, a typical product developed for a particular institution collected video from 3,000 cameras.
Though the user base and core technology behind the two companies was very different—and a lot had changed in machine learning since 1998—the new company seemed like a natural pivot to Pinhas, who has a master’s in machine learning. It was a good fit for his Superfish co-founder as well; Pinhas met Michael Chertok when he came to work at Vigilant.
Superfish has leapt several big hurdles. The company has developed a core technology that works. It has used that technology to create products that are now bringing enough revenues to turn a profit. But despite these successes, Superfish still finds that there are more obstacles in its path. Pinhas’s current challenge is convincing potential partners like Samsung that it makes sense to add image search within a connected camera. After all, search functions bring ad revenue, and the more accessible they make it, the more people will use it.
Pinhas believes such image searches will be an indispensable part of the future. “Imagine scenario where you order a dessert, take a picture, and get the recipe,” Pinhas says. “Or you can find which friend uploaded similar dessert pictures to Facebook.” Or you take a picture of your kids in a park and find similar play structures in your area. In his mind, these capabilities aren’t far off. “In two to three years, visual search is probably going to be everywhere,” he says. “Part of every camera, part of every browser. You’re going to be able to search and find similar content in every category, across every type of image.”