Yahoo Challenges Apple with a Cocktail of Mobile Publishing Tools

1/26/12Follow @wroush

This is a story about what goes on under the hood of your smartphone or tablet device. It’s also about Yahoo, the troubled Santa Clara-CA based advertising and information giant. But Yahoo doesn’t make a single mobile gadget of its own. So what’s the connection?

It turns out that Yahoo (NASDAQ: YHOO) has ambitious plans to help publishers get more efficient about how they push content out to mobile devices. Specifically, Yahoo wants to become the new middleman of the mobile publishing world, giving media companies software that they could use to reach users of iPhones, Android devices, Windows phones, and other gadgets without having to bow to the programming approaches favored by their powerful makers—namely Apple, Google, and Microsoft.

To show how the system might work, Yahoo launched a fancy personalized news app back in November called Livestand. The app lets you select feeds from Yahoo partners like Forbes and ABC News and browse their stories on customized, magazine-like pages. It’s full of nifty user-interface elements like a 3D sideways-scrolling publication gallery. So far Livestand only runs on the Apple iPad, and at first glance it’s pretty similar to Flipboard, Zite, Google Currents, and a number of other social news reader apps. But Livestand’s true importance is as a demonstration of what’s coming. The unique and potentially revolutionary thing about the app is its software design: it may look and act like a native iOS app, but it’s mostly written in Javascript and HTML5, the languages of the Web.

Bruno Fernandez-Ruiz

How Yahoo pulled this off, and what it could mean for content owners who don’t want to put all their eggs in Apple’s basket—or Google’s, or Microsoft’s—was the focus of a long Xconomy interview last week with Bruno Fernandez-Ruiz, chief architect for Yahoo’s platform technology group in Sunnyvale, CA. I’d heard Fernandez-Ruiz speak before about how Yahoo is betting on HTML5—the next-generation version of the markup language underlying all Web pages—as an antidote to overreliance on proprietary operating systems like Apple’s iOS. “If you only work in iOS you are bound to the rules of iTunes,” he said at a December talk in San Francisco. “Publishers want pixel-precise, ‘Cupertino-like’ experiences—and we can do that, but also make layouts fluid,” he said.

I wanted to know more about exactly how Yahoo can do this, so I invited Fernandez-Ruiz to my office and quizzed him about the state of mobile software architecture, the role of the Platform Technology Group inside Yahoo, and the true significance of Livestand. The story he told will be eye-opening for anyone who was under the impression that the future of mobile apps is in Apple and Google’s hands alone. Those two companies may control the lion’s share of the smartphone market at the moment, but if Yahoo goes through with plans to share the tools behind Livestand with outside developers, it could help push the siloed mobile-app world back in the direction of the open Web, where no single company is able to dictate how online software and services should work.

The first thing you need to understand about Yahoo’s publishing vision is that it’s coming from the Platform Technology Group. This is the same part of the company that created and then open-sourced key technologies that are now part of the Web’s infrastructure, such as Hadoop, which allows companies to run big, distributed software systems, and YUI, a library of JavaScript tools for building rich Internet applications. Yahoo built many of these tools as part of an effort that began more than half a decade ago to reduce what Fernandez-Ruiz calls a “technical debt.” The company was weighed down by all of the separate technologies its engineers had built to support services like Yahoo Music and Yahoo Movies, and it needed a central platform. “There was a realization around that time that we had to switch the company from being vertical to being horizontal, and start creating reusable technology that we could deploy across the whole place,” he says. “That is how Hadoop got started, for example.”

Technologies created by the Platform Technology Group, such as Yahoo’s Content Optimization and Relevance Engine (C.O.R.E.), also help the company and its partners tailor content to appeal to specific users based on their demographics. Fernandez-Ruiz says click-throughs increased 300 percent after Yahoo applied C.O.R.E. to its own home page and news site—and the company can provide similar boosts to partners such as ABC News who want to run the same technology on their websites.

“Knowledge as a Service is the term we coined,” says Fernandez-Ruiz. “You give us data and content, and we will give you content recommendations, personalization, a targeting strategy, and computational advertising to help you make more money.” (Computational advertising is a sub-discipline of computer science, developed partly inside Yahoo’s research division, which aims to show Web and mobile users the best possible ads based on their locations, contexts, or activities.)

Fernandez-Ruiz says Knowledge as a Service “is really our secret sauce at Yahoo—applying science and art at scale.” The more publishers who use Yahoo’s Web technology, after all, the more opportunities the company has to deliver ads or other business-to-business services. Which explains why the Platform Technology Group has been putting a lot of work lately into a set of cross-platform programming and publishing technologies that it calls Cocktails, introduced last November at a Yahoo “Product Runway” event. Livestand is powered by Cocktails— which, like any good cocktail in the real world, has several key ingredients. To understand where Fernandez-Ruiz’s group is going with all this, we need to walk through them one at a time.

Mojito

Smartphones and tablet apps are a huge boon for consumers. But they’re a huge headache for developers, since the three major mobile platforms use three wildly different programming languages. (Apple’s iOS uses Objective-C, Google’s Android OS is largely based on Java, and Windows Phone is based on C#, originally developed for Microsoft’s .NET initiative.) “Ideally, you’d want a single programming language, because each languages carries with it a lot of luggage—tools, IDEs [integrated development environments], consultants, a whole ecosystem,” says Fernandez-Ruiz. “For every app you build, you need to hire a separate team with separate skills. So your costs go up with every new platform. Plus there are rules on how you get in and what you can do.”

If you were a big, neutral player like Yahoo and wanted to move mobile development toward the nirvana of “write once, run everywhere,” you could develop and promote a new fourth language—something Yahoo certainly has the resources to do. Or you could just decide to improve on an existing, widely used language designed to run inside the most common of all programming environments, the Web browser. That’s the course the Platform Technology Group decided to take, by basing Cocktails on JavaScript and HTML—the same technologies behind the AJAX (Asynchronous JavaScript and XML) development approach that undergirded a new generation of Web 2.0 services in the mid-2000s.

But while JavaScript has big advantages—it’s powerful, non-proprietary, and understood by millions of developers—there’s a problem applying it to mobile applications. JavaScript-heavy sites use lots of code. Sites like Twitter.com or Google Docs, for instance, might require 400 kilobytes or more of pure JavaScript. When you’re sending that much code down to a browser, it gets stretched out across lots of Internet packets, which have to be carefully reassembled before any of the code can be executed. And on wireless data networks, which often suffer from high packet loss and re-transmission, that reassembly can take a long time. “It’s not linear,” says Fernandez-Ruiz. “On a bad network, 20 kilobytes might take 1 second, but 200 kilobytes doesn’t take 10 seconds—it might take 20 or 30 seconds.”

Unfortunately, most JavaScript-based Web applications don’t fail gracefully. If you don’t get all the packets, you have nothing: the page just doesn’t load. “We think that is an unacceptable result,” Fernandez-Ruiz says. So his group came up with a workaround: a way to run JavaScript code on the server side of a Web interaction, rather than just on the browser side. That’s what Mojito is about.

The etymology explains everything. Lots of Web applications run partly on a server and partly in a browser. The code directing things on the server side, usually written in a language like Python or Ruby, is called a “module,” and the code executing actions on the browser side, usually written in JavaScript, is called a “widget.” Fernandez-Ruiz’s team designed self-contained packages of JavaScript code that can run on either side. They’re both modules and widgets—hence the name “mojits.” (Add some mint and lime juice, and it’s a short step to “Mojito.”)

The point of a mojit is that it can start running on the server side long before the code has been fully reassembled and executed on the mobile-browser side. Once the mojit finishes loading on the device, control is handed back to the browser, but in the meantime, the server sends the browser a low-bandwidth image—a kind of mirror. “We execute the same JavaScript on the server, render a page that is usable, and give that page back to the browser so that the user has something he can click on,” Fernandez-Ruiz explains.

The login page in Livestand is a mojit. So is the scrolling, 3D publication gallery. That means the app works seamlessly for users, no matter how much new content they’re accessing during a given session, or how good or bad their network connection might be. “We are shifting computation from the client to the server side, so your user experience as a whole is better,” says Fernandez-Ruiz. But the developer experience is also better, since mojits consist mostly of JavaScript, HTML, and CSS3 (the style sheets that control the look and formatting of Web documents). Only a few elements of Livestand are written in the iPad’s native Objective-C, Fernandez-Ruiz says.

Yahoo expects to open-source all the code behind Mojito “a few weeks from now,” Fernandez-Ruiz says. He says the aim is to allow other mobile developers to experiment with hybrid server/browser architectures and build smartphone and tablet apps that will run well even in parts of the world with poor wireless connectivity.

Chromeless Web Runtime

The second ingredient in Cocktails doesn’t have a brand name yet, which is why it’s currently going by the somewhat clunky name “chromeless Web runtime.” (In keeping with the cocktail theme, I suggest “Rusty Nail.”) This boils down to an effort to bypass the browsers baked into the main mobile platforms. On the iPhone and the iPad, for example, developers who want to include a “Web view” in their apps—that is, any browser-like page that calls information from the Web—are usually forced to use the mobile Safari browser that comes with iOS. That’s a frustration, says Fernandez-Ruiz, because “the browser environment is fully managed,” meaning the local operating system handles key tasks like allocating and deallocating memory. This is a key function on mobile devices, which, as a rule, have less RAM to go around than laptops or desktop machines, and third-party developers would ideally like to control it themselves, or at least have a better picture of what’s going on.

“Chrome” is developer-speak for the buttons, borders, and other common user interface elements in a browser or application window, but by extension it describes the whole relationship between a mobile operating system and its native browser. To create Chromeless Web Runtime, Fernandez-Ruiz says his team asked “How can we create a browser without a browser? In other words, a Web execution environment where you get a DOM [a document-object model], a JavaScript engine, all the stuff you would normally get with a browser, but you provide your own chrome?”

In Livestand on the iPad, when you think you’re opening a Web browser, you’re not actually opening Safari—you’re running Chromeless Web Runtime. The JavaScript in the mojit is being executed inside a neutral environment, rather than the operating system’s native browser, giving developers the ability to see and control things like memory allocation. “We use that mostly for application performance and instrumentation,” Fernandez-Ruiz says. “It allows us to optimize our own code. It can tell developers that if code is very slow, for example, you might want to consider shifting [a mojit] to the server side.”

Chromeless Web Runtime is a joint project between Yahoo and two other companies that Fernandez-Ruiz says he can’t yet name. For political reasons, it won’t be open-sourced right away. “In the long run, it is not a technology that we want to be closed,” he says. “We are discussing a number of options for working with others to make it open source.”

Manhattan

The last ingredient in Cocktails will probably never be open-sourced, but it will be available to publishers who want to partner with Yahoo. It’s a private cloud service called Manhattan.

At its core, Manhattan is simply Node.js, a free server-side platform developed by San Francisco-based cloud technology provider Joyent for managing distributed applications. But on top of that, Yahoo has added functions like fault isolation, security, user ID and login systems, and access to C.O.R.E. “You could use Mojito without Manhattan,” Fernandez-Ruiz says. “You could just run your own Node.js on Amazon EC2. But you will be pretty much on your own. We have optimized [Manhattan] to run mojits and give you access to all of these Yahoo APIs: the user ID, the recommendation, the whole Knowledge as a Service stack.”

So here’s the big picture: A publisher like Forbes wants to create an interactive iPad or iPhone version of its publication. It hires some Web developers to write JavaScript mojits to handle the app’s main user-interface elements. Much of the content gets pulled straight from the publication’s existing HTML or XML repositories, and is displayed inside Chromeless Web Runtime. To track registration and subscriptions, the publisher uses its own account database, or uses Manhattan to tap Yahoo’s account system, so users can log into the app using their Yahoo usernames and passwords and also get content recommendations based on Yahoo’s own vast database of customer-behavior information. When the developers are finished, they wrap everything up inside a bit of native iOS code and they submit the app to the iTunes App Store.

Then when Forbes wants to come out with an Android version of the app, the same developers simply repurpose the existing mojits, putting them inside a Java wrapper instead. (Fernandez-Ruiz says an Android version of Livestand, demonstrating how Cocktails works on the Google operating system, is coming soon.) In essence, each new Cocktails app would be a standalone, publisher-branded version of Livestand—but simpler, since there won’t be a need to navigate between publications.

Once you know the whole story behind Livestand, the app itself becomes a little easier to parse. In a review of the main social news reader apps, published before my interview with Fernandez-Ruiz, I wrote that Livestand “gets the award for the most agonizing distance between technical virtuosity and actual usability.” In retrospect, that was a little harsh. My main complaints about the app relate to its somewhat confusing navigation scheme and to design choices that, in my eyes, make the individual stories harder to read than they should be. It’s now clear to me that I was missing the point, since Livestand wasn’t really designed for usability. It’s a technical demonstration, aimed at publishers as much as consumers.

“I don’t think we want to be a social news reader,” says Fernandez-Ruiz. “What we want to be is a partner for a lot of publications to gather magazine-style content onto the iPad easily. Where Livestand has not been positioned, as some of the competitors have done, is as an app that gets a bunch of stuff from a number of places, but the brand of the publisher is gone. On Livestand, you will notice that the publisher brand is very obvious everywhere.” That’s precisely because Yahoo sees Livestand as the prototype for a series of independent apps; it’s like a brochure for the Yahoo-powered tablet magazines of the future.

The Data Asset

As everyone knows, it hasn’t been smooth sailing for Yahoo over the last few years. The company still runs a huge display-advertising business. But as a child of the first era of dot-com innovation in the 1990s, it hasn’t managed to adapt swiftly to the social networking and media-sharing revolutions. Its stock value has slipped, its advertising revenues haven’t kept up with those of competitors Google and Facebook, and there’s been a lot of turnover in the top ranks. Ex-CEO Carol Bartz was ousted last fall after a fractious split with Yahoo’s board, and was finally replaced this month by former PayPal president Scott Thompson, just before the departure of co-founder Jerry Yang.

The new CEO hasn’t yet laid out his vision for the company. But he did say this week, after the release of the company’s quarterly earnings report, that Yahoo’s proprietary data may be its “single most underrated, under-appreciated and under-used asset.”

Which fits with everything Fernandez-Ruiz told me. “Regardless of what the strategy is for Yahoo, you are going to need storage,” he says. “You are going to need a big data platform. And that is what we have been doing [in the Platform Technology Group]—solving those hard technical infrastructure problems and reducing the technical debt that has accrued over 15 years.”

While Yahoo might look from the outside like a consumer company, Fernandez-Ruiz summarizes, “We also have a lot of B2B businesses, and they all have the same mission, which is to increase segmentation, increase monetization, increase CPMs, increase sell-through. If you multiply all these things together, that’s where the money is for us and for publishers. So in the grand scheme of things, [Cocktails] is a natural outcome. I agree, it’s transformational to our business. But it is a necessary condition for our business going forward.”

[Update, April 2, 2012: Yahoo has officially released Mojito as an open-source JavaScript framework.]

Wade Roush is Chief Correspondent and Editor At Large at Xconomy. You can subscribe to his Google Group or e-mail him at wroush@xconomy.com. Follow @wroush

By posting a comment, you agree to our terms and conditions.