Delays In Software Patch Pushed Security Firm to Disclose VMware Flaw
VMware’s five-month delay in issuing a fix for a security hole that could leave three of its workstation virtualization programs vulnerable to takeover by hackers—and its elusiveness about announcing a patch date—prompted a Boston-based security firm to break with its usual practice last week and publicly disclose the problem.
Iván Arce, CTO of Core Security, whose engineers discovered the vulnerability last October, told us of his firm’s mounting frustration with VMware—and its concern for commercial users of the affected software—after we went back to both parties this week for more details of the disclosure. Arce says his firm doesn’t normally release its findings about vulnerabilities in commercial software until the vendor has prepared a patch. But in this case, given VMware’s lack of progress, Core Security felt obliged to publicize the bug so that users could protect themselves from potential hacker attacks.
“With every day that passes, the chances of somebody finding this on their own and exploiting it increase,” Arce says. “Instead of waiting for an official fix, we figured the best thing we could do was to publish the information and a workaround, so that people could protect themselves and implement the measures they think necessary.”
VMware (NYSE: VMW) is still working on a fix for the security hole. The company tells Xconomy that it has no specific date to announce for the release of a patch for the affected programs, which include VMware Workstation 6, Player 2, and ACE 2.
Nobody is accusing VMware, a subsidiary of Hopkinton-based EMC (NYSE: EMC), of negligence or foot-dragging. In fact, Arce says VMware is far from the worst vendor Core Security has worked with on a security problem. But it’s not the most efficient, either, he says. And the whole episode provides an interesting look at the sometimes awkward pas de deux between security companies and commercial software vendors that occurs after a vulnerability is discovered. This dance is usually hidden from view—but in this case it’s very public, thanks in part to Core Security’s practice of publishing the entire timeline of its communications with vendors when it issues a security advisory.
The vulnerability in VMware’s software relates to the “Shared Folders” feature of the Windows versions of VMware Workstation, Player, and ACE, which all support the creation of a virtual guest machine on a host computer—a computer-within-the-computer that can run a different operating system and applications than the host. The Shared Folders feature is intended to give users an easy way to transfer files from designated folders on the virtual computer to designated folders on the host computer, or vice-versa.
In March, 2007, security company IDefense Labs discovered a way to change the names of the targeted folders and save files anywhere. In effect, any hacker who already had control of the guest system could exploit this flaw to gain control of the host computer as well—violating the supposed isolation between guest and host that is a big selling point for VMware and other virtualization vendors. VMware patched the problem, but in the process of testing the patch last fall, Core Security’s engineers found yet another way to outwit the software’s system for filtering out invalid folder names.
Core Security notified VMware’s security team about the new problem on October 17, 2007. The response wasn’t exactly alacritous. Here’s Arce’s version: “VMware told us that they were planning to fix it with the next release of the VMware products in December. Then they told us that since it was the end of December [when many employees would be away on holiday] they would let it slip to early January. We said ‘Okay, no problem; we’ll just wait for that to publish our advisory, since we’d prefer to have a fix for a problem before we make it public.’ Just before January ended, we were informed that the date for the release had slipped to the second week of February. Then we heard that that date was not achievable anymore and that they were scheduling the release for the third week of February.
“In view of that,” Arce continues, “we said, ‘Okay, the developers’ estimates keep changing and now we have a new estimate. Since this may or may not be realistic, we should just publish this and provide a workaround for all users.’”
The workaround is simple: disable Shared Folders. And that’s exactly what VMware advised customers to do in a “critical security alert”—but it didn’t issue that alert until late February, after Core Security issued its own advisory.
I asked VMware whether Arce’s version of events squared with theirs, and whether they could shed any light on the process going on inside the company’s security team since October. I also asked them whether it’s normal for the company to let five months or more go by before it issues a patch for a vulnerability of this magnitude.
Nand Mulchandani, the company’s senior director of products, sent the following response: “Security is something we take very seriously at VMware. The trust our customers place in VMware and our products is paramount. When a potential security issue is discovered, the VMware security team immediately begins an investigation. If a security threat is found, our engineers begin working on a patch. Concurrently, we create and educate our customers on best practices to workaround a particular issue until a patch is ready for public use. We issue patches on an as-needed basis and as quickly as possible.”
I also asked VMware whether the company has a new timeline for issuing a patch. “We do not have a set schedule for updates, like a ‘patch Tuesday,’” Mulchandani said. “However, we do try to aggregate patches for our customers in a single update. VMware has been actively working on a patch since being first notified of this particular issue in late last year. The patch will be issued once complete. At this time, we don’t have a specific date to announce.”
Mulchandani emphasizes that the Shared Folders feature is disabled by default in Workstation 6, Player 2, and ACE 2. So the folder-renaming exploit that Core Security discovered is only a danger if a customer has turned on the feature—and then only if they’ve configured specific folders for sharing. “If an end user turns on the shared folders feature, a security warning is presented,” Mulchandani adds.
But that warning is no substitute for a real patch, according to Arce, who says he is surprised at the amount of time it’s taking for VMware to issue a fix. “I don’t know the internals of the VMware engineering process, but from the outside, you’d say that this shouldn’t be that hard to fix, especially because there was a similar bug reported in the same products about a year ago,” he says.
At the same time, Arce acknowledges that very few software problems have an instant solution. “It’s normal than when you report a vulnerability to a vendor, there is an extended conversation that can go on for weeks or even months,” he says. “In this case VMware was not as responsive as they could have been, but they weren’t bad, either. We’ve been through much worse with other vendors.”
Arce says his suggestion for VMware would be to better integrate its security operation with the teams responsible for product updates. “I think there is a lot of room for improvement in terms of the processes and the relevance of their security group within the VMware organization,” he says. “They have a dedicated security group and they have all the right skills. I think they need to get all those ingredients together and make them work in a more efficient manner.”