As the Technical Director of Solution Engineering at Nutanix, Jon Kohler understands the complexity involved in securing the multitude of applications and solutions required to power such a large organization. “Security is something that we care about intensely here at Nutanix,” Jon said, “because it’s part of our bedrock. It's why customers like us, and we have to take it seriously.”
In December of 2021, software companies around the world discovered that Log4j, an open-source logging library bundled in many software packages, contained significant vulnerabilities, one of which was a 10/10 on the CVSS scale. The Federal Trade Commission called the library “ubiquitous.”
Jon discovered that the offending module recurred throughout their build. “The more we dug,” Jon explained, “the more we realized this bug was everywhere and nowhere at the same time.”
Nutanix moved quickly, despite having multiple build and artifact management systems, as well as a large monorepo with many component branches and hundreds of git repositories. In under four days, Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability.
Speed was of the essence, but the timing of the Log4j news, which broke right before many employees go on vacation for the winter holidays, didn't make things easy.
However, Nutanix armed its engineers with Sourcegraph. Within a couple of days, a few Sourcegraph queries identified every instance of the Log4j vulnerability.
With the Log4j 1.x vulnerability, for instance, codebases were only insecure if they used JMSAppender. Jon used Sourcegraph to see where JMSAppender existed, fixed it, and sent out a release. “That took almost less than five minutes,” Jon said. Sourcegraph released a blog post that explained how other companies addressing Log4j could use code search for similar benefits.
This speed gave the team a head start on mitigation.
Deploying these fixes required quality assurance and testing as well as discovery and fixing.
With the help of Sourcegraph, Nutanix was able to release three back-to-back patches relatively quickly compared to other companies. Nutanix's customers reported satisfaction with both the speed of the patches and their quality.
“We tried to take a measured approach,” Jon said. “We wanted to get things done quickly but without completely flying by the seats of our pants.”
“It's nice when you can just run a report and say, 'Here it is,' or 'Here it isn't.' It's much better than having to say, 'Well, boss, I think we got it all.'”
— Jon Kohler
Tracking down the Log4j vulnerability was, in Jon's words, like “herding cats who were herding mice at the same time.”
Without Sourcegraph, Jon would have either been tracking down whoever built each component to ask them how and where they used Log4j or stumbling through all of the company's repositories.
Using Sourcegraph, Jon discovered every instance of Log4j and was fully confident in the results. “It's nice,” Jon said, “when you can just run a report and say, 'Here it is,'' or 'Here it isn't.'' It's much better than having to say, 'Well, boss, I think we got it all.’”
Nutanix needed that confidence because of its sprawling codebase. One thing that made Log4j especially complicated for Nutanix—as it does for other large-scale enterprises—is that there were multiple source control systems in play. Sourcegraph provided them with “unified visibility,” according to Jon. “I can't imagine the pain of having to do that either with grep or OpenGrok,” he added.
Confidence spread from Jon to the rest of the team and throughout the company. With search contexts, Jon was able to share relevant contexts and queries, showing the team how they could verify whether a given Log4j instance was present or absent. He could show them precisely what they changed.
“We used Sourcegraph contexts to see specifically where a service was at any given point in time,” Jon said. Without Sourcegraph, the team would've had to use code scanning, which takes a lot of time, or manual build inspections, which aren't foolproof.
Nutanix was able to see where JMSAppender existed, fix it, and send out a release in less than 5 minutes.
Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability.
Nutanix was able to confidently identify every instance of Log4j across its sprawling codebase.
With Sourcegraph's help, Nutanix was able to transform a trust-threatening risk into a trust-building opportunity. Their customers, Jon explained, were worried about hundreds of other vendors, all of whom were likely affected by Log4j.
“That's hopefully something customers will remember us for,” Jon said. “We quickly either provided them with clarity or gave them a line on the next available patch because we were able to identify the issue and start fixing it ASAP.”
Jon explained that, at many enterprises, dependencies are unseen and forgotten. “You might check in on a dependency and find it hasn't been reviewed for four, five, six, or even ten years. But it works, so why update it?”
Log4j is one of many reasons why monitoring and updating is now a renewed priority. “Companies will have to be more diligent,” Jon said. With Sourcegraph, Nutanix is prepared to find and fix the next vulnerability.
Technical Director of Solution Engineering at Nutanix
Nutanix has 20,000 customers, an annual revenue of nearly $1.394 billion, and over 6,000 employees. Organizations around the world rely on Nutanix software as a single platform to manage any app at any scale for their hybrid multicloud environments.