Case Studies

Nutanix fixed Log4j quickly and confidently with Sourcegraph

Sourcegraph was the right product at the right time.
Jon Kohler, Technical Director of Solution Engineering at Nutanix
Nutanix

Challenge

  • Inability to efficiently find all the instances of Log4j in their large codebase.
  • Lack of code-level clarity that vulnerabilities were completely resolved.
  • Uncertainty about the scope and impact of the Log4j vulnerability.

Solution

  • Able to quickly and efficiently find every instance of the Log4j vulnerability.
  • Full confidence that all Log4j-vulnerable code was identified and resolved.
  • Able to quickly validate that no known vulnerabilities exist in the codebase prior to each release.

Results

  • Nutanix was able to see where JMSAppender existed, fix it, and send out a release in less than 5 minutes.
  • Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability in under 4 days.
  • Nutanix was able to identify every instance of Log4j across its sprawling codebase with 100% confidence.

As the Technical Director of Solution Engineering at Nutanix, Jon Kohler understands the complexity involved in securing the multitude of applications and solutions required to power such a large organization. “Security is something that we care about intensely here at Nutanix,” Jon said, “because it’s part of our bedrock. It's why customers like us, and we have to take it seriously.”

Log4j: The vulnerability that rocked an industry

In December of 2021, software companies around the world discovered that Log4j, an open-source logging library bundled in many software packages, contained significant vulnerabilities, one of which was a 10/10 on the CVSS scale. The Federal Trade Commission called the library “ubiquitous.”

Jon discovered that the offending module recurred throughout their build. “The more we dug,” Jon explained, “the more we realized this bug was everywhere and nowhere at the same time.”

Nutanix moved quickly, despite having multiple build and artifact management systems, as well as a large monorepo with many component branches and hundreds of git repositories. In under four days, Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability.

Nutanix used Sourcegraph to identify every instance of Log4j within 2 days

Speed was of the essence, but the timing of the Log4j news, which broke right before many employees go on vacation for the winter holidays, didn't make things easy.

However, Nutanix armed its engineers with Sourcegraph. Within a couple of days, a few Sourcegraph queries identified every instance of the Log4j vulnerability.

With the Log4j 1.x vulnerability, for instance, codebases were only insecure if they used JMSAppender. Jon used Sourcegraph to see where JMSAppender existed, fixed it, and sent out a release. “That took almost less than five minutes,” Jon said. Sourcegraph released a blog post that explained how other companies addressing Log4j could use code search for similar benefits.

This speed gave the team a head start on mitigation.

Deploying these fixes required quality assurance and testing as well as discovery and fixing.

With the help of Sourcegraph, Nutanix was able to release three back-to-back patches relatively quickly compared to other companies. Nutanix's customers reported satisfaction with both the speed of the patches and their quality.

“We tried to take a measured approach,” Jon said. “We wanted to get things done quickly but without completely flying by the seats of our pants.”

It's nice when you can just run a report and say, 'Here it is,' or 'Here it isn't.' It's much better than having to say, 'Well, boss, I think we got it all.'

Jon Kohler

Nutanix has renewed confidence in its vulnerability remediation

Tracking down the Log4j vulnerability was, in Jon's words, like “herding cats who were herding mice at the same time.”

Without Sourcegraph, Jon would have either been tracking down whoever built each component to ask them how and where they used Log4j or stumbling through all of the company's repositories.

Using Sourcegraph, Jon discovered every instance of Log4j and was fully confident in the results. “It's nice,” Jon said, “when you can just run a report and say, 'Here it is,'' or 'Here it isn't.'' It's much better than having to say, 'Well, boss, I think we got it all.’”

Nutanix needed that confidence because of its sprawling codebase. One thing that made Log4j especially complicated for Nutanix—as it does for other large-scale enterprises—is that there were multiple source control systems in play. Sourcegraph provided them with “unified visibility,” according to Jon. “I can't imagine the pain of having to do that either with grep or OpenGrok,” he added.

Confidence spread from Jon to the rest of the team and throughout the company. With search contexts, Jon was able to share relevant contexts and queries, showing the team how they could verify whether a given Log4j instance was present or absent. He could show them precisely what they changed.

“We used Sourcegraph contexts to see specifically where a service was at any given point in time,” Jon said. Without Sourcegraph, the team would've had to use code scanning, which takes a lot of time, or manual build inspections, which aren't foolproof.

5 minutes

Nutanix was able to see where JMSAppender existed, fix it, and send out a release in less than 5 minutes.

4 days

Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability.

100%

Nutanix was able to confidently identify every instance of Log4j across its sprawling codebase.

Log4j is the tip of the open-source vulnerability iceberg

With Sourcegraph's help, Nutanix was able to transform a trust-threatening risk into a trust-building opportunity. Their customers, Jon explained, were worried about hundreds of other vendors, all of whom were likely affected by Log4j.

“That's hopefully something customers will remember us for,” Jon said. “We quickly either provided them with clarity or gave them a line on the next available patch because we were able to identify the issue and start fixing it ASAP.”

Jon explained that, at many enterprises, dependencies are unseen and forgotten. “You might check in on a dependency and find it hasn't been reviewed for four, five, six, or even ten years. But it works, so why update it?”

Log4j is one of many reasons why monitoring and updating is now a renewed priority. “Companies will have to be more diligent,” Jon said. With Sourcegraph, Nutanix is prepared to find and fix the next vulnerability.

Jon Kohler

Jon Kohler

Technical Director of Solution Engineering at Nutanix

About Nutanix

Nutanix has 20,000 customers, an annual revenue of nearly $1.394 billion, and over 6,000 employees. Organizations around the world rely on Nutanix software as a single platform to manage any app at any scale for their hybrid multicloud environments.

Try Sourcegraph for free today

Experience code intelligence with a free 30-day trial of Sourcegraph for you and your team.

Explore other case studies

HashiCorp logo

HashiCorp streamlines cross-repository code search and fixes with Sourcegraph.

Read the case study
Codecov logo

Codecov uses Sourcegraph to resolve incidents 12 times faster.

Read the case study
FactSet logo

FactSet migrates from Perforce to GitHub.

Read the case study
Cloudflare logo

Cloudflare accelerates debugging and improves security.

Read the case study