The 2020 State of the Software Supply Chain Report is available!

Study Shows High-Performance Dev Teams Fix OSS Vulns 26x Faster | Press Release

blog-logo Sonatype Blog

UPDATE: 21 SaltStack Breaches with 2,900 Still Vulnerable

May 31, 2020 By Derek Weeks

UPDATE for May 31, 2020:

We first published this story over a week ago, but adversaries don't rest.  On Friday, Cisco announced that they have discovered SaltStack compromises on six of their salt-master servers -  part of the Cisco Internet Routing Lab Personal Edition service infrastructure.  This brings the total number of breaches for this open source application to 21 for the month of May.

If 21 breaches in a single month feels daunting, wait for what June may bring.  A recent update from Censys warns that over 2,900 internet-facing SaltStack instances are still vulnerable.

  • On May 1, Censys found 5,841 exposed and likely vulnerable Salt servers connected to the Internet.  
  • On May 6, that number went down to 3,722 Salt servers exposed - a 36% reduction in just 5 days. 
  • On May 12, the number stands at 2,928 Salt servers still exposed - a 21% reduction from last week, and a 50% reduction overall since the CVE was announced.

Here is our updated timeline of the breaches:

ORIGINAL POST from May 18:

Since 2017, I’ve reported in our annual State of the Software Supply Chain Report that the average time between an open source vulnerability being announced and subsequently exploited was three days.  We saw this exploit time with Equifax, GMO Payment Gateway, Canada Revenue Agency and several more.  

Then our CTO, Brian Fox, started chronicling a disturbing turn of events that showed that a shifting landscape of attacks based on OSS consumption was emerging. In the past three years, we’ve seen a consistent increase in open source and supply chain attacks that make one thing clear: adversaries are not slowing down.

Twenty more open source breaches occurred in the first three days of this month. The opportunity for adversaries emerged when SaltStack revealed two critical vulnerabilities in its code base (see: CVE-2020-11651 and CVE-2020-11652).

Salt - a PyPI-based open source project for event-driven IT automation and configuration management  - urged its users to update to newly released versions quickly or be at risk of a breach. As one security researcher involved with discovering the vulnerability wisely observed, this was a “patch by Friday or be compromised by Monday” situation.

What follows is not a critique of SaltStack, as they made all of the necessary moves to protect their open source community. It is a reminder of the strategies we can all take to protect ourselves. The SaltStack breaches serve as a textbook example of our three days to breach scenario.

Blog Post Speed of Adversaries (SaltStack)

SaltStack Breaches: What Happened

A vulnerability was discovered in SaltStack’s open source configuration framework, available as a PyPI package, on March 12 by a security research team at F-Secure. Through a proper coordinated disclosure process, SaltStack confirmed the vulnerability March 24. During the disclosure process, F-Secure had also informed SaltStack that 6,000 Salt Masters were publicly exposed and at risk of compromise.

On April 23, SaltStack published a notice informing users of the vulnerability and forthcoming patch on April 29 - a critical “heads up” for their community. Then on April 29th SaltStack released the fix and responsibility disclosed the CVEs. The same day F-Secure sent an advisory urging upgrade, saying, “We expect that any competent hacker will be able to create 100% reliable exploits for these issues in under 24 hours.” 

They were right. Anyone who missed the notice, did not have auto-updates enabled for SaltStack, or were unable to manually update their SaltStack code quickly were at risk.  Subsequently, the SaltStack vulnerabilities led to 16 breaches on May 2nd - just three days after the vulnerabilities were announced. Four more breaches were recorded the following day. 

The press immediately reported breach confirmations at LinageOS, Ghost, Algolia, DigiCert, and Xen Orchestra.  Sixteen more breaches were recorded by Salt users on GitHub that included installation of cryptocurrency miners, unknown programs executing, and firewalls being disabled.

How to Protect Open Source Software Components in Your Software Supply Chain

Every open source project with known vulnerabilities is susceptible to attack.  This is even true of those with known safe versions that have been fixed to thwart attacks. Just last week, the Cybersecurity and Infrastructure Security Agency (CISA), the Federal Bureau of Investigation (FBI), and the broader U.S. Government revealed their 10 most exploited vulnerabilities list which still includes the Struts vulnerability from 2017. 

One of the laptop stickers we give away at community events advises DevOps and AppSec pros to automate faster than evil. Automation -- of surveillance, of notification, of remediation -- is the best, proactive defense against adversaries. How do you do this? Here are specific recommendations. 

Have an SBOM and a Plan

When a vulnerability is announced in an open source project, you should be asking two questions immediately: have we ever used that open source component, and (if yes) where is it?

A full inventory of open source components being used in an application is referred to as a Software Bill of Materials (SBOM).  Those organizations who keep up-to-date SBOMs for their applications know the answers to the two questions above.

According to our 2020 DevSecOps Community Survey of over 5,000 developers, only 45% of organizations with mature DevOps practices keep a complete SBOM for their applications.  Even more worrisome are the 26% of organizations with immature practices that keep a complete SBOM; this means 74% of them would not know if they had used or where to find newly announced vulnerabilities in open source components across their environments.

Blog Post Speed of Adversaries (SaltStack)-1

According to Gartner’s Technology Insight for Software Composition Analysis report, “By 2024, the provision of a detailed, regularly updated software bill of materials by software vendors will be a non-negotiable requirement for at least half of enterprise software buyers, up from less than 5% in 2019”. The report goes on to say, “By 2024, 60% of enterprises will automatically build a software bill of materials for all applications and services they create, up from less than 5% in 2019”.

There is no substitute for preparation. Facing adversaries requires stone cold realism about your risks, optimism that your processes and systems will protect you, and the right tools to respond quickly. 

Invest in Software Composition Analysis 

A key solution to have in place is Software Composition Analysis (SCA). SCA focuses on OSS governance throughout the SDLC. SCA supports modern software development teams by scanning applications to identify open source components, and by creating the aforementioned SBOM. Ultimately, SCA tools surface risk using metadata about overall component quality (vulnerabilities, licensing, architecture, etc.). 

SCA’s primary use case is automating OSS governance for DevOps and AppSec teams, but becomes especially useful when a vulnerability is discovered and speed of identification and remediation are critical.  SCA as a practice is not new to development teams; the 2020 DevSecOps Community survey revealed that 58% of mature DevOps practices already have SCA properly integrated into their development pipelines.

Blog Post Speed of Adversaries (SaltStack)-2

For hundreds of Sonatype customers using SaltStack that already have our Nexus Repository or Nexus Firewall solutions in place, notice of the new vulnerabilities have already been made available.  For example, our analysis of Salt version 2016.11.6 from PyPI shows the following:

image (37)

Organizations using the Repository Health Check feature within Nexus Repository would be alerted to issues with this application package in their daily reports.  Furthermore, Nexus Repository or Artifactory customers who have deployed automated security policies through Nexus Firewall would also be alerted to vulnerable downloads of Salt when governance policies were violated.

Having SCA matters during times of threat. Organizations that control workflows have faster response times. They also produce more secure software and happy developers. Simply stated: organizations without SCA are at significant disadvantage when open source vulnerabilities are discovered compared to those who are prepared. 

Stay Several Steps Ahead

It is impossible to know every angle an adversary might take to compromise your software supply chain. However, you can increase your protection by automating as much as you can, as early as you can, in the SDLC using Software Composition Analysis tools. 

As the SaltStack exploit demonstrates, it is crucial to understand:

  1. Which open source components are in your software supply chain
  2. The status of each component, particularly if a new vulnerability is discovered
  3. The whereabouts of each component, so you have the best chance of remediating vulnerabilities before exploits happen

Be proactive. Construct a plan, use an up-to-date SBOM, and implement SCA tooling for best protection. 

If a new vulnerability were announced today in an open source component that your organization relied upon, what would your next three days be like?

Tags: vulnerabilities, AppSec, SCA,, News and Views, SBOM

Written by Derek Weeks

Derek serves as vice president and DevOps advocate at Sonatype and is the co-founder of All Day DevOps -- an online community of 65,000 IT professionals.