About a year ago during my talk at the Nexus User Conference, and during a Virtual Session for RSA Conference APJ, I mentioned that a pipeline shouldn’t fail just because a security vulnerability was detected by scanning tools. That statement was met with a few record scratches in the audience but I still stand by the idea today. I think a deeper conversation on the topic may be necessary because my statement is the exact opposite of The Three Ways described in The DevOps Handbook and the Phoenix Project.
Breaking the Build, or Breaking Bad?
Knowing when to break a build is an interesting conversation to have when teams decide to adopt DevSecOps practices. In an effort to “shift left” as much as possible, teams don’t always consider that the product stakeholders can be adversely impacted when the pipeline stalls. Stakeholders come to an opinion that “shift left” means “break often”.
A few adverse effects may occur if a pipeline breaks in the wrong place. An application’s functionality should be available for either automated or manual testing or user acceptance testing. Security vulnerabilities, although extremely important, are in reality non-functional. They don’t alter functionality and really can’t be classified as a pure defect. It’s a subtle difference.
Key stakeholders should review progress and exercise the application as well. What happens when remediation takes longer than expected? Or when the feature delay backs up testing? Stakeholders left waiting for product delivery begin thinking that the engineering department isn’t productive. Fingers start pointing, and then we may as well not even be talking about DevOps.
In general, organizations will (or should) have defined vulnerability remediation SLA’s in their Governance and Compliance tools. For example, there may be SLA’s which state that Critical vulnerabilities need to be addressed immediately, High vulnerabilities within 90 days, Medium in 180 days, and Low vulnerabilities in 365 days. Even though these SLA’s seem extremely long, they are in use at larger enterprises today.
When the security organization sets SLA’s for vulnerability remediation then expectations need to be made clear to the product owners prioritizing work. There is a very real possibility that miscommunication at this point could have major cultural or financial ramifications. If remediation is documented, why hold up deployment for vulnerabilities lower than a critical severity? The question is an extremely difficult one to answer - especially if the business is not delivering.
What happens when the cost of cyber liability insurance is less than the expected cost of a fine imposed for HIPAA or PCI violations based on risk calculations?
Delivery is Not Deployment
Just because there is a security issue in a codebase doesn’t necessarily mean that subsequent tasks in a pipeline can’t be completed. In the case of a security finding, the pipeline should fail further along in the process – just before a production deployment.
Issue information and valuable measurement data can still be pushed left as soon as possible to the developers. Vulnerability remediation can still be tracked in a defect tracking tool during the delivery stages of the pipeline. When a critical vulnerability is detected developers should be immediately notified. Although there is no break, the build will not move past delivery and be deployed. The deployment is gated for the specific build instance.
Deliver to internal stakeholders -- don’t deploy to customers.
Where Do We Go Now?
When adopting DevSecOps it is extremely important to address culture and process before selecting any tools. When it comes to when and where a pipeline breaks, the idea needs to be shared and agreed upon by the entire team - security, development, operations, and others.
The best place to start is to review the existing Governance and Compliance documentation. (Or, if necessary, create one.) With organizational alignment and documentation, teams can benefit from a shift to DevSecOps and enact current technology and security trends without the downsides to approaches taken in decades past.
Shift left, shift right, or shift wherever needed. Be sure to shift intelligently.