In many ways, their story is one we’ve heard before. Their struggles showed the need for change: long lead times for software delivery; software quality issues were found late in the game; many handovers and approvals dominated the process; inefficient cooperation between dev and ops; late code merges; and, large, non-frequent releases to production.
Something needed to change. Enter DevSecOps.
ABN AMRO has numerous software delivery pipelines to manage. While this magnifies the effort to implement CI/CD, it also magnifies the benefits. Additionally, the more pipelines you have, the more security risks you have - hence the pressing need to implement security into their DevOps practices.
So, how did they go about including security into DevOps? To start:
- Secure coding/open source libraries
- Hybrid cloud and container security
- Credentials management
First, they needed address open source software risks. Open source software libraries are invaluable. Yet, they come with risks. If libraries become outdated, your applications could become vulnerable. Stefan and Wiebe addressed this with standard Continuous Integration (CI) pipelines and build breakers. If a developer is delivering unsecure software or implementing unsecure open source libraries, the Jenkins build will break and the developer is forced to fix the issue.
In the past, as with many organizations, there were lots of awareness efforts and discussions. While this helped, after they implemented build breakers, more issues arose, highlighting the fact that a few discussions weren’t enough. They needed true buy-in from developers. After taking the time to make this transition a priority - the company has more commitment, broader awareness, and deeper understanding of why open source governance is so important. The quality gates and build breakers implemented forced developers to become more aware, and issues started getting fixed quicker.
After their initial implementation - where do things stand?
- An updated, and mostly adhered to, open source policy
- Use of Micro Focus’ Fortify and Nexus Lifecycle together
- Included automated on-boarding pipeline and security scans in standard Java, FrontEnd, Mobile, and Microsoft pipelines
- Conduct application security training and awareness sessions
What’s next for ABN AMRO?
- Provide CI/CD metrics dashboard to visualize security issues per grid/domain, both for security issues in development and production
- Track progress via senior management meetings
- Increase security awareness via senior management
- Reward teams who have the right focus on security
They have also implemented a hybrid cloud strategy using IBM CMS for their private cloud and a combination of Azure and AWS for their public cloud. They use a cloud-native approach to harness the full advantages of the public cloud’s Platform as a Service so developers can focus on developing the custom applications.
Inherent in sound Continuous Integration/Continuous Delivery (CI/CD) practices are containers, which also have to be secured. Stefan and Wiebe use Docker ES to secure the Docker engine, and then for containers running: run-time scanning; scanning images on build; and, syntax and security checks at code level. Their Docker image pipeline runs on Jenkins Enterprise on AWS, and Jenkins is on containers too.
Finally, Stefan and Wiebe address credentials management - a huge vulnerability for many organizations. They cite a report that 75% of organizations do not have a privileged account security strategy for DevOps, and they mention some high profile breaches caused by poor credential management: Uber, Vine, and Ashley Madison.
They remind us that you have to know where your secrets are - or you don’t know where they are being exposed, and they suggest focusing on these areas to improve credentials management:
- Key rolling
- Granular access permissions
- Secure storage
- Detailed audit logs
- Must fit seamlessly in the DevOps environment
Stefan and Wiebe are seeing the benefits of a well-rounded, and well executed DevSecOps program. It’s a story we’d love to see more of - and a story that could be yours. Listen to what they have to say in their own words here. You can view all sessions from the 2018 Nexus Users’ Conference, held in June, are here.