Part 2: How Your Software Is Like a Car – A Closer Look at Today’s Software Supply Chain


April 30, 2014 By Wayne Jackson

Part 2 of 3 in a Blog Series: ‘How Software is Like a Car’ 

Part 2: A Closer Look at Today’s Software Supply Chain

In part one of my blog, It’s Just the Way Software is Made, I discussed the realities of how software is made, the birth of agile development, and the advent of component-based software development. Today, we will drive down the software supply chain to understand where your software is really coming from. I’ll also discuss why it’s important for us to instill high quality standards and governance policies in our “parts” ecosystem.

The Next Wave: A Fly In The Ointment of Supply Chain Management

Building quality cars requires sourcing high quality components. But the best quality components don’t simply arrive on the receiving dock and work as anticipated. Toyota understood this to an extreme. They placed expectations on their suppliers to deliver high quality parts, they tested those components to ensure they delivered the proper outcomes, and they empowered people on their production lines to report any concern or suggest any improvements to those parts or the assembly process. It was not just a policy that Toyota’s manufacturing executives put on paper, they made sure that the policy met practice all the way down to each worker on the production line, each component used in the cars, and every process undertaken to deliver their vehicles to market.

While software development practices have begun to mimic Toyota’s lean manufacturing, a closer look at software’s supply chain management practices, reveals a fly in the ointment. By comparison to Toyota’s supply chain management practices, modern software development supply chain is, for the most part, complete chaos. Developers are selecting components (“parts”) based on arbitrary criteria, most likely whatever they or their buddy used the last time.

Human-Speed Monitoring Can’t Keep Pace With Development

The parts of organizations that are trying to assert control over the development practices (e.g., Open Source Review Boards, IT Architects, Application Security teams) are doing so with manual review processes and paper-based policies that are too far removed or out-of-step with developers and the time-to-market expectations they are driven by.

The sheer volume and variety of open source components sourced into the typical software development supply chain is enormous. What the review boards, risk and compliance groups, and application security teams are coming to realize is that their manual review practices, approval policies, and observations have no earthly hope of keeping pace with the volume, complexity, and pace of change in the open source ecosystem.

Let’s bring the scenario more down to earth. Dealing with thousands of new open source components being sourced by a given developer each year would be analogous to every production worker on Toyota’s line sourcing each tiny car part they need with little governance over those choices. Imagine Toyota letting their assembly line workers choose whatever suppliers they prefer. Imagine Toyota letting them use outdated parts that were known to be defective.

No Chance for the Software Equivalent of a Recall

Bill of MaterialsTo my amazement, most organizations don’t even keep a bill of materials for the applications that they deploy. Without one, critical security (or performance, or stability, etc.) disclosures go ignored. There is no chance for the software equivalent of a recall, because the organization hasn’t kept track of the parts (including version numbers) being used in production. While not perfect, the auto industry has clear visibility and records for every part used in their vehicles so they can organize industry-wide recalls when necessary.

By comparison, today’s open source community can identify vulnerabilities in components and post notice of those issues in a central database, but the proliferation of communications, practices to stop using those faulty parts, and the ability to track down which components have been deployed is nearly impossible in today’s software industry. The word-of-mouth and manual practices used today, like those recently employed to alert businesses of the April 2014 Heartbleed Open SSL bug, are far from comprehensive and cannot keep up with the current pace of application development. Automation within the software supply chain management is long past due.

In part three of my blog, I will share a real world story on how thousands of corporate and government organizations sourced one “bad part” last year. Even worse, they sourced that open source component five year’s after the vulnerability had been disclosed.