10.     Establish mechanisms to monitor the effectiveness of your open source governance program

Your OSS governance program is in place. You’ve educated everyone on the policy, standardized components, built open source management into your development process and are continuously monitoring production applications.  You’re done, right?  Almost, but not quite.   You’ll want to set up check points to monitor its effectiveness to learn what’s working and what’s not.

A good place to start is by monitoring open source downloads from outside sources.  Just because a component was downloaded doesn’t necessarily mean it’s being used in an application, but if you see problematic components coming in,  it’s probably worth investigating.  If you’re using an enterprise repository manager (and we suggest you do), you’ll want to know if it’s being used or being circumvented.

You may also want to audit your key applications. It would be surprising to find problematic components, especially if the apps were developed after your policy was in place. But, if you did, it would be an indication that something went wrong. Maybe a development group was unaware of the new policy, or perhaps they don’t yet have the tools needed to effectively follow it.

A final place to monitor is code delivered by outside contractors, consultants, or ISVs. You’ll want to be sure that they are following your governance policies and haven’t inadvertently included components with security or license issues.

Now that you know what you want to monitor, the next question is how? That’s why we created Insight. Insight reports all of your organizations open source downloads.  You’ll also have tools for analyzing and reporting details of all the components used in your Java projects, whether they were developed internally or delivered by a 3^rd party.

You can learn more about Insight here.

9. Continuously monitor your production applications to learn of newly discovered defects

What happens during development if you learned of a critical flaw in an open source component you’re using?   Most likely, you would update to the latest version, regression test, and move on.

But what happens once the application has shipped or been put into production?  No one on the development team is likely to give a second thought to the application you just deployed. That’s typically how it works at most organizations. Unfortunately, this may leave you exposed to known security flaws or quality issues.

It’s not like component bugs stop coming up after you ship, right?  The open source community doesn’t stand still.  Projects are constantly being evaluated and improved. Bugs are discovered and fixed.  The majority of improvements won’t affect you at all, but every so often a critical vulnerability is found.  Updates will typically be released quickly.  Applications that include the updates will be safe.  Those that don’t will be vulnerable to exploit.

This is more than just a theoretical possibility. In March 2009, the United States Computer Emergency Readiness Team and the National Institute of Standards and Technology (US-CERT/NIST) issued a warning that the Legion of the Bouncy Castle Cryptography API was vulnerable to remote attacks.   A new version, free of known vulnerabilities was issued.  Despite this, almost two years later over 1,600 organizations downloaded the flawed version of the component – in a single month.

One way to approach this problem is to create a complete bill of materials for each application prior to release.  A team could regularly review the aggregated component list to identify newly discovered issues.  Unfortunately, there really isn’t a good way to monitor the steady stream of open source updates as the list grows to thousands of components across tens or hundreds of applications.  This manual approach just doesn’t scale reliably.

That’s why we created Sonatype Insight.  Insight analyzes and continuously monitors each of your applications, alerting you when critical updates are available.  You can learn more about Insight here.  To hear more about the Bouncy Castle vulnerability and how Insight could have helped, click here.

8. Build open source management into your software development process

We’ve found that many organizations wait until development is nearly complete before they run scans to verify that their open source policies have been followed.   What else have we found?  That developers find this maddeningly frustrating.   For example, if an application scan discovers that a component is unacceptably licensed, such as with GPL, you’ll have to find a replacement component, rework the code, and retest the application.  The project will cost more than expected and be delivered late. The disruption will likely impact the next project too as the team will be stuck on the original project longer than expected.

Frustrated developers often tell us that they’d much rather catch and fix issues during development rather than wait until the end. We agree.  Here are some of our ideas for ensuring open source compliance without disrupting development:

• During Design: Having access to security and licensing information for each component will allow you to validate them before including them in your project.  This sounds obvious, but it’s often tough to discover this information, especially when considering the hidden dependencies in Java components.
• During Development: Open source projects are constantly being updated.  Many updates can safely be ignored, but you’ll want to follow them all so you don’t miss any that address security or quality issues discovered by the community.
• During Build: The CI system is the ideal place to verify that all the components used meet your standards. You’ll catch issues as soon as they are introduced and be able to fix them quickly. Think of it as using the CI system to alert you of compliance issues just as it alerts you to quality issues today.
• In the Repository: Restricting the repository to approved artifacts is another method for ensuring that applications are free of defects.  Some organizations approve every artifact before allowing it in the repository (whitelist), while others allow developers to add any component by default and only restrict those with known issues (blacklist).

At this point you may be thinking that all of this sounds great, but would be really hard to implement in practice without automated tooling.  We agree with you, which is why we created Sonatype Insight.  Learn more about Insight here.

7. Establish a policy of service and support

When something breaks in an open source project, who do you call?  Software being software, there will be bugs.   Plan for it; it’s going to happen. And when it does, you’ll be glad you thought through your support plan in advance.

The complexity of the open source project, and the importance of your application along with your tolerance for risk will determine the type of support you’ll want. There are a few options to consider, including:

• Self-support.  For simpler and less critical components you’ll likely just need good online resources so that someone on the team can become your internal expert.  But this option contains hidden costs because when something breaks, you’ll need to spend time and attention resolving the issues.
• Community support.  Active open source projects have vibrant communities willing to lend a hand when you have a problem.   Projects typically have message boards and on-line defect tracking to facilitate this type of support.  This is typically a good choice for more complex components used in less critical applications.  Like self-support, you’ll still need to dedicate internal resources, but the community will have your back.
• Commercial support.  For the most complex and/or critical applications, commercial support is the logical choice.  Vendors like Sonatype have dedicated teams who can help you resolve problems quickly. They’ve seen all the common issues and have the expertise to solve more complex cases quickly.  Often they have have direct access to key project committers or help run the open source project so that your feedback and bug fixes can be quickly addressed. When researching support options, you’ll want to think about a few key things, such as support hours, response times, and how you’ll interact with the vendor (web, email or phone).  For your most critical projects, you’ll probably need a service level  agreement that guarantees minimum response times.

Once you’ve thought through your support plan you’ll be ready to deal with issues when they arise.  And arise they will – they always do.

6. Standardize on a common set of open source components

The number of versions of each component consumed by a sample organization. Each letter represents a different component.

There are over 30,000 unique components in the Central Repository – many of which perform the same function. It’s not surprising to find independent development groups within the same organization using different components to perform the same task.  It’s also quite common to see many versions of the same component being used. The following figure shows the actual version-dispersion for an organization downloading components from the Central Repository.  This is not at all out of the ordinary.

So, why should you bother to limit the number of components in use?

• Lower maintenance costs by reducing the number of components that need to be supported. Many organizations we work with follow each of the open source projects they utilize so that they’ll know when a critical bug or security flaw is found.  Most companies also typically need expertise and support for each component family (either in-house or outsourced). Fewer components mean fewer open source projects to follow and support. 

• Limit the number of components that need to be evaluated. Evaluating components can be time consuming even with automated tools.  If you are already using something that works in one project, why use something different elsewhere?

By standardizing on a set of open source components you’ll lower your costs and reduce your risks.  Standardizing can be challenging, but worthwhile if you use lots of components and work in an organization that has many critical applications.    We created Sonatype Insight to help you improve your management of open source components. Insight provides theinformation you need in your existing development tools to help you choose the right components.

5.  Evaluate open source components before using them in development

There are over 30,000 unique open source components available in the Central Repository to choose from. More often than not you’ll have multiple choices for any given need, making it hard to choose. Here are a few things to consider when evaluating components.

• Determine if the component meets your requirements for quality, security and licensing. You’ll want to be sure that candidates are appropriately licensed for your application and are free of known security vulnerabilities. License information is sometimes included in the project POM, but we’ve found that this field is often blank or does not match the licensing embedded in the source files.  If you want to be sure about licensing, you need to look at each source file in the project, including all dependencies.  You can find security information in a number of free online sites, including cve.mitre.org and osvdb.org.  Unfortunately these resources don’t utilize the standard Maven coordinates (GAV) and so can be difficult to search for Java components. 

• 

  Problematic open source components may be hidden deep within the dependency tree

  Analyze the component’s dependencies to discover hidden security or license issues. For each open source component you include, there are often dozens of other components it depends upon.  Any of these included components may have a hidden security risk or be licensed inappropriately for your application.  The figure provides an example of this situation.  The complexity of Java dependencies make it difficult to know what your applications are truly made of without using automated tools.

• Ensure you have a trusted source for each component, such as the Central Repository. You’ve picked the components for your application and verified that they meet your quality, security and licensing needs.  Now you need to download the components from a trusted source to ensure what you download hasn’t been modified maliciously.  You’ll want to choose a source, like the Central Repository, that includes a PGP signature for ensuring the integrity and authenticity of components.  You can read more about Central’s use of PGP signatures here.

Following these guidelines will help you choose better components and avoid risk.   This work is not always easy, but it’s important when you’re building critical apps. We created Sonatype Insight to help you perform these tasks faster and more efficiently. Insight provides the component information you need in your existing development tools.

4.     Start with a pilot program

A small pilot project lets you validate all the elements of your program before rolling it out to the whole organization. The success of the governance program requires new processes, new technology, and the buy-in of key stakeholders. It’s much easier to work out the problems in a small, focused pilot than in the midst of an organization-wide deployment.

• Start with a few groups of developers and a few key applications. There is no need to attempt an enterprise-wide roll out immediately as this just adds risk and expense.   Keep the ultimate goal in mind, but get there in a series of small steps.
• Develop a plan to move from pilot program to department to enterprise-wide based on specific success criteria.  Identify both the key objectives and stakeholders for each stage.  Don’t move on to the next stage until you’ve met your success criteria.  It’s better to repeat the process a few times in the beginning in order to prevent problems later.
• Ensure policies are both enforceable and non-punitive to ensure acceptance and adoption. For policies to be truly effective, you must have an automated way to monitor and enforce them.  Yes, everyone wants to “do the right thing”, but when under pressure to “just ship it,”  teams may take shortcuts.  We recommend instrumenting key development stages to automatically catch policy violations. For example, the build system could report and automatically fail builds that contain problematic components.  You might also require that projects be validated prior to promoting them to the quality assurance team or ultimately to production.

That wraps up today’s tip.   In our next post, we’ll talk about a technique you can use to choose components wisely.

In the meantime, check out Sonatype Insight.  Insight helps you build better software faster without unnecessary quality, security, or licensing risks and without disrupting your development process.  Learn more at www.sonatype.com/insight.

3.     Establish an open source governance program

In Critical Strategies to Manage Risk and Maximize Business Value of Open Source in the Enterprise, Gartner Research Vice President Mark Driver notes “Above all other considerations, the primary factor in balancing risk versus reward from open- source-software (OSS) assets hinges on the successful execution of an enterprise open- source governance program.”  Yet, Sonatype’s 2011 developer survey (see figure below) revealed that 87% of organizations did not have an effective policy in place for choosing open source components.

Some things to consider when developing your policy:

87% of organizations do not have an effective policy in place for choosing open source components

• Include guidance on quality, security and licensing. You’ll want to be sure to cover all three areas as a problem in any one of them could negatively affect your organization.  As licensing issues can be confusing to those new to this space, we’ve prepared a brief primer that summarizes the key issues.
• Design a measurable policy that works for application development as well as legal, risk management, and security.  Your policy needs to take into account the needs of various groups – including application development, operations, security, and legal compliance. But, if it doesn’t work for developers it won’t work at all. If the policy introduces time-consuming controls or bureaucracy on the development team, it will likely be ignored or actively avoided. On the other hand, if the policy does not adequately address open source issues, you may end up with unwanted risk even though everyone follows the policy.

That wraps up today’s tip on getting started with open source governance.   In our next post, we’ll talk about how to get started with your program.

In the meantime, check out Sonatype Insight.  Insight helps you build better software faster without unnecessary quality, security, or licensing risks and without disrupting your development process.  Learn more at www.sonatype.com/insight.

We’ll be exploring each item in more depth through a series of five blog posts.   But for now, let’s start at the beginning with understanding your current usage of open source components.

1.  Understand where you are.

• Analyze your consumption to understand what and where components are being downloaded. By doing this you’ll know how open source components are used throughout your organization for development and which development groups are the heaviest users. Many IT organizations are surprised at the level of open-source penetration that has occurred “under the radar”.
• Identify problematic components currently being used in development projects. If your organization doesn’t yet have an open source policy, or at least one that’s followed, then it’s likely at least one group is inadvertently using components with license or security issues. Completing this review will give you a good idea of how well (or poorly) your organization is doing.
• Classify existing projects based on business importance to establish the role of OSS within the enterprise’s existing software portfolio. You’ll typically want to introduce new processes to a few projects at a time before rolling them out to the whole enterprise.  As any change can be disruptive, you may want to first try out your new procedures on less critical projects to avoid disruption. Once proven, you’ll want to quickly implement them on your most important projects to realize the benefits.

2.  Analyze your key production applications for security vulnerabilities and licensing issues

You need to first understand where you stand before you can make informed decisions as to what changes, if any, need to be made. We recommend the following steps:

Java dependencies are complicated and can introduce hidden risks

• Examine the complete bill of materials for your applications, not just first level dependencies. Java’s building block approach makes it fast and easy to build new applications using open source.  But at the same time, this makes it difficult to identify all the dependencies as these may be nested several levels deep. Unfortunately, what you don’t know could indeed hurt you as illustrated in the figure.
• Analyze new and existing applications  – it’s never too late to find and fix issues. Your existing applications, finished before you had a complete open source governance program in place, are likely to contain hidden risks.  New vulnerabilities are being discovered all the time, so even if your development team carefully vetted every component before choosing them, it’s likely that new issues have been discovered since you put the application in production.

While its possible to manually implement these tips, you’ll really want automated tools, especially if you are part of a large organization.  While you could cobble together your own tools, we recommend you check out Sonatype Insight, a set of tools and services we developed to help our customers address these very issues.

Once you implement these two tips you’ll have a really good idea of where you stand in terms of open source governance.   In the next post we’ll provide tips on how to start up your governance program.