Another thing we’ve been saying is that it is our responsibility, as developers, to start paying attention to security vulnerabilities, and if we don’t take responsibility for application-level security, someone else will impose this requirement on us…

…and that’s exactly what’s we’re seeing both in the EU’s reform of Data Protection Laws and as the US Congress responds to the latest data breach at LinkedIn. Now, who knows what sort of regulations we’re going to see in the coming months, but one thing is sure, the fact that lawmakers feel compelled to act is proof that we’re not doing enough as an industry to address security.

The best security is a layered approach: multiple levels of network security, security policies for production resources that limit access to individuals that need it, secure password policies, and application security. Sonatype’s focused on that last item, application security, and our approach focuses on the components you assemble to create your applications. If you develop software today, you understand that much of your work is spent creating applications that sit atop frameworks like Spring and Hibernate. It isn’t enough for your own software and infrastructure to be secure. These days, you need to account for vulnerabilities in your dependencies.

And, again, this isn’t operation’s responsibility. Security is a shared responsibility across both development and operations. This is something that developers need to take ownership of. While we’ll probably never know how sites like LinkedIn, eHarmony, and Last.fm were compromised, there’s a good chance that some of these sites were compromised via known vulnerabilities in outdated components. Components like Tomcat or frameworks like Struts are among the list of artifacts that have known problems.

Don’t get hacked because you didn’t upgrade to the latest version of Tomcat or because you happened to be using some ancient version of Spring with a known vulnerability. If you are consuming artifacts from Central (and if you are a Java developer, you probably are), you need to start using Nexus Professional to keep track of your dependencies. If you are using Hudson or Jenkins, take some time to evaluate Insight for CI.

“Nearly two-thirds of security leaders say their senior executives are paying more attention to security today than they were two years ago, due in large part to media attention.” and “60 percent of the advanced organizations named security as a regular boardroom topic, compared to only 22 percent of the least advanced organizations”

Instead of simple three-tiered applications following a standard Apache -> Tomcat -> RDBMS pattern, today’s scaleable applications involve a portfolio of technologies: Redis, Hadoop, real-time BI systems, integration with 3rd party APIs, Node.js, with more and more companies adopting a portfolio of technologies. It is becoming increasingly difficult to draw a line around a particular application and evaluate security vulnerabilities in isolation.

Today, you need to have your security group sitting next to you evaluating a complex application as it evolves…. but, back to the article, it isn’t just the evolution of technology that is making security a focus for business, it is a series of high-profile, embarrassing data breaches. A CEO that wouldn’t have thought very much about security technology a few years ago, sees what happens to a Stratfor or Global Payments and they understand the risks. Data security is front and center in the news, and a data breach can be a business-ending event.

So get out in front the problem. Start tracking your application dependencies and identify known vulnerabilities with Insight.

When we launched Nexus Professional and integrated Sonatype Insight information we gave you the ability to keep track of your overall exposure to security vulnerabilities. Your IT organization gained a window into the intersection of known vulnerabilities with the artifacts you download from Central. That was a good start, but the real benefit is Insight for CI. We launched Insight for CI this week, and it’s the tool you’ll want to use to address security vulnerabilities in specific products. If it is your responsibility to keep up with security, one of the easiest ways to take a more proactive approach is to start using Insight for CI to track your application’s dependencies.

Click here to get started with Insight for CI. It works with either Hudson or Jenkins, and it covers both license and security information.

Important projects, projects like Apache httpd, Apache Tomcat, JBoss, or Spring all have something in common. Each of these projects has a formal security team and a process for addressing security flaws. The presence of a security team in an OSS project is a signature of how seriously a project takes vulnerabilities. If you are evaluating projects for security, you should evaluate a project not by the number of vulnerabilities you identify with a tool like Nexus 2.0, but on the process a project uses for addressing security risks.

Just because a project has a large number of vulnerabilities doesn’t mean this project is insecure. Take, for example, Tomcat.

An Example: Is Tomcat Insecure?

If we were comparing OSS projects on the number of vulnerabilities alone, Tomcat would rank very high on the list of projects with known vulnerabilities. Does this mean Tomcat is insecure? No, on the contrary, the Apache Tomcat project sets the gold standard for how to respond to security issues. They have a dedicated security team that quickly responds to vulnerabilities and they regularly produce detailed reports that identify how specific commits relate to specific vulnerabilities. Yes, there are a high number of vulnerabilities, but you couldn’t ask for a better response.

Components like Tomcat, GWT, Spring top the list of security vulnerabilities because these projects are so widely used. No, Tomcat is not insecure, but here’s the nuanced point we’ve been trying to make for a few months. Tomcat is very insecure if you are not paying attention to this constant stream of vulnerabilities and fixes. You can evaluate Tomcat for security, conclude that they take security seriously, install Tomcat 6.0.5, and then stop paying attention to these updates. Several months later you will be vulnerable to the recently identified Hash-based DOS attack and some exotic AJP vulnerability. If you don’t pay attention, all of this security effort is for nothing.

Tomcat’s Security Depends on Your Paying Attention

The effective security of a project is a function of several factors:

• How widely used is the project? – A larger ecosystem for an OSS project will result in more eyeballs reporting flaws and more interest in quickly addressing vulnerabilities.
• Does the project have a mature security effort? – Going back to a project like Tomcat. Does the project accept vulnerability reports? do they address vulnerabilities on a private list? and how quick are they to fix security bugs?
• Are you paying attention? – If you are not paying attention, a project’s investment in security is worthless to you. If you use Tomcat and you don’t pay attention to the Tomcat security team your own security will degrade over time. (You might as well just not evaluate security at all.)

If you want to start taking application security seriously, you can take a first step by starting to scan your dependencies with the Nexus 2.0 Repository Health Check.

Here are some of the highlights from Wayne’s presentation followed by the video of his talk and his slide deck:

• “The benefits of ‘many eyeballs’ in open source does create better software but you can only leverage that if you know about it. That’s particularly troubling in the context of the fact that more than 80% of the modern software application is [comprised of] open source and the components that are used to build those applications are surprisingly complex.”
• “That complexity is compounded by the fact that when issues arise their implications are viral and the big problem is that when those issues are resolved in the root components the solutions are not [similarly viral] . Spring Beans 2.5.6 compromised 1400 open source components and God knows how many downstream applications. When Spring Beans 2.5.6 was fixed, none of the others were fixed.”
• “You can imagine the ripple effect of compromising open source. And the combination of things like the lack of notification infrastructure and the complexity of open source componentry is how you get situations like this. 6,982 organizations including the Dept of Homeland Security and several financial institutions are still using a 3 year old crypto library with an “as bad as it gets” Level 10 flaw that has known exploit code.”
• “Sonatype is creating an extraordinary infrastructure for finding out everything knowable about a given component. So that when flaws are discovered, we can know and we have the ability to deliver that knowledge into the tools that developers are using every day. This family of technologies is called Insight.”
• “Critical to that is the Central repository. Central houses hundreds of thousands of components from nearly every open source project in the world and it is used by tens of thousands of organizations.”

I just wanted to reiterate the key point of yesterday’s security brief which is: “You and everyone else in the world are likely downloading vulnerable components.” If you don’t believe me, then take a look at this graph:

First, note the logarithmic scale – downloads over an entire year. Then, take a look at the left-side of the chart. See anything familiar? GWT, Spring, Struts, CXF, Xerces? If you use these components, you should try to identify which versions are affected by widely known CVE vulnerabilities. It’s that simple, if you use these components it would be a good idea to browse the CVE database, or to take a look at Nexus Professional’s Repository Health Check.

Really, attackers aren’t going to go to the trouble…

Developers, you might be thinking, “an insecurity in GWT or Xerces, who’s going to trouble of doing that much research? Who’s really going to hack into Megabank via some obscure AJP vulnerability in a Tomcat connector?” And if you are asking these questions as a way to shuffle this all under the rug, I understand. There’s enough work in the pipeline already and you don’t need another thing to worry about. As developers we’re not going to turn into security professionals overnight, but we can start using tools like Nexus Professional to help identify vulnerable components and isolate us from deploying known security problems to production.

It isn’t the likelihood that someone will hack GWT that is the issue, it is the idea that deploying any code with a known security vulnerability needs to be identified as a disqualifier. The idea that if you get compromised and someone realizes that it was a known vulnerability (for years): developers need to be motivated to avoid this embarrasing situation. The point I’ve tried to make on this blog is that we (developers) are not really paying attention to this problem because we just assume that it is someone else’s problem.

Ignoring Security: It isn’t a question of if you’ll get hacked, it’s when

The issue of data and systems security has repeatedly been front-page news time and time again over the past year. Groups like Anonymous and Lulzsec made a public sport in 2011 of hacking into serious organizations and making every effort to embarrass and ridicule them for lax security. The last few years have been pretty embarrassing years for a lot of security departments at large corporations and a few governments. 2012 promises to be even more active with McAfee predicting the reorganization of Anonymous, but focusing on these high-profile, news-generating events ignores the scope of the problem. It isn’t about volume, it is about your exposure to this risk.

I’ve seen some recent attacks in action. Attacks on both Java-based web architectures and PHP-based architectures. While it’s true that PHP-based applications present a much larger and more insecure surface area to attack, it has to be said that Java-based web applications and .NET present a much more lucrative target. An attacker can compromise all the two-bit Drupal instances in the world without stumbling upon anything worth intruding, or they can focus on a multi-month strategy of social engineering and direct attacks to compromise one the Global 100 financial institutions that are downloading insecure dependencies every day.

Welcome to the Security Theater

If you are banking on the fact that attacking Struts 2 or Log4J is just too esoteric for most hackers to do, you are participating in something Bruce Schneier calls Security Theater, and that’s really what I’m taking away from this study. Some of these institutions are so invested in presenting an image of trust and security that they will spend millions on Super Bowl ads and marketing efforts to purchase customer trust. But, at the end of that day they continue to download vulnerabilities. It doesn’t match up, we need a change of culture in development and security needs to be top of mind.

It’s time for developers to start taking security seriously. You could choose to be proactive about the problem and use tools like Nexus Professional to automatically correlate CVE vulnerabilities from CERT with your artifacts, or you can wait until someone replaces your company website with a funny picture and lose the ability to download artifacts from Central altogether. The choice is yours.

• Global 500 organizations downloaded more than 2.8 million insecure components in one year.
• Financial services firms are the most exposed: Global 100 financial services firms alone downloaded more than 567,000 insecure components in one year.
• 48% (a little under half) of organizations don’t have an inventory of Open source software used in production. (If there’s a new vulnerability discovered in something like GWT, who knows if we have that in production.)

To access the executive brief, “Addressing Security Concerns in Open-Source Components,” visit www.sonatype.com/securitybrief. You can follow the conversation on Twitter using the hashtag #OSSsecurity.

NOTE: Now, Developers, I know what you are thinking, you see the word “Executive Brief” and immediately dismiss this as C-level corporate-speak. Sure, there’s a little bit of that, but you’ll also learn how to own any unpatched Struts 2 application with a known vulnerability. If you use Struts, maybe you should read this report before your boss uncovers a vulnerability in your application?

There’s a good chance that you might not be focused on this problem.   You might not be constantly evaluating your project’s dependencies to analyze the risks.

I’ve been developing enterprise software for years, and it just isn’t something most companies worry too much about.   While a company might spend a great deal of money on systems and personnel to keep operating systems patched and networks secured, that same company is likely using an older version of Commons HttpClient 3.1 that presents a denial of service (DoS) vulnerability.    In other words, we appreciate the vulnerability of machines and operating systems while simultaneously ignore the security characteristics of the software that runs on these platforms.

As open source becomes more important to the modern enterprise this exposure will only increase.    The critical question to ask yourself given the increasing rate of change in open source is “can you keep up?”.

It just so happens that we recently launched Sonatype Insight to help with this very issue. Watch this short video to see how Sonatype Insight. can help you keep up.

Learn more about Sonatype Insight.

Problem/Anti-pattern: Too Many Logins

No matter how mature your process is, I’ve always found credentials to be especially difficult for a new hire.   It takes days, and then, even when you think you’ve created all of the necessary passwords, there are those other systems that jump up from out of nowhere.

“Do you have a login for Subversion?  No.  Talk to Tom about that?   How about JIRA?  I’m not sure who setup JIRA, let me check on that.   Ok, we have this VPN, and you are going to need to get those credentials in order to check your email which, unfortunately, is a whole different set of passwords.”

The first few days of employment are meeting people, downloading Gigabytes of software, and remembering an encyclopedia of passwords. Some of these password might be managed by HR, others are managed by your Technology department, but this post focuses on those credentials that affect development infrastructure.  If your development infrastructure is mature, your organization might solve this by consolidating authorization and access control information on an LDAP server or Atlassian’s Crowd.  If you don’t use one of these products, then you are dealing with an expanding constellation of moving parts: JIRA, Subversion, GitHub, Basecamp, Bugzilla, Confluence, Twiki, Matrix, CruiseControl, etc.

Complicating matters further is the fact that some enterprises are branching out into the world of hosted or distribute development infrastructure.   If you are working at a business that is experimenting with services like GitHub, Basecamp, or a hosted Atlassian stack, you’ll understand that this complicates things.   If you’ve already standardized on LDAP or Crowd, many of these hosted services do not have any provision for tunneling back into your VPN and authenticating against an internally managed authorization and access control server.

First Step: Use LDAP or Crowd

The first step is easy, use LDAP or Crowd.   Consolidate all of your credentials and select development infrastructure (like Sonatype Matrix and Sonatype Nexus) that can be easily integrated with these services.  Sonatype used to use OpenLDAP to manage this issue, but we switched to Atlassian’s Crowd server as it offered a cleaner interface.   If it takes more than one hand to count the number of developers in your group, it is probably time to invest in a similar solution.

Second Step: Use a Password Wallet

How do you deal with hosted, distributed infrastructure?   What if you are working on an open core product when portions of the system are stored on external source control systems?   What if you are using GitHub for some components?     The answer here is that there really is no way to get to a single password for everything.   The reality of today’s development infrastructure is that not everything can talk to your LDAP or Crowd server, it just isn’t realistic.  In these cases you will want your developers to use a password wallet.

Sonatype Professional’s Developer Onboarding tool provides just this structure to consolidate all of these credentials in a Security Realm.    When a Developer materializes an Eclipse workspace using Developer Onboarding, Sonatype’s Installer asks for all of the credentials and passwords up-front and stores these credentials in secured storage.   This serves a few purposes.

First, we ask for the credentials up front.  You can’t complete a materialization unless you can connect to the necessary resources.  This prevents a developer from starting a process and realizing half way through that they don’t have the appropriate access.

Second, we store this information in Eclipse in secured storage and we protect it with a password.   Sonatype Developer Onboarding product knows how to access this information and every time it needs to access secured storage to retrieve a password it will prompt you for your storage password.

This is the same sort of functionality that you can get through a product like PasswordWallet or 1Password.    Sonatype has taken the idea of a password wallet and integrated it into the development environment.

Sonatype OSS Repository

hosting open source project repositories

If you are running a large instance of Nexus to manage internal development, or if you are responsible for an open source project’s installation of Nexus, you can use the approach outlined in this post.

Each Project Controls a groupId

On the Sonatype OSS Nexus instance, every project has a unique maven groupId.  This groupId determines the repository path of all its artifacts.    For example, all projects released by the ehcache project have a groupId of net.sf.ehcache.  This means that all of the artifacts from the ehcache project are stored under the repository path /net/sf/ehceche.  Take Plexus as another example, its maven groupId is org.codehaus.plexus, and its artifacts go under the repository path /org/codehaus/plexus.   When projects have unique groupIds, there is no repository path collision.   This means that we can consolidate hundreds of repositories into a single shared repository.

With a single, shared repository we need to find a way to provide isolation between each project (each groupId). Ehcache administrators should only be able to deploy artifacts to /net/sf/ehcache, and Plexus developers should only be able to deploy artifacts to /org/codehaus/plexus. Administrators should be given CRUD privileges on a project’s repository path, other users are only allowed to read this path.

Implement Partitioning with Nexus Security

To implement this partitioning, I use Nexus Repository Targets, Privileges, and a custom Nexus Role.  Let’s take a look at the Ehcache groupId and walk through the process of creating a repository target for the ehcache repository path, a set of privileges that grant CRUD access to this area of the repository, and an Ehcache Admin role.

Step 1: Create a Repository Target

To create a repository target, click on Repository Targets under the Administration section of the Nexus application menu.   Once you click on Repository Targets, click on the Add button above the list of repository targets.   The groupId of Ehcache is net.sf.ehcache.  In the following figure I’m creating a repository target with pattern of ./net/sf/ehcache/., this pattern matches all artifacts under path /net/sf/ehcache/. What if one project requires more than one groupId? Like Ehcache, it also needs net.sf.jsr107cache. It’s ok, I can add more than one pattern for a repository target, in this case I also add “./net/sf/jsr107cache/.“.   The name field in this figure is free form, it doesn’t have to be a package name or a groupId, but I set the name to the main groupId to make it easier to find this target in the future.

Step 2: Create Privileges

Once the repository target is created, we can create privileges based on it (along with a repository or group). Open the Privileges panel by clicking on Privileges under in the Security section of the Nexus application menu.  Once you see the Privileges panel, click the Add button, select Repository Target Privilege. You need to fill in the name, description, repository, and repository target of the privilege. Here I choose repository target net.sf.ehcache and I target the repository Release (Repo).  You can also choose “All Repositories” if you want to define a global privilege.  I usually set the name and description to match the name of the groupId and which repositories the target applies to.

Click Save, then your CRUD privileges are created on repository Release, based on repository target net.sf.ehcache.   You should see the following four privileges after clicking on Save.

Step 3: Create a Nexus Role (or a Nexus User)

The next step is to create a user for the open source project that needs to have these privileges.  In the case of ehcache, we define an ehcache administrative user, and we assign the privileges we just created to this new user.

Alternatively, we could create a new Nexus Role which contains these four new privileges and we could assign this role to users that need to have access to the ehcache repository target.  I can assign these privileges to the Ehcache Admin role (it’s easy to create a role in Nexus, just click on Roles under Security in the Nexus menu).  Users who are assigned this new role role can then deploy artifacts to /net/sf/ehcache/ or /net/sf/jsr107cache/ of repository Release, but they can not deploy artifacts to other paths of this repository.

Conclusion

Following this pattern, we can create privileges for Plexus artifacts, for Sonatype artifacts,and for any other project as long as we know the groupId.  I’ve found that this approach preferable to the approach that used hundreds of independent Maven repositories.   We now have a single repository with multiple, project-specific privilege sets, and the process of adding new roles and privileges is very straightforward.   We’ve made it past 100, now we’re ready for 1000.