But what about the components that developers use to build applications? Many organizations that we talk to assemble their applications from open source components. They no longer write a lot of custom code, they stitch together components from various sources – in many cases 80-90% of modern applications are made up of components. This may seem surprising until you think of the various types of components that are used to develop  applications: utility classes, logging, caching, database access, testing frameworks, web frameworks, collection handling, etc. Why develop those feature from scratch when you can reuse components freely available on the Web?

So why compare Linux, Apache HTTP Server, and MySQL with open source components like junit, commons-collections, log4j? I think it helps illustrate the need for a dramatically different management approach.

When it comes to major decisions like operating systems, web/application servers & databases, many organizations…

• Architecture Review -  conduct a comprehensive technology selection process driven by the architecture team… .vs. OSS components that are often selected by individual developers.
• Vendor Selection - go through a deliberate vendor selection process, including RFI/RFP, POCs, etc… vs. OSS components where the project team is not vetted.
• License Indemnification – protected from potential license issues via vendor indemnification… vs. OSS components with transitive dependencies on components with problematic licenses.
• Contractual Procurement - officially contract and procure software through purchasing departments… vs. OSS components that are “free”.
• Production Monitoring - monitor as part of an overall enterprise level BAM strategy… vs. OSS components that are often hidden in plain site (organizations don’t even know what they have).
• Financial Budget - built into the regular IT budgeting cycle… vs. OSS components – again, aren’t they “free”.
• Updates/Patches - update periodically via a pre-planned patch / update process… vs. OSS components where regular updates are not even considered.

Although organizations probably don’t think risk management per se when making major open source infrastructure decisions, that really drives their decision process – minimize risk by selecting infrastructure software that is reliable, easily maintained and cost effective.

Shouldn’t you be doing the same at the application level? With components making up the bulk of your applications, it makes sense to manage the components in a systematic fashion. But you can’t use the same process for OSS components as you do for operating systems, databases, etc.

How to start? We call it Component Lifecycle Management. Stay tuned as we introduce this concept over the coming weeks and months.

We are celebrating this anniversary with a dual release of Flexmojos 3.6 and Flexmojos 4.0-pre-alpha-1.

Flexmojos 3.6 Release

Flexmojos 3.6 most relevant changes:

• Add a new source-view goal which goal produces a syntax highlighted version of the as, mxml and html if they are bundled in a SWF (thanks to Julien Nicoulaud for this contribution)
• Some fixes to Flexbuilder metadata generation
• Add support to Flex SDK 4.0.0.13555 and newer
• Add support for building applications using Flashplayer 10.1 and Air 2.0
• New configuration includeAsClasses for SWC compilation. Wildcard support for includeClasses
• Generator mojo now supports package translation between Java and generated AS3
• Support for granite generator 2.1 (thanks to Kyle Lebel)

For a full list of changes on 3.6, visit this page: https://issues.sonatype.org/browse/FLEXMOJOS/fixforversion/10629

Flexmojos 4.0 Pre Alpha 1

First, Flexmojos 4.0-pre-alpha-1 disclaimer: “IT IS A PRE-ALPHA, use at your own risk.”   Many features available in 3.x are simply not implemented yet, but it also supports many compiler options currently not available in Flexmojos 3.x.

The biggest change in Flexmojos 4.0 vs. Flexmojos 3.x is that we’re starting to use the compiler OEM API and not the OEM wrapper.  Why are we making this change?   The OEM wrapper was seriously out of date, and it was almost impossible to keep up with the latest changes and available options.   By coding directly to the OEM API, we will be able to ship with the latest stable FDK available at release date.   Flexmojos is the first version of Flexmojox that is designed for Maven 3.

Again, Flexmojos 4.0-pre-alpha-1, as the name suggests, IS A PRE-ALPHA. Not destined for production. If you decide to use it – keep in mind it may need patches, bug fixes, breaks some work on backward compatibility.

In summary, it has been a great two years of working on Flexmojos.  We’ve made a difference for developers with Flexmojos, and I hope the next two years are as fruitful as the last two years.

Why we chose the GPL license for Nexus

Normally we would have used a BSD/MIT/Apache license for software that we develop.   This is what we’re used to, with most of our developers having been active in the Apache community for years, we’re all very familiar with the philosophy behind this license.   When we announced that Nexus was going to be released under a GPL license, some of our colleagues wanted to know how a group of Apache participants decided to use the GNU Public License? When we started Nexus, the plan was to create a commercial product.  We hadn’t thought about creating Nexus as a commercial product built atop an open core.   Once we got into the effort, we soon realized that creating a commercial-only product with a team full of open source developers wouldn’t be very fun.   When you develop a commercial product, you limit yourself to a small group of developers working in an isolated environment.   It didn’t take us long to remember that close-source development isn’t as productive (or interesting) as developing a product out in the open, and as we like working with other developers, we very quickly decided to take the hybrid, open-core approach to Nexus.

Once this decision was made, we had to select the appropriate license for Nexus.   As a company that is dedicated to the development of a sustainable product, we decided to use the GPL.   This decision wasn’t made in a vacuum, we can see other companies like Redhat having succeeded with the GPL license, and, quite frankly, we wanted to protect our investment in Nexus.   As a company, we feel that the GPL provides us adequate protection and still allows people to view, hack, and contribute back if they want to.

We stated clearly from the introduction of Nexus that we intended to create a commercial version of Nexus.  GPL was the most effective way to protect our investment. I don’t believe this has adversely affected users and it doesn’t appear to have hampered the adoption of Nexus at all.

While the core of Nexus is covered under the GPL license, many of the libraries that we develop as a part of the Nexus effort are covered under the Apache Software License.   Spice, a collection of utilities and libraries that do much of the heavy-lifting in Nexus is covered under the ASL because we’re sold on the value of releasing widely-used components under the least restrictive license possible.

Our License Selection Guidelines

I reserve the right to add to the following list of general licensing guidelines as we continue to learn from experience, but here are three things we think about when selecting licenses for our project and products.

Acknowledge Reasonable Limits to Copyleft

One recent trend for companies in our situation is to use a license like the Affero GPL.   The Affero GPL was designed as the ASP-killer, unlike the GPL which requires you to “give back” any additions or changes you have made only if you distribute your software to others, the Affero GPL compels you to give back any modifications to a piece of software if you use it to deliver a service.  We specifically did not choose the Affero GPL because we wanted to make sure that our users could extend Nexus without having to give back the changes and extensions they needed to make for internal use.   If we had selected Affero, I can guarantee that we would have seen lower adoption of the tool and less community participation.    There are limits to viral, copyleft licenses, and overstepping these boundaries has adverse effects on community involvement.

We have been fortunate that many plugins that have been developed by user organizations have been donated back to Nexus. We feel that using the GPL is appropriate for Nexus because it is not restricting users from extending Nexus to suit their needs and the Sonatype staff generally do the hard lifting to implement features requested by users and provide fixes as necessary.

Always Move Toward Less Restriction

One thing to be worried in any OSS product are licenses that move from less restrictive licenses to more restrictive licenses. Sonatype chose the GPL at first because we felt that was the way to honestly express our intent with the project to become a product. If we were ever to change our license it would be to a less restrictive license. I believe it is an unacceptable practice to start a project with a less restrictive license, like the ASL, and then switch to a more restrictive license. I call this the Open Source bait and switch.

Many organizations find less restrictive licenses more acceptable and we understand these motivations (again, we’re very active in the Apache community).  Less restrictive licenses provide more freedoms to users and organizations. For a project to remove those freedoms after having released code under a less restrictive license is a violation of the OSS social contract. You can’t start a project using a less restrictive license to grow a community – because this tends to be easier with less restrictive licenses – and then switch.  Such a practice is dishonest and you will never see Sonatype make a similar move.

Constantly Evaluate Restrictive License Decisions

From time to time, you are going to see us moving code from more restrictive to less restrictive licenses.    As our offerings expand, and as portions of the code we develop become critical to the community that supports it, it only makes sense for certain components to become less restricted.    Less restrictive licenses enable wider integration and make it easier to drum up community participation.   Take the Nexus Indexer as just one example.   The Nexus Indexer is released under the Eclipse Public License, a less restrictive license that is comparable to a BSD-style license.   Because this component is released under the EPL, it can be integrated into all of the major repository managers, and it can be reused by anyone interested in creating a repository index.   It should also be noted that we’re in the process of donating the Nexus Indexer to the Apache Software Foundation to move it to an even less restrictive license: the Apache Software License.

Over time, you’ll see some of our components make this transition.    We understand the differences between the ASL and the GPL.   We’re committed to both when they are appropriate, and we’re not dogmatic about either.   When it makes sense for a component to be released under the EPL or the ASL, there will be no hesitation.   If we need to rethink a restrictive license for a component and release the component under a less restrictive license, we’ll do that.

What about documentation licenses?

Our documentation is a whole different situation.  First, documentation isn’t software.  The distributable product isn’t “code” in the same way that Nexus or Maven is, and, because of this, we didn’t want to just use the standard Apache Software License or GNU Public License because both of these licenses have very specific clauses that relate to “software”.     We wanted to find a license that allowed our readers to distribute our documentation, but we also wanted to find a license that offered us some protection.   Even though the books are “free”, they were the product of a substantial investment of our time and money.  Because Sonatype has an active training product, we didn’t want to turn around and see competing companies using our books as a foundation for competing products.

Given these constraints, we elected to release our books under a Creative Commons license with three clauses: No Derivatives, Non-commercial, and Attribution.   You can download and redistribute our books as much as you want, but you cannot modify or remix the work and sell the end-product for profit.   You can print our books and distribute them at cost, but you can’t base a training product on the material contained in the books.   The CC 3.0 BY-ND-NC has been good for us, we’ve attracted external contributors to our books over the past few years, and each one of our books has at least two external developers with direct commit access.

So what?

This topic might be a little wonky for most.   Most end-users are familiar with major licenses like the GPL and the Apache license, and they come to a basic understanding of the differences.    The users the care the most about this issue are the users who, themselves, participate in the open source development community.    At Sonatype, we pay attention to the nuances of these licenses, and we’re committed to provide details behind the decisions that go into our products.   I hope that this post has helped clarify these licenses for people who might have wondered why and how we chose them.      If you have any opinions about these decisions, please feel free to let us know.

Sonatype is focused on improving the foundational infrastructure which will allow us to improve the quality of artifacts and their accompanying metadata in Maven Central and Maven repositories around the world.  A lot of this is not especially glamorous work and though many people complain about the state of some of the Maven repositories, very few take action.    Here are some of the things Sonatype is doing with Nexus to improve the state of the Maven ecosystem and expand its scope.

Improving the Quality of Public Maven Repositories

Any effort to improve the quality of the Central Maven repository needs to begin with the major feeder repositories.  For years, we have been giving rsync access to many of the organizations with large feeder repositories like Apache and Codehaus.   When we started this effort, we were optimistic that these organizations would take care of their Maven repositories.   We thought that repository maintainers and projects would make sure that all artifacts were signed and that all POMs contained a bare minimum of useful elements such as “license” and “description”.    With hundreds of projects pushing artifacts into their respective repositories on a daily basis, it has become obvious that without some mechanism to guarantee quality, without a well-defined process, the Central Maven repository will contain artifacts and metadata of questionable status.  While the vast majority of artifacts have appropriate PGP signatures and metadata, the fact that a minority of (often very popular) artifacts lack proper dependency definitions or license elements means that we see a steady stream of complaints about the quality of the repository from our users.

These problems can be anything from one project trying to publish another project’s artifacts because they weren’t in Maven Central, incorrect, bad, or missing metadata, missing javadocs or source JARs, to invalid transitive closures.  Whatever the problems are, the overall issue stems from the lack of process surrounding how artifacts are published to our feeder repositories, and how these artifacts are then certified to be published to the Central Maven repository.

Sonatype’s answer to this problem has been to provide tools that can automate the process of repository maintenance for Central’s largest customers – the main feeder repositories.   We’ve provided a solution to the largest of the feeder repositories which allows them to use the Nexus Staging Suite to validate that all release artifacts contain the bare minimum of metadata, javadoc, and a valid PGP signature.    As Ate Douma wrote in Monday’s post about using the Nexus Staging suite to support the Apache Portals project, we’ve supplied a solution that reduces the amount of error-prone, manual work associated with software releases, and we’re going to continue to find ways to address the needs of large, open-source enterprises with Nexus by providing Open Source projects and organizations with a free license and free support.   Here is a list of some of the organizations with large feeder repositories that we are supporting directly:

• The Apache Software Foundation
• Codehaus
• Alfresco
• ExoPlatform
• Glassfish
• Open QA
• Scala-Tools

With all of these organizations and projects using the Nexus Staging Suite, we are confident that the quality and reliability of artifacts and metadata will increase over time.  The PGP signatures provide the security assurances organizations need, and the sources, javadocs and correct POM information like SCM information make for a better developer experience in the IDE.  In M2Eclipse, for example, javadocs and sources can be dynamically retrieved as required and binary dependencies can be materialized to source from from SCM coordinates. This makes contributing patches to an dependent project an order of magnitude easier.

We are also fortunate to be working closely with Atlassian. We have Atlassian Crowd support in Nexus, so Nexus is an ideal fit for Atlassian and for organizations that make use of Atlassian’s compelling products. You can find Atlassian’s Nexus instance here. It’s just another validation point for us that Atlassian sees fit to use our products as part of their daily development. We have a lot of respect for the Atlassian folks, and we’ve standardized our entire development environment on tools like JIRA , Greenhopper, and Confluence.

Decreasing the Time to Reach Maven Central

Many users and projects have complained that it can take a while to get their artifacts in Maven Central so we’re starting to focus on reducing the time to reach Central. The biggest obstacle to automating the process of publishing artifacts to Central is one of quality.   Artifacts would reach central faster if there was a better way to enforce a minimal set of quality standards.  The legacy process involved projects uploading “bundles” to a JIRA project and repository maintainers manually inspecting the bundles to see if they complied with a set of standards.

In the previous section, I described how Nexus is already powering the major feeder repositories for the Central Maven repository.  What we offer projects is described here. Basically we will help you cleanup, and migrate your project’s Maven repository to our hosted infrastructure and help you setup your project POMs to use Nexus’ Staging. Projects are setup with the default staging rules which ensure the presence of PGP signatures, javadocs, sources, and a POM which contains decent information. We provide all the instructions for the setup on the Maven side so there is little work you have to do in order to take advantage of this service.  We are also providing a mechanism by which the standard bundle uploads, typically done via JIRA, can be handled by Nexus. Internally within Nexus the upload bundle is exploded and placed in a staging repository, as would happen if you performed the release against Nexus from your Maven build.  We then apply the same staging rules to ensure quality.

We have already started rewarding projects with good Maven releases by automating their synchronization with Maven Central.   Over time we are going to start enforcing standards for security and quality of metadata.    If you have proper project metadata, PGP signatures, javadocs and sources your artifacts will fly into Central as quickly as possible.  Projects that submit poor Maven releases are going to have a more difficult time getting artifacts into Maven Central.   We’ll start to enforce these standards gradually and we’ll give the community time to adapt.  Any tool that creates bundles for deployment to Central is capable of producing these artifacts. The staging rules don’t care how the releases were constructed. So use whatever tools you like, you’ll just have to pass a minimal set of requirements to make it into the Central repository. Over time this should greatly improve the quality of Maven Central.

Providing Metadata about Repositories

For a long time we have been providing a way, through Nexus, and the stand-alone Nexus Indexer, to produce the Maven repository index.  The repository index contains information about all the artifacts in the given repository including class file information and project identifiers.  It is primarily used as part of IDE integration to help developers find dependencies based on artifact coordinates or class references, like import statements.   Using m2eclipse, all you need to do to add a new dependency is type the name of a class into your code and search the index for matching dependencies.    You can also search the repository index for all of the available Maven archetypes when you are creating a new Eclipse project.    Both of these IDE use cases are possible because of the standard repository index format that was defined as part of the Nexus project.

The Nexus index format is also storing information about the presence of PGP signatures, javadocs, sources, and checksums.   Using Nexus and other tools that can read the Nexus index format, you can aggregate multiple repository indexes together and perform quick searches across multiple public repositories (as well as your own hosted repositories). The Nexus indices from the OSS repositories around the world are proving to be a critical resource, we can tell because it is the most requested item from Maven Central.  Index downloads amounted to 28TB of transfer last month.

Polyglot Component Repositories

The explosion of language choices has transformed the way most developers approach problem solving.   Just four years ago it would have been normal to walk into a corporation and see Java at all levels of the stack.  In 2010, most businesses have started to incorporate multiple languages and hybrid architectures into enterprise systems.   A system designed today might use Ruby on Rails (running under JRuby) to power a web site which interacts with a set of services coded in Java.  Services like Twitter rely on a foundation of Scala code executing on the JVM while other portions of the Twitter architecture use different languages and different technologies where they are most appropriate.

We are a polyglot industry, and while our software enterprises are run on a mixture of different languages, our development tools and build technologies are often locked into a single language or a single technology.   A Ruby application is built using Rake, a Scala application is built using Sake or the Maven Scala plugin, and Java or OSGi applications are built using Maven.   A Ruby library might generate and consume gems while a Java application might generate and consume JAR files.   There is currently no single, consolidated “Tower of Babel” to help developers translate between different types of software artifacts.   We’ve put a lot of effort into making the foundation of Nexus as agnostic as possible about what it is storing, and, because of this, we’re moving to add support for even more artifact types.

Nexus currently supports the two OSGi formats of P2 and OBR and we are just finishing our first draft of RubyGems support.   Polyglot Maven will drive us toward a Polyglot Nexus. As part of the work we’re doing on Polyglot Maven we may find that different scripting language implementations have slightly different requirements for dependency management or provisioning runtimes, and we’ll be ready for that.

Where do we go from here?

Next we’re thinking about ways to make statistics for a given project’s artifacts available to the project’s developers.  We have already implemented user signup in Nexus and we are currently working on project signup as well. What this means is that projects can register with a given groupId, or set of groupIds, and optionally be provisioned a repository which can be operated by a set of users. Once a project registers we will know what slice, or slices, of the statistics they need to see.   Our initial thought is that project statistics, number of downloads should only be made available to the public with the permission of each individual project. Brian and I along with Greg Luck and Dain Sundstrom have been working on a simple statistics mechanism that we hopefully can provide to projects early this year.

Apache Portals is a collaborative software development project dedicated to providing robust, full-featured, commercial-quality, and freely available Portal-related software on a wide variety of platforms and programming languages. This project is managed in cooperation with a number of people worldwide (both independent and company-affiliated experts), who use the Internet to communicate, plan, and develop Portal software and related documentation.

Hippo actively promotes the use of the Apache Portals Jetspeed-2 Enterprise Portal as a superior Enterprise level / OEM solution because of its performance and flexibility. At Hippo, we rely on Jetspeed-2.  Jetspeed-2 is an Open Portal Platform and Enterprise Information Portal which provides the necessary infrastructure for portlets.   Jetspeed-2 is a fully integrated part of the collaboration suite consisting or the Hippo CMS and Portal, designed for use in internal and external Enterprise Portals.

Before using Nexus, our releases were managed manually. The release manager had to painstakingly copy the project release artifacts to a predefined location and structure on one of the internal Apache servers. This solution was very time-consuming and error-prone, which caused us to look at Nexus Professional.

We at Apache Portals project were early adopters of the Nexus repository as provided to Apache by Sonatype. The Apache Software Foundation received a free Nexus Professional subscription from Sonatype using their Open Source License Request process early this year. Any currently active open source project which is publicly accessible over the internet can apply and be considered for a free subscription.

At the same time as we migrated our release process to leverage Nexus, we also made the switch to use the new Nexus repository as hosted on http://repository.apache.org. To make it all work together, we had to make a few structural changes in our (master) Maven pom.xml, but (all in all) it was rather easy to do. We were delighted to see that our project configurations were improved and cleaned up as a result of the migration.

Once everything was set up, we configured the Nexus Professional Staging Suite to “stage” our releases, something which was practically impossible with our old manual release process. With the Staging Suite, we have created a staging repository, allowing us to manage the promotion of artifacts from a staging repository to a release repository.

The Nexus Staging Suite has provided us with a workflow to control our releases. Now everyone can do a fully-automated and properly validated release with just a few Maven commands, available directly from within our build environment.

Thinking back to all the tricky and cumbersome manual steps we had to go through before the migration, and comparing to how easy it has become now, switching to Nexus was a good decision, which definitely improved our project management in a major way.

Note: Apache and Apache Portals are trademarks of Apache Software Foundation.