You watch this embedded video, or watch the presentation over on the Parleys.com site.

Note: To switch between the slides and the video of me talking, click on the smaller video in the upper right-hand of this video embed.

Maven Stack Infrastructure

I’m going to talk about some of the under pinnings of the technologies we’re using as part of our Maven work. Why we selected the technologes and some of the current work that’s happening.

• Guice
• From Plexus to Guice (#1): Why Guice?
• From Plexus to Guice (#2): The Guice/Plexus Bridge and Custom Bean Injection
• From Plexus to Guice (#3): Creating a Guice Bean Extension Layer
• Peaberry
• Apache Shiro
• Enunciate

Maven

There are several develops going on in the various Maven projects. Maven 3.x is on the way, but we have some very interesting work happing with OSGi within our Tycho project. The Flexmojos project and NAR (the C/++ framework for Maven) are also popular.

Tycho

• Tycho Site
• Tycho @ Github
• Tycho/PDE Integration
• P2 Site

Flexmojos

• Flexmojos Site

NAR

• NAR Site
• NAR @ Github

M2Eclipse

The primary IDE integration we work on at Sonatype. The most important thing to talk about in M2Eclipse is the configuration framework.

• M2Eclipse Site

Hudson

Most of the work we’ve been doing on Hudson is still not really good enough for public consumption but we’re testing the changes we’re making on the Sonatype grid.

Hudson Site

Nexus

• Nexus Site
• Nexus: Improving Maven Central and Supporting the Maven Ecosystem
• Selecting OSS Software: 10 Questions Answered for Sonatype Nexus
• OSS Repository Hosting
• OSGi Central
• Ruby Central (We’re working on making that publicly available

Proviso

Proviso is not publicly released yet, but Alin and I have been working together to get our provisioning framework ready for a public release.

Git

We are starting to use Git heavily and soon likely exclusively. We are helping out on the EGit, and JGit projects at Eclipse and we’re trying to put a little Git server together based on JGit, MINA, and Apache Shiro.

• JGit Site
• EGit Site
• Tycho building EGit/JGit
• SShJGit @ Github

Maven Extensions

I’ll chat about these in the talk.

• Polyglot Maven Site
• Polyglot Maven @ Github
• Maven Shell
• Maven Shell @ Github

I’ll likely keep adding to the entry leading up the talk, but I thought I would get the ball rolling!

Sonatype is focused on improving the foundational infrastructure which will allow us to improve the quality of artifacts and their accompanying metadata in Maven Central and Maven repositories around the world.  A lot of this is not especially glamorous work and though many people complain about the state of some of the Maven repositories, very few take action.    Here are some of the things Sonatype is doing with Nexus to improve the state of the Maven ecosystem and expand its scope.

Improving the Quality of Public Maven Repositories

Any effort to improve the quality of the Central Maven repository needs to begin with the major feeder repositories.  For years, we have been giving rsync access to many of the organizations with large feeder repositories like Apache and Codehaus.   When we started this effort, we were optimistic that these organizations would take care of their Maven repositories.   We thought that repository maintainers and projects would make sure that all artifacts were signed and that all POMs contained a bare minimum of useful elements such as “license” and “description”.    With hundreds of projects pushing artifacts into their respective repositories on a daily basis, it has become obvious that without some mechanism to guarantee quality, without a well-defined process, the Central Maven repository will contain artifacts and metadata of questionable status.  While the vast majority of artifacts have appropriate PGP signatures and metadata, the fact that a minority of (often very popular) artifacts lack proper dependency definitions or license elements means that we see a steady stream of complaints about the quality of the repository from our users.

These problems can be anything from one project trying to publish another project’s artifacts because they weren’t in Maven Central, incorrect, bad, or missing metadata, missing javadocs or source JARs, to invalid transitive closures.  Whatever the problems are, the overall issue stems from the lack of process surrounding how artifacts are published to our feeder repositories, and how these artifacts are then certified to be published to the Central Maven repository.

Sonatype’s answer to this problem has been to provide tools that can automate the process of repository maintenance for Central’s largest customers – the main feeder repositories.   We’ve provided a solution to the largest of the feeder repositories which allows them to use the Nexus Staging Suite to validate that all release artifacts contain the bare minimum of metadata, javadoc, and a valid PGP signature.    As Ate Douma wrote in Monday’s post about using the Nexus Staging suite to support the Apache Portals project, we’ve supplied a solution that reduces the amount of error-prone, manual work associated with software releases, and we’re going to continue to find ways to address the needs of large, open-source enterprises with Nexus by providing Open Source projects and organizations with a free license and free support.   Here is a list of some of the organizations with large feeder repositories that we are supporting directly:

• The Apache Software Foundation
• Codehaus
• Alfresco
• ExoPlatform
• Glassfish
• Open QA
• Scala-Tools

With all of these organizations and projects using the Nexus Staging Suite, we are confident that the quality and reliability of artifacts and metadata will increase over time.  The PGP signatures provide the security assurances organizations need, and the sources, javadocs and correct POM information like SCM information make for a better developer experience in the IDE.  In M2Eclipse, for example, javadocs and sources can be dynamically retrieved as required and binary dependencies can be materialized to source from from SCM coordinates. This makes contributing patches to an dependent project an order of magnitude easier.

We are also fortunate to be working closely with Atlassian. We have Atlassian Crowd support in Nexus, so Nexus is an ideal fit for Atlassian and for organizations that make use of Atlassian’s compelling products. You can find Atlassian’s Nexus instance here. It’s just another validation point for us that Atlassian sees fit to use our products as part of their daily development. We have a lot of respect for the Atlassian folks, and we’ve standardized our entire development environment on tools like JIRA , Greenhopper, and Confluence.

Decreasing the Time to Reach Maven Central

Many users and projects have complained that it can take a while to get their artifacts in Maven Central so we’re starting to focus on reducing the time to reach Central. The biggest obstacle to automating the process of publishing artifacts to Central is one of quality.   Artifacts would reach central faster if there was a better way to enforce a minimal set of quality standards.  The legacy process involved projects uploading “bundles” to a JIRA project and repository maintainers manually inspecting the bundles to see if they complied with a set of standards.

In the previous section, I described how Nexus is already powering the major feeder repositories for the Central Maven repository.  What we offer projects is described here. Basically we will help you cleanup, and migrate your project’s Maven repository to our hosted infrastructure and help you setup your project POMs to use Nexus’ Staging. Projects are setup with the default staging rules which ensure the presence of PGP signatures, javadocs, sources, and a POM which contains decent information. We provide all the instructions for the setup on the Maven side so there is little work you have to do in order to take advantage of this service.  We are also providing a mechanism by which the standard bundle uploads, typically done via JIRA, can be handled by Nexus. Internally within Nexus the upload bundle is exploded and placed in a staging repository, as would happen if you performed the release against Nexus from your Maven build.  We then apply the same staging rules to ensure quality.

We have already started rewarding projects with good Maven releases by automating their synchronization with Maven Central.   Over time we are going to start enforcing standards for security and quality of metadata.    If you have proper project metadata, PGP signatures, javadocs and sources your artifacts will fly into Central as quickly as possible.  Projects that submit poor Maven releases are going to have a more difficult time getting artifacts into Maven Central.   We’ll start to enforce these standards gradually and we’ll give the community time to adapt.  Any tool that creates bundles for deployment to Central is capable of producing these artifacts. The staging rules don’t care how the releases were constructed. So use whatever tools you like, you’ll just have to pass a minimal set of requirements to make it into the Central repository. Over time this should greatly improve the quality of Maven Central.

Providing Metadata about Repositories

For a long time we have been providing a way, through Nexus, and the stand-alone Nexus Indexer, to produce the Maven repository index.  The repository index contains information about all the artifacts in the given repository including class file information and project identifiers.  It is primarily used as part of IDE integration to help developers find dependencies based on artifact coordinates or class references, like import statements.   Using m2eclipse, all you need to do to add a new dependency is type the name of a class into your code and search the index for matching dependencies.    You can also search the repository index for all of the available Maven archetypes when you are creating a new Eclipse project.    Both of these IDE use cases are possible because of the standard repository index format that was defined as part of the Nexus project.

The Nexus index format is also storing information about the presence of PGP signatures, javadocs, sources, and checksums.   Using Nexus and other tools that can read the Nexus index format, you can aggregate multiple repository indexes together and perform quick searches across multiple public repositories (as well as your own hosted repositories). The Nexus indices from the OSS repositories around the world are proving to be a critical resource, we can tell because it is the most requested item from Maven Central.  Index downloads amounted to 28TB of transfer last month.

Polyglot Component Repositories

The explosion of language choices has transformed the way most developers approach problem solving.   Just four years ago it would have been normal to walk into a corporation and see Java at all levels of the stack.  In 2010, most businesses have started to incorporate multiple languages and hybrid architectures into enterprise systems.   A system designed today might use Ruby on Rails (running under JRuby) to power a web site which interacts with a set of services coded in Java.  Services like Twitter rely on a foundation of Scala code executing on the JVM while other portions of the Twitter architecture use different languages and different technologies where they are most appropriate.

We are a polyglot industry, and while our software enterprises are run on a mixture of different languages, our development tools and build technologies are often locked into a single language or a single technology.   A Ruby application is built using Rake, a Scala application is built using Sake or the Maven Scala plugin, and Java or OSGi applications are built using Maven.   A Ruby library might generate and consume gems while a Java application might generate and consume JAR files.   There is currently no single, consolidated “Tower of Babel” to help developers translate between different types of software artifacts.   We’ve put a lot of effort into making the foundation of Nexus as agnostic as possible about what it is storing, and, because of this, we’re moving to add support for even more artifact types.

Nexus currently supports the two OSGi formats of P2 and OBR and we are just finishing our first draft of RubyGems support.   Polyglot Maven will drive us toward a Polyglot Nexus. As part of the work we’re doing on Polyglot Maven we may find that different scripting language implementations have slightly different requirements for dependency management or provisioning runtimes, and we’ll be ready for that.

Where do we go from here?

Next we’re thinking about ways to make statistics for a given project’s artifacts available to the project’s developers.  We have already implemented user signup in Nexus and we are currently working on project signup as well. What this means is that projects can register with a given groupId, or set of groupIds, and optionally be provisioned a repository which can be operated by a set of users. Once a project registers we will know what slice, or slices, of the statistics they need to see.   Our initial thought is that project statistics, number of downloads should only be made available to the public with the permission of each individual project. Brian and I along with Greg Luck and Dain Sundstrom have been working on a simple statistics mechanism that we hopefully can provide to projects early this year.