EclipseMagazine Interview with Jason van Zyl on the Maven Ecosystem


March 29, 2010 By Jason van Zyl

Recently, I was asked to do an interview for EclipseMagazine about the future of Maven and release of Maven 3.0.  JAXenter published part one and two of the interview over two weeks.  Below is the full interview, which covers everything from changes Maven 2 users can expect when migrating to Maven 3, Nexus repository manager, the Maven Shell, Polyglot Maven, and more.

__With the switch from Maven 1.x to 2, developers had to manage some fundamental changes. What challenges can users expect when migrating from Maven 2 to Maven 3?__

We are planning that, in most cases, Maven 3.0 will be a drop-in replacement for Maven 2.x. We have gone to great lengths to ensure backward compatibility while reimplementing a good portion of Maven’s internals. From the command-line perspective we are trying to be fully compatible. Maven 3.0 will not allow duplicate dependency or plugin declarations, so those problems would need to be fixed, but aside from that no changes to your POMs will be required. In all other regards we have created backward compatibility layers to protect users from the many internal API changes that we have made. I really hope that the Maven community can move forward to Maven 3.0 without grief, and use the new features as it is convenient for them.

__Is there anything users should keep in mind when creating a new project, to be prepared for Maven 3?__

It honestly shouldn’t be any different from Maven 2.0. That’s the intended goal. So much has changed under the covers that we didn’t want to change the POM format. The primary goal is a path forward for all Maven users, efficient embedding, increased performance, synchronizing the Maven 3.0 code base with m2eclipse, and adding extension points for tools like Tycho, Polyglot Maven, and the Maven Shell.

__Will Maven 3 support the integration of dynamic languages (e.g Groovy)?__

It would be more accurate to say that Maven 3.x can support dynamic languages. This will not be directly available in the Maven 3.0 release, but support for dynamic languages is available now in Sonatype’s Polyglot Maven project which you can find at http://polyglot.sonatype.org. You also have a question below specifically about Polyglot Maven below so I’ll expand on Polyglot Maven there.

__Will there still be a settings.xml or can we expect another method of configuration in Maven 3?__

In Maven 3.0 we are again not attempting to change the models that users are accustom to. Internally in Maven 3.1 we will expose a security manager and use the settings.xml implementation as the default implementation, but Sonatype is planning on creating an implementation that interacts with Nexus, our repository manager. In dealing with many large organizations we have found that integrating with in-house security systems and custom implementations is necessary so we will be providing an extension point for this with the security manager.

In Maven 3.1 we will also introduce POM mixins to help make configuration more maintainable and portable. One of the frequent complaints with Maven 2.0 is that sharing configuration can only be done via inheritance. This is obviously problematic as not all projects within organizations, and certainly not all projects in OSS, share lineage. POM mixins are a form of POM composition where a parameterized fragment of a POM can be injected into the current POM with a simple reference.

For me, the first mixin that I wanted to create was a mixin to encapsulate a standard release capability for Maven projects. Apache projects that use Maven inherit the Apache POM, which contains configuration for our release process.  This incorporates license attribution, PGP signing, staging and promotion with Nexus, an Apache compliant source archive containing the original project structure, and the production of the standard source and javadoc JARs. I wanted to create a very easy way for any project to incorporate this capability in their project. The release mixin consists of plugin declarations, their configuration which can be parameterized, and a specification of what POM elements are required in the target POM for the mixin to operate correctly. The POM mixins can then be deployed to a remote Maven repository to be shared via a standard Maven coordinate. In the target POM you specify the coordinate of the POM mixin along with any parameters required by the POM mixing, and Maven will retrieve the POM mixin and inject it into the target POM.

What I expect to see are projects in the community creating mixins for capabilities that require the interaction of many plugins along with the accompanying configuration. One of the biggest problems in Maven, given it’s distributed nature and there being so many plugins, is finding the right plugins that work together and figuring out how to configure a set of plugins to arrive at the desired result. The POM mixins will remove much of this pain.

__Can you tell us a bit about Nexus?__

I’m going to let Brian Fox, Nexus lead and Maven PMC Chair, answer this one.

Nexus is a repository manager built by the Maven experts at Sonatype. Repository managers serve a few of purposes in an organization: they act as highly configurable caching proxies for remote repositories, and they provide a centralized location to deploy, store, index and share all your internal artifacts. We use the analogy that a repository manager are to binaries what a source control management systems are to sources. In both cases you can use a flat file system but quickly learn that this isn’t scalable.

Nexus is unique among repository managers for several reasons. First, it is designed to be light on resources and heavy on performance. Second, the data is stored in the native repository layout on disk, which ensures that traditional backup and synchronization tools can be used. You don’t need to be a DBA to get access to your artifacts. Third, Nexus supports many repository formats and can convert on the fly between them: Maven 1, Maven 2, OSGI Bundle Repository (OBR), P2, RubyGems. The community is even working on RPM support.

Nexus is the preferred choice for the largest repositories in the community such as Apache, Codehaus, Jboss, Atlassian, Glassfish and many others. It is available in two flavors: an open source version http://nexus.sonatype.org and a professional version that extends the open source version http://www.sonatype.com/products/nexus with commercial add ons, to further manage the binary lifecycle in enterprise organizations.

__In the Maven Repository, there are all kinds of plug-ins, ranging from early Alpha-Versions to stable plug-ins. Do you have any plans for bringing order to this “plug-in chaos”? One could imagine a similar process to Eclipse, with Incubators and a Simultaneous Release Train providing reliable, high quality plug-ins…..__

This is something that Sonatype plans to tackle in the near future where we provide a higher quality set of plugins. Maven, by it’s distributed nature, makes it pretty hard for us to guarantee the quality of Maven plug-ins in every Maven repository around the world and what ultimately lands in Maven Central. But what we can do is “battle test” a smaller set of plug-ins and make sure they have integration tests, perform well, and integrate properly with m2eclipse. Once Maven 3.0, m2eclipse 1.0, and Tycho 1.0 are released we plan to start this project.

__Sonatype have just released version 0.10.0 of their m2eclipse Maven Plug-in for Eclipse. What new features does 0.10.0 have to offer? And will version 1.0 bring major changes?__

m2eclipse 0.10.0 is almost a complete rewrite from 0.9.8 so there are many changes and improvements. The most important aspect is that we have the m2eclipse code base synchronized with the Maven 3.x code base. When we need fixes in m2eclipse that are the result of problems in Maven, it’s only a matter of hours before we can roll a fix through the system and produce a new m2eclipse build. There have been several hundred fixes across Maven and m2eclipse in the last year. We’ve probably put in close to eight thousand man hours across Maven and m2eclipse as we approach the Maven 3.0 and m2eclipse 1.0 releases.

The performance will be the first thing users notice. Even with a standard Maven project general operation is faster, but if you leverage the new incremental Maven Plugin API and the customizable lifecycle mapping you truly have incremental behavior with Maven inside of Eclipse.

The new incremental Maven Plugin APIs allow Eclipse to communicate very precise information back to Maven. For example, if you change a single resource that needs to be filtered, Eclipse will no longer trigger Maven’s entire lifecycle, but rather communicate the information about the single file that has changed and execute Maven’s resource plugin to operate on the single file that changed. The customizable lifecycle mapping allows m2eclipse to eliminate most of Maven’s execution from within Eclipse. You can selectively choose what plugins you want to execute as part of the build within Eclipse. The end result is that if you tune your project, you can get extremely fast results. We want to make this tuning easier as we move toward the 1.0, but it’s still relatively easy to do this today.

We also have improved integration with Nexus so you can search the content of your repositories from within Eclipse, and we have simplified the configuration of repository indexes by automatically configuring them based on repositories you have configured in your settings.xml file.

__There’s already an official Eclipse project for integrating Maven: IAM (formerly ‘Q4E’). What’s the difference between m2eclipse and IAM? And is a merger of the two projects conceivable?__

IAM is not the official project for integrating Maven. Both m2eclipse and IAM are in incubation, so neither has official status at Eclipse. It’s unlikely there will be a merger. The IAM code base is a year behind m2eclipse at this point, and there are no active Maven 3.x committers on the IAM team. Sonatype has full-time developers and QA staff on Maven 3.x and m2eclipse. We’ve done a lot of work over the last year to make sure Maven and Eclipse work well together. m2eclipse is the de facto standard. You can see this in the community as questions on the user mailing lists are always about m2eclipse, and vendor support is already in place for m2eclipse – Genuitec, JBoss, and SpringSource all integrate m2eclipse, not IAM. Eclipse may ultimately let both projects graduate from incubation, but only one project will be official and shipped with distributions, and it very likely that will be m2eclipse. It simply boils down to manpower. We always hope for community support, but at any given time Sonatype has up to eight full-time people working on m2eclipse so users can be assured bugs will be fixed and new features will be developed.

__Can you tell us a bit about the P2 project at Eclipse and how Sonatype has become involved with the P2 project.__

The P2 project at Eclipse is the basis of the update manager in Eclipse, but it is also a general purpose provisioning framework. Sonatype is interested in provisioning user desktops, provisioning development infrastructures, and provisioning production run-times. P2 has become a critical technology for Sonatype, so we recently hired Pascal Rapicault who is the lead of the P2 project at Eclipse. Pascal will remain heavily involved in the community, in much the same way that Sonatype is heavily involved in the Maven community. We are also planning to invest in some research and collaborate with Daniel Le Berre through the National Centre for Scientific Research in France. Daniel is the creator of SAT4J, which is the core solver used in P2 for determining the selection of dependencies. I am interested in trying to merge the artifact resolution strategies of Maven and P2.

__Maven Tycho is your new OSGi-Toolchain designed to be a drop-in replacement for the Eclipse PDE Build. What are the goals of the Tycho project? And what features are already available?__

The focus of Tycho is to create an effective hybrid of Maven and Eclipse-based OSGi technologies for building, publishing, and distributing Eclipse plugins, update sites, RCP applications and OSGi bundles. Tycho is a manifest-first approach to building and leverages Eclipse’s Equinox, JDT and P2 projects. The alternate approach is a POM-first approach which is the strategy employed by the Maven Bundle Plugin which uses Peter Krien’s BND tool to generate a manifest from the classes post build.

Both of these approaches will be with us for quite a while, so another goal of Tycho is to allow these different approaches to interoperate. Both approaches produce bundles, but in order to use Tycho the bundles must be available in a P2 repository and have the accompanying P2 metadata. So with a POM-first approach we dynamically generate the required P2 metadata for bundles that do not have it. This effectively allows us to take all bundles available in Maven Central and allow them to be used within Tycho. I believe that an unmanageable rift will form if tools are not capable of interoperability between these two common forms of development. Many people use Maven only to produce bundles, and many people use standard Eclipse-based technologies to produce bundles, and if these worlds are not united there will be a massive duplication of effort. Tycho is a bridge between these two worlds.

We have also integrated Tycho with m2eclipse so that you can import a Tycho project into Eclipse as a PDE project. We have also added support to allow the resolution of POM-first bundles in the Eclipse workspace. Currently there are two major projects at Eclipse that are using Tycho to build: the Tigerstripe project driven by Cisco, and the EGit project. The EGit project is interesting because it uses the hybrid approach of using a POM-first build for JGit, we then generate a P2 repository from the Tycho build which is then consumed by the EGit build which uses a manifest-first Tycho build.

__Can you tell us a bit about Polyglot Maven?__

Polyglot Maven is the project for experimenting with domain specific lanaguages (DSLs) and terse markup languages (TSLs) in conjuction with Maven 3.x’s core. What is currently available today can be found http://polyglot.sonatype.org. What we have are starting points of internal DSLs for Groovy, Ruby, Scala, and Clojure. There is also an external DSL that I have developed using XText, which is a fantastic tool at Eclipse.org for creating parsers and editors for external DSLs.  We are also providing support for YAML-based POMs using the excellent SnakeYAML library.

The DSLs are meant to be more then merely different representations of the POM. The DSLs will have full access to Maven’s internals, which have been redesigned for integration and extension. What this means in practical terms is that implementors of DSLs can leverage Maven’s infrastructure but create custom ways to interact with, or extend, Maven. A DSL could change Maven’s lifecycle, decorate lifecycle phases, create new configuration mechanisms, or allow a more procedural mechanism for their users. I honestly don’t know how useful these features are for general users, and more specifically enterprise users, but I only see benefits in allowing greater leverage of Maven’s infrastructure.

What I really care about and what really concerns me is interoperability at the repository level and tooling. This is why the code lives in the Polyglot Maven project. I don’t want to misrepresent to users that this is ready for prime time. It fully works, but there needs to be more thought about how projects being constructed with possibly many DSLs can still play nicely together. What will the resultant metadata look like that is exported to Maven repositories? Currently we feel that this is likely to be a Maven 2.x POM created during the deployment process. What will the tooling look like for Maven DSLs? We are hoping that the DSLs can preserve many of Maven’s internal models so that they can leverage m2eclipse. To simply provide the infrastructure for these DSLs is not enough; repository interoperability and tooling is important.

In the last 18 months I have surveyed nearly 2000 people from the various talks and presentation I’ve given, and approximately 5% of the people surveyed are interested in DSLs for Maven or different representations of the POM. But these 5% are very important. These are the very creative hackers that want to do things in a project from one end to the other in a fully programmatic way. They want to code everything including the build. This is why I created Polyglot Maven because I believe this 5% can potentially contribute as much to Maven as the remaining 95% of the Maven user base. The folks who want the polyglot support are highly effective people. In the end it’s all good for Maven.

__Can you tell us a bit about the Maven Shell?__

The Maven Shell is Maven embedded in a long-lived shell process. The shell avoids the start-up costs of invoking Maven repeatedly, caches the parsed POMs, provides Maven Archetype integration, provides Nexus integration, has a built in help system, and on the OS X provides Growl support. In typical cases you’ll see your build times cut in half. We have many features planned for the 1.0, such as the integration of the make-like reactor mode so that within the shell only the modules that have changed will be built, project workflow support to help guide users do the right thing, Hudson support for creating jobs and querying statuses, and ultimately the integration of Tycho and Polyglot Maven features.

__There are many different build systems to choose from. In your opinion, which is Maven’s biggest competitor? Maybe Gradle…..?__

Maven’s biggest competitor in the enterprise are homegrown Maven-like solutions created with Ant. These are the systems that Maven typically replaces in enterprise settings. We honestly don’t see anyone trying things like Gradle or Buildr for large enterprise projects. I think there is a lot of hype around some of these technologies. We understand that people are constantly looking for new solutions, but we’re also convinced that the enterprise build space is full of complicating factors that don’t lend themselves to over-simplification.

__In three sentences: Why should we switch to Maven 3?__

Maven 3.0 will be the best version of Maven ever made. We are doing everything in our power to make it backward compatible and we have put in thousands of man hours in testing and performance tuning. Maven 3.0 is the future of Maven and the path forward for all users who wish to use new technologies like m2eclipse, Tycho, the Maven Shell, Polyglot Maven, and Nexus.

__And is there a release date?__

We are planning for the Maven 3.0-beta-1 around the beginning of April, and based on feedback and good weather permitting we will release Maven 3.0 at the beginning of May.