• Modifier SNAPSHOT is a modifier for a “base” version. If you specify 1.0-SNAPSHOT, this translates to “give me the latest development version for the base 1.0″. The assumption behind the SNAPSHOT modifier is that developers are continuously pumping out version after version in order to get to 1.0. A development team is constantly and repeatedly deploying snapshot into repositories, and others can use them without requesting the exact version.
• Version name LATEST. This is used without a base version and simply means – find and give me the latest possible version of an artifact be it a SNAPSHOT or a RELEASE.
• Version name RELEASE . Same as LATEST, but excludes any SNAPSHOT dependencies.

The last two are true virtual versions as there is no POM with such version set. In Maven 2, virtual versions are only used by plugin dependencies, Mercury treats them as first class citizens and can process virtual versions anywhere in artifact metadata (with the exception of version range definitions).

SNAPSHOT is a much more convoluted virtual version, and there are a lot of problems and contradictions involved in the resolution of SNAPSHOT dependencies:

The local repository could house both timestamped snapshot binaries and -SNAPSHOT binaries, like mercury-artifact-1.0-alpha-6-20090305.234653-4.jar and mercury-artifact-1.0-alpha-6-SNAPSHOT.jar. Let’s call them timestamped snapshots (TS) and non-timestamped snapshots (SN) respectively . While the install plugin does create non-timestamped (SN) binaries, the deploy plugin only creates timestamped snapshots (TS) binaries in the remote repopository. Remote repositories usually contain only timestamped (TS) snapshots, but I saw several cases where people copied the contents of a local repository to a server and hoped to use it as a remote repository.

The file’s last modification date on timestamped snapshot (TS ) could be before or after the non-timestamped snapshot (SN), if one exists. And it’s hard to select a correct version based on either name or modification date.

All these different options prevent consistent processing of virtual versions by repository management code. While LATEST and RELEASE are more or less well defined and allow formal processing, SNAPSHOT leaves a lot to wish for. The above listed scenario’s don’t allow a consistent treatment of snapshots. Take for example Mercury – if it uses a select non-timestamped (SN) artifacts if they exist strategy, there is a good chance that the non-timestamped artifact isn’t as recent as the timestamped artifact – leading to possible compilation errors. On the other hand, if Mercury tries to avoid this situation by looking at file timestamps, one bad file date could wreak havoc.

To summarize – there is no “right” solution, as each approach could be broken easily by primitive and often inevitable repository modifications. By removing these literal “SNAPSHOT” files and requiring all snapshots to be timestamped, we can make the SNAPSHOT virtual version as consistent and reliable as the RELEASE and LATEST virtual version. Mercury, tries to be as backward-compatible as possible, recreating the already established algorithm for resolving SNAPSHOT dependencies.

The real solution, the way to fix this mess, is to get rid of literal “SNAPSHOT” files in the repository altogether.

Please check this posting first. It contains the latest information on the subject.

<br/> <br/>

What exactly is provided:

• Configuration
  • named authentication elements to be used by repository or proxy authentication
  • supports both name/password or certificate pointer (file or URL) plus optional password
  • named repository configurations
  • local repositories (not limited to one)
  • remote repositories
  • stream verification per repository – separate for reading and writing
  • currently supported SHA-1 and PGP (.asc)
• named dependency sets
  • lists dependencies and optionally – their Maven scopes
  • standardized on a one-string naming convention
  • supports OSGi-syntax version ranges
• ant path creation/alteration
  • a task that either creates a new ant path or adds to existing one
  • optionally allows to specify Maven-like scope for the path resolution
• repository writes
  • allows to write named files into repositories
  • optional classifiers, types
  • signature generation is configured per repository
  • SHA1, PGP

Where to get mercury-ant-tasks?

All the development releases will be deployed to this site; grab the latest snapshot and:

• put it into ${user.home}/.ant/lib
• if you plan to use PGP signatures – also put there bcpg-jdk15-140.jar and bcprov-jdk15-140.jar from this central repository location
• if interested – my test build.xml contains all the unit test scripts

mercury-ant-tasks howto-by-example

This snippet shows how to define mercury namespace in the ant project markup

<project name="mercury-ant-test" default="compile"
xmlns:merc="antlib:org.apache.maven.mercury.ant.tasks" > ... </project>

This snippet shows how to define two repositories – one remote, one local. You don’t have to always define a local repository, but in this case binaries would reside in temp. location and will be reloaded from the remote on each invocation of the merc:resolve task

<merc:config id="conf">
  <merc:repo id="localRepo" dir="/maven.repo"/>
  <merc:repo id="central" url="http://repo1.maven.org/maven2"/>
</merc:config>

This snippet shows how to define authenticated repository and hide the password in the external file, so that each OS user can have their own credentials

<property file="~/.secret/secret.properties"/>
 
<merc:config id="conf">
  <merc:auth id="repo-auth" name="${repo.user}" pass="${repo.pass}"/>
  <merc:repo id="localRepo" dir="/maven.repo"/>
  <merc:repo id="central" url="http://repo1.maven.org/maven2"
             authid="repo-auth"/>
</merc:config>

This snippet shows how to configure a repository for PGP signature generation and checking; here secret keyring password is also kept in external file. Key ID always is a 16-digit hex number – use GnuPG GUI to look it up (and also generate/manipulate your keys). The verifier will accept any key from pubring.gpg for the signature

<property file="~/.secret/secret.properties"/>
 
<merc:config id="conf">
 
  <merc:auth id="repo-auth" name="${repo.user}" pass="${repo.pass}"/>
 
  <merc:repo id="localRepo" dir="/maven.repo"/>
 
  <merc:repo id="central" url="http://repo1.maven.org/maven2"
                    authid="repo-auth">
 
    <merc:verifywrite type="pgp">
      <property name="keyring" value="/home/me/pgp/secring.gpg"/>
      <property name="pass" value="${secret.keyring.pass}"/>
      <property name="key" value="0EDB5D91141BC4F2"/>
    </merc:verifywrite>
 
    <merc:verifyread type="pgp">
      <property name="keyring" value="/home/me/pgp/pubring.gpg"/>
    </merc:verifyread>
 
  </merc:repo>
 
</merc:config>

This snippet shows how to define authenticated repository and hide the password in the external file, so that each OS user can have their own credentials

<property file="~/.secret/secret.properties"/>
<merc:config id="conf">
  <merc:auth id="repo-auth" name="${repo.user}" pass="${repo.pass}"/>
  <merc:repo id="localRepo" dir="/maven.repo"/>
  <merc:repo id="central" url="http://repo1.maven.org/maven2"
                    authid="repo-auth"/>
</merc:config>

This snippet shows how to define a dependency set. Junit will only be added to paths that define scope=”test”

<merc:dep id="my-libs">
  <merc:dependency name="asm:asm:[2.0,3.0)"/>
  <merc:dependency name="junit:junit:[4.0.1,):::test"/>
  </merc:dep>

This snippet shows how to resolve a dependency set and add the resulting binaries to “compile-path” path. This path will not include JUnit from previous snippet

<merc:resolve pathid="compile-path" depid="my-libs" configid="conf" />
<javac srcdir="${src}" destdir="${target}" classpathref="compile-path" />

This snippet shows how to resolve a dependency set in test scope and add the resulting binaries to “test-compile-path” path. This path will have JUnit

<merc:resolve pathid="test-compile-path"
                      depid="my-libs"
                      configid="conf"
                      scope="test"
/>

This snippet shows how to write a jar file and it’s sources to a repository

<merc:write repoid="remoteRepo"
                   name="t:t:1.0"
                   file="${basedir}/target/t.jar"
/>
<merc:write repoid="remoteRepo"
                   name="t:t:1.0:sources"
                   file="${basedir}/target/t-src.jar"
/>

mercury-ant-tasks reference

Now, let’s have a closer look at ant mercury markup. To make it available to ant, provide a namespace definition on the project level, for example: <project name="test" default="compile" xmlns:merc="antlib:org.apache.maven.mercury.ant.tasks">

Everywhere here I mention named – named configuration, named set. It means named in ant sense: something that has an id attribute value, and then we can reference that particular object (configuration, set, etc) by this value.

• Decouple major Maven components, making them available as stand-alone building blocks rather then having Maven as as one big monolith, not usable outside of its environment.
• Artifact – clearly separate an Artifact from its metadata.
• Repository – convert a repository into active component. It used to give back just pathOf(), now it accepts GAV collections and gives back either metadata or full blown Artifacts.
• Transport – an API in development.
• DependencyTreeBuilder main API for dependency graph creation and conflict resolution.
• Decouple container, so that these components are just plain pojos.
• Introduce Jetty-based HTTP/HTTPS and WebDAV transactional transport layer.
• Asynchronous downloads and uploads. One of the few successful usages of Java NIO in OSS.
• Transactional operations – all-or-nothing for file sets.
• Move integrity control into transport layer, upper level components should not care about these details.
• Abstract out metadata cache, provide at least one implementation there.
• Integrate these changes back to Maven 3.x to make it even better platform than it is right now (if it is possible to be better )

Currently Mercury can already be used for accessing repositories, and conflict resolution is under testing.

All that, plus the necessity to provide tool integration, led to the idea to introduce an event framework into Mercury. All major processing object implement EventGenerator interface, that allows client code to register MercuryEventListener implementations and unRegister them as well.

MercuryEvent in itself has

• local TZ based start timestamp
• long duration in millis
• enumeration based type, which is also used for event partitioning
• name – obviously to identify this event
• tag to record lightweight information about event
• result to record the outcome of the event
• Map based payload – for tool integration to get access to typed objects, event wants to carry to the tool

Client code has several options to play with:

• event listener can declare a BitSet based mask of event types it wants to be notified about
• event manager could be configured with a mask of event types it will propagate
• there is a system property to define which event types should be propagated by the event manager, if one is created by the client code

The event manager itself separates event dispatching from event generator code via a thread pool, so that events are only placed into an event queue in the event generator thread, and then are sent out to listeners in parallel to the main code.

There is also a DumbListener, implemented to just peek at the events – prints them out to system.out or any other output stream.

Oleg needed a version of project builder that didn’t require a bunch of dependencies from the Maven trunk, as he designed Mercury to be a standalone module, resolving of any type of artifact, not just Maven specific ones. The bulk of the project builder work was already standalone and isolated into maven-shared-model, a component that handles inheritance and interpolation of generic models (including XML ones). So I stripped out some of the interfaces and a few classes from the maven-project component, which had unneeded tie-ins with the rest of Maven core.

Next came the implementation. Mercury has an interface called DependencyProcessor, with the following method that I needed to implement (MavenDependencyProcessor):

public List getDependencies(ArtifactBasicMetadata bmd, MetadataReader mdReader)

The point of this method is to build a full project model, including inheritance, interpolation and profiles, returning the list of ArtifactBasicMetadata dependencies of the processed model to Mercury.

The ArtifactBasicMetadata is a simple value class, giving the MavenDependencyProcessor instance information like version, groupId, artifactId of the artifact for which to process the returned dependency list. The MetadataReader allows the MavenDependencyProcessor instance a way to call back into Mercury to get the parent of the project.

Since the project builder uses basic property lists to store all Maven properties, converting from dependencies to ArtifactBasicMetadata couldn’t be simpler: The core of the conversion is:

List modelProperties = container.getProperties();
ArtifactBasicMetadata metadata = new ArtifactBasicMetadata();
for ( ModelProperty mp : modelProperties )
{
  if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.groupId)) {
       metadata.setGroupId(mp.getValue());
  } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.artifactId)) {
       metadata.setArtifactId(mp.getValue());
  }  else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.version)) {
       metadata.setVersion(mp.getValue());
  } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.classifier)) {
       metadata.setClassifier(mp.getValue());
  } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.scope)) {
       metadata.setScope(mp.getValue());
  } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.type)) {
       metadata.setType(mp.getValue());
  } else if(mp.getUri().equals(ProjectUri.Dependencies.Dependency.optional)) {
       metadata.setOptional(mp.getValue());
  }
}

One problem we did encounter was how to handle passing in system and user properties for interpolation and profile activation to the MavenDependencyProcessor, as these properties are currently passed by the MavenEmbedder to the project builder. This requires that Maven core explicitly initialize the MavenDependencyProcessor, passing in all properties through the MavenDependencyProcessor constructor.

Mercury integration is pretty exciting because it will allow us to hemorrhage a lot of resolving code which we have to use in the core, as well as providing a deterministic resolver. Full Mercury integration is not slated for the Maven-3.0-alpha1 release and will likely be a core focus of the alpha-2 release. Initial integration looks promising, as Mercury is already able to process classpaths and dependencies from POMs.

In Mercury we made an attempt to bring some contemporary ideas into Maven, make Maven not a monolithic build system, but rather a lego-like construction set that allows users to create systems they need. One set of building blocks in particular – implementation of the idea that dependencies is a universal commodity, that exists outside of Maven world. And operation of resolving conflicts on those dependencies can apply to a broader range of dependencies, not just <dependencies/> tag in the POM file.

For example – it should be relatively easy to plug in an implementation to read OSGi bundle dependencies. Or read a .properties file with dependencies.

Or – my favorite – keep dependencies in a Drools-backed DSL file. I will try to create this implementation after we integrate Maven POM dependency reader into Mercury.

How is it done:

• Dependency trees and manipulated by DependencyTreeBuilder object
• DependencyTreeBuilder instance is created out of a collection of Repositories (to find Artifacts) and and instance of DependencyProcessor.
• DependencyProcessor abstraction is that externalization component for reading/processing depedencies, making the builder independent of actual dependency information storage

Example code

1
2
3
4
5
6

    DependencyTreeBuilder dtb
        = new DependencyTreeBuilder( null, null, null, reps, processor );
    MetadataTreeNode root
        = dtb.buildTree( new ArtifactBasicMetadata( "org.apache.maven:maven-core:2.0.9" ) );
    List cp
        = dtb.resolveConflicts( ArtifactScopeEnum.compile );

This example demonstrates full dependency resolution cycle:

• full tree is created in root variable
• compile classpath is stored in variable cp.

The two initialization steps required:

• create a collection of repositories in reps
• get an instance of DependencyProcessor into processor

Also interesting – resulting classpath is a list of artifact metadata objects, not artifact binaries. So in order to actually bring the binaries to local machine, we’ll need one more call – to VirtualRepositoryReader, which I will describe it in a separate posting.

What’s important here is the fact that operation of fetching binaries from remote repositories is separated from building dependency tree as well as from resolving conflicts. Which means that you can call all of those operations independently to construct a workflow you need for a particular application. All the metadata that was fetched in the process of creating dependency tree is cached by a RepositoryMetadataCache component, that complies with repository update policy.