To illustrate this problem, consider a 1.0 build of your product that depends on foo-1.2.jar. It works great. Then you build 2.0 of your product which still depends on foo-1.2.jar.  Since then, foo-1.2.jar was patched and now breaks your application.  However the application still works for half of your developers because the original foo-1.2.jar is in their local repository and possibly proxied in another Nexus instance.

The solution is to release a new version foo-1.2.1.jar. (ie foo-1.2.2.jar or foo-1.2.1.1-jar), but that alone isn’t enough. You want to make sure that you are using a repository manager that prevents someone from updating a release artifact once it has been published.

It has always been possible to stop people from doing this in Nexus but the solution was difficult to explain.  In the Nexus 1.4 release, we have reworked and simplified the interface to encourage this best-practive.

To disable redeployment, edit your repository and set the ‘Deployment Policy’ field to ‘Disable Redeploy’ then click save.

Upgrading from a previous version of Nexus 1.4.0 will set this field to ‘Allow Redeploy’, so no existing repositories will be changed and the behavior matches previous releases. However the default value when creating a new repository is now ‘Disable Redeploy’.

When you use Nexus, you are not just using a capable repository manager, you are adopting the best-practice.

• Why Putting Repositories in your pom is a bad idea
This entry explains how to properly manage repository definitions and why you shouldn’t declare them in your poms.
• Best Practices for Releasing with 3rd Party Snapshot Dependencies
This entry describes how to most effectively create your own internal release of a snapshot dependency.
• Maven Continuous Integration Best Practices
This entry describes several techniques for running Maven builds from a CI system.
• How to Detect if you have a Snapshot Version
This entry shows the code that Maven uses to detect a snapshot version. This is helpful if you have non-standard release versions to make sure they won’t be detected by Maven as a snapshot.
• Optimal Maven Plugin Configuration
This entry gives some tips on how to configure plugins in a way that makes it easier to understand what is happening in your pom.
• Adding additional source folders to your maven build
This entry shows how to use the buildhelper plugin to add additional source folders to your build. This is common if you have tools that generate sources that don’t have a maven plugin to add the source folder automatically.
• Misused Maven Terms Defined
This entry explains several terms that are often misquoted. It also specifically discusses classifiers and attached artifacts.
• How to override a plugin’s dependency
This shows how you can override a plugin dependency in your own pom…for example if you want to use a newer version of a tool wrapped by a plugin (pmd, checkstyle, cobertura are common examples)
• How to share resources across projects
This entry will show you how to share resources in a repeatable way without using relative paths in your poms. It also shows a technique for sharing checkstyle and pmd configurations across all your modules.
• How Plugin versions are determined
This entry is actually part of the Maven 2.0.9 announcement, but explains how Maven determines which version of a plugin to use, and why you should use PluginManagement to control the versions yourself.
• How to convert from ant to maven in 5 minutes
• How to make an executable jar

Note: If you have suggestions for further topics, please leave a comment with your ideas.

Should I put the urls to my repositories in my poms or in my settings?

The short answer is: settings.

The long answer is: it depends.

There are two scenarios to consider here. Enterprise software (generally not published externally) and public software. Lets take Enterprise software first. Continue reading this post for a full explanation of both scenarios.

Enterprise

In an Enterprise scenario, you will typically want to have a repository manager like Nexus sitting on your front lines. It will be proxying all external repositories and hosting your internal ones.

When Maven looks for an artifact, it walks down the list of repositories defined in the POMs and settings until it finds what it wants. If you define your repository manager as a repository in your POMs Maven will still fall back to the built-in Central repository when artifacts are not found in the repository manager.

One possible solution to this is to overload the “central” repository id and point this at your repository manager. This can work, but you would want to do this in your corporate pom so you don’t need to define it in every project you have. Once you’ve overloaded “central”, you have a “Catch-22″. In a clean environment, you need to know where the repository is to find the parent POM — to tell you where the repository is. You will end up having to define this repository in your settings anyway just to bootstrap.

Simply redefining Central has a few other problems. Specifically, it doesn’t redirect repositories that may be defined in other POMs you include transitively. This causes a few problems: first, it will slow down your builds as Maven goes out to all the external repositories looking for artifacts…possibly even your internal ones that would never be found there. Second, it means something may build for one developer but not another. Third, it means as an organization you have no idea where your artifacts are coming from.

As an organization, you generally will want all your developers using the same set of repositories for their builds, and make all requests via a controlled mechanism. This is best accomplished with a mirrorOf * entry in your settings to redirect all lookup requests to your repository manager. It is not possible to define a mirrorOf in a POM. See this section of the Nexus Book for examples of what a good setup will look like.

With all these things considered, you can see that defining the repositories in your poms doesn’t really help you much and generally just gets in the way.

This all presumes that your artifacts are not consumed externally. If they are, then you also fall into the next category. (They are not mutually exclusive)

Open Source Projects

If you are publishing your software and others will check it out and build it, then there are more points to consider. If all your dependencies are available in the Central repository, then you have nothing else to do. If however, you have artifacts that may exist only in your repository (think snapshots of your parent POMs), or other third party repos, then developers will have a hard time building your source. Only in this case does it make sense to add repository entries to your POMs. However, there are side effects to this that should be noted:

• The entries you have defined will be burned forever into your released POMs. This means that, should the URLs change down the road, anyone consuming your POMs will face these broken URLs and have to track down the new ones manually.
• Each of these entries will end up in your consumer’s builds since Maven needs to check them all for missing dependencies. Maven currently isn’t able to tell when you introduce a repository which dependencies are supposed to be found there (and it gets worse when you consider transitivity).

If the product of your build is a tool (like Nexus) rather than components used as dependencies, then adding the repository to the POM is fairly safe. In this case, it is unlikely others will depend on your artifacts directly in some other build, and the above concerns are nullified since presumably the new source would have the correct repository URLs.

If you are publishing jars that will be consumed by others, then you should think about getting them synced to the Central repository. This means that they will always be available to all users no matter what happens to your repository or URLs, and you won’t accidentally introduce new repositories to their builds. Central will eventually reject POMs with repository elements for this reason.

Summary

So to sum up, if you want to have repeatable builds and good control over your organization internally, use a repository manager and use a mirrorOf entry in everyone’s settings.xml to point at that url.

If you are exposing your source and want to make it easy for others to build, then consider adding a repository entry to your POM, but don’t pick a URL lightly, think long-term, and use a URL that will always be under your control. If your URL has to change down the road, make sure that you will always be able to track 404s and write the appropriate mod_rewrite rules to ensure that future builds will be able to find the appropriate artifacts.

PS. If you got here from the Jfrog blog, then I suggest you read my response.

1. Validate no local changes against your SCM
2. Validate that there are no SNAPSHOT dependencies
3. Convert the modules to the to-be released version
4. Ensure the build and Unit/Integration tests succeed
5. Commit the changes to SCM, then Tag the release
6. Checkout the tag to a clean location and build/deploy the artifacts

Dealing with failures on Step #2 is where I want to focus today.

Background

Lets step back a moment and give some background on SNAPSHOTS. In Maven, all artifacts must have unique coordinates comprised of Group, Artifact, Version (there are a few more like classifier and type that aren’t relevant for this discussion… see the free maven book for more info on these.)

By definition artifacts are either Releases or SNAPSHOTS. Releases are expected to be immutable and SNAPSHOTS are temporal. A SNAPSHOT version has the SNAPSHOT keyword in the version string like 1.0-SNAPSHOT. When these artifacts are deployed to a remote repository, the SNAPSHOT portion is transformed into a time-stamp (in UTC) and an incremental build number is appended. Thus 1.0-SNAPSHOT becomes something like 1.0-20090105.231034-52 when it’s deployed.

The Problem

The reason you can’t release an artifact that depends on a SNAPSHOT is that later on, this artifact may change or simply may not be available. (Remember, SNAPSHOTS are temporal)

If you are attempting to perform a release and you have dependencies on 3rd party SNAPSHOT artifacts, you will need to resolve this before moving on. The first thing you might try is revert back to the previous release, leading to Best Practices #1 and 2:

Rule #1: Only use 3rd party SNAPSHOTS when absolutely necessary

Use SNAPSHOTs temporarily to validate fixes but don’t commit your poms this way unless…

Rule #2: If you do need it, record why you need it

Preferably as a comment in the Pom, referring to a bug number. Adherence to this rule will make it easy to determine if you actually need the new dependency. It will also make it easy to decide if you can work around it or could live without it.

Assuming you really do need it, you have a couple of choices. The first is to request a release of the artifact. Chances are though if you are at this stage of a release, it’s too late to wait.

The obvious alternative is to control your own destiny and release anyway.

Not the solution

A common anti-pattern is to find the exact time-stamp-build number of the SNAPSHOT you are depending on and put that as your dependency version. We refer to this as locking to a timestamp.

Rule #3 Never release using the Time-Stamped snapshot

There are several reasons why this is a bad idea.

1. First is that Maven and Maven Tools that mimic Maven properly are able to detect a SNAPSHOT based on the timestamp pattern. (See my previous entry showing how this is done correctly.) The release plugin will still fail if you do this because it correctly understands you have SNAPSHOT dependencies. The plugin has a flag to allow bypass this check and people unfortunately use it far too often.
2. Since Maven and related tools see these time-stamped artifacts as a SNAPSHOT, you will have to keep this artifact in a SNAPSHOT repository. This might be acceptable if you take that artifact and host it in your repository manager but you must remember to never delete it. This would be complicated if you wanted to share your release with people outside your organization.
3. The dependency could disappear from the external repository. If you aren’t using a repo manager (What? Didn’t you get the memo?) then you are totally subject to the cleanup process of the external repository.Not a good place to be.

The Solution

There are a few approaches to this that are acceptable. Which one you choose depends on your level of paranoia. This process can be recursive if your SNAPSHOT dependency has other SNAPSHOT dependencies. They are all approaches to rule #4:

Rule #4: Convert the SNAPSHOT version to a Release version and host it in your repository

• Just deploy the artifact: The simplest approach to this is to take the SNAPSHOT binary artifact itself and deploy it to your repository. Remember to adjust the Pom.xml as needed. If you use Nexus, then you can upload via the UI and the artifact will be renamed based on the version you choose. (others can upload via the ui, but I’m not sure if they rename the file for you) If you use this approach, be aware that the artifacts produced by Maven have a MANIFEST that contains the SNAPSHOT version. Maven 3.x uses this for plugins and may refuse to use this artifact based on the version not matching what was expected. The trouble with this approach is that should you need to patch this artifact down the road, you don’t know exactly what sources produced it.
• Build from the sources: The next approach (and this is my preferred method) is to checkout the sources and build the snapshot yourself. This way I know exactly what I have should I need to patch it later. If you are uber-paranoyed you can commit the source to your scm for posterity. If you trust the remote SCM (as I do for the Apache projects) then simply recording the scm revision used to retrieve the source is enough. You’ll need to change the versions of the artifacts in the POMS and then deploy it to your repository. I like to put the svn revision right into the artifact version such as 1.0-[my company name]-[svnid]. Look at an example of how we did this this for 2.1-SNAPSHOTS of Maven code used by Nexus and M2eclipse.

Rule #5: Change only the version, not the groupId or ArtifactId

When you make these new versions, resist the urge to change the groupid as well since this will change the coordinates and Maven wouldn’t detect collisions. You could end up with your copy and the official copy of the same dependencies in your build. Having changed only the version makes it much easier to switch back to the official public release when it becomes available.

#1 Automate Snapshot Deployment

In my experience, it is best to let your CI system deploy your snapshots. This is the most reliable way to guarantee that the contents of your repository are kept in sync with your source control system. In order to do this in a practical way, you need to couple CI with a repository manager like Nexus that can automatically purge snapshots. I’ve managed projects that produced >300gb of snapshots in less than a week. Using a repository manager will save your sanity.

#2 Isolate Local Repostitories

Another critical component of a good CI setup is local repository isolation. The local repository in Maven is the temporary holding spot for all artifacts downloaded and produced by Maven, and it is not currently setup to be multi-process safe.   There is a remote possibility of a conflict, but it does exist.

The main reason I like to have a local repository per project is that it’s the only way to test that your project is build-able against the artifacts in the corporate repository. If you don’t have separate local repos, then the product on one build will be seen by another build on CI, even if it’s not in the corporate repository. This is important since one function of CI should be to validate that the code is buildable by a real developer.

Tip: use -Dmaven.repo.local=xxxx to define the unique local repositories for each build.

#3 Regularly Purge Local Repositories

To further validate the contents of the repository, and to manage the disk space, I purge the local repostories every night. This way if changes in the repository or artifacts are removed, the CI system will detect this. To keep it easy to purge all the local repositories, I tend to structure them under a single common folder such as /opt/repos/*.

Obviously having many local repositories requires more disk space than a single monolitic one due to dependency duplication, but even on our large grid the repos are less than 10gb total. Local repos get giant when you don’t control the snapshots and purging them nightly keeps this under control.

Tip: use your CI system itself to schedule the local repo cleanup. This way anyone can clean the repos manually right from the UI if Maven gets confused.

Over time, I’ve also picked up a few more simple tricks:

#4 Enable Batch Mode

Tip: Enable -B (batch) mode on the build. This will make the logs shorter since it avoids the dependency download progress logging. It also ensures that the build won’t hang due to waiting for user input. (to enable globally in settings.xml:<interactiveMode>false</interactiveMode>)

#5 Enable Full Stack Traces

Tip: Enable -e to cause Maven to produce the full stack trace if there’s a build exception. This will make it easier to comprehend any problems in the resulting build failure log/email without having to build it again.

#6 Print Test Failures to Standard Output

Tip: Enable -Dsurefire.useFile=false. This is a favorite of mine since this causes surefire to print test failures to standard out, where it will get included in the build failure log and email. This saves you from having to dig back onto the machine to find the surefire report just to see a simple stack trace. (to enable globally in settings.xml:<properties><surefire.useFile>true</surefire.useFile></properties> in an active profile)

#7 Always check for Snapshots

Tip: Enable -U to cause Maven to always check for new snapshots. (to enable globally in settings.xml: <updatePolicy>always</updatePolicy>….this goes on a repository definition)

Summary

Using the above settings and process causes every build to push the artifacts to the repository. The next downstream build will have its own clean repo and check the repository manager for the latest snapshots. Then at least once a day, everything is dumped locally and all dependencies are pulled out of the repository manager.

Naturally, doing all of this updating and purging puts some network load between the CI and repository manager. This works best if they share a highspeed network. If your repository manager isn’t close to your CI system, then you should put one there, if only to proxy the artifacts and reduce the impact of the daily local repo purge.

Note: If you are going to follow these tips, it is essential that you download a copy of Nexus, purging the contents of your local repository and downloading everything from the Central Maven repository once a day (per project) is exactly the sort of behavior that causes traffic problems on the Central Maven repo.

Here is the relevant code that Maven uses to determine if an artifact is a snapshot or not:

String LATEST_VERSION = "LATEST";
String SNAPSHOT_VERSION = "SNAPSHOT";
Pattern VERSION_FILE_PATTERN = Pattern.compile(
 "^(.*)-([0-9]{8}.[0-9]{6})-([0-9]+)$" );
public boolean isSnapshot()
{
     if ( version != null || baseVersion != null )
     {
       Matcher m = VERSION_FILE_PATTERN.matcher( getBaseVersion() );
       if ( m.matches() )
       {
          setBaseVersion( m.group( 1 ) + "-" + SNAPSHOT_VERSION );
          return true;
       }
       else
       {
         return getBaseVersion().endsWith( SNAPSHOT_VERSION ) ||
         getBaseVersion().equals( LATEST_VERSION );
       }
     }
     else
     {
       return false;
     }
  }

Essentially this regex ( regular expression ) is checking if the version ends with SNAPSHOT or if the pattern matches a timestamped version such as “2.0.10-20080415.233129-15″.

The regular plugins section also allows the version and default configuration to be defined, and this is where the confusion lies. It is technically valid to define the plugin version and default configuration here, but I find it easier to grok the pom when following this guideline:

• If the plugin block is not defining an execution (and thus binding maven to do something in the lifecycle), put that block in pluginManagment

This means that configuration for plugins brought in by the default lifecycle such as resources, compiler, jar, etc will almost always go in the pluginManagement section. Doing so tends to keep your regular plugins section rather small and only showing the config for plugins that are doing additional things during the lifecycle.

Following the above defined guideline will also have you locating the config for plugins that are primarily run from the command line in pluginManagment…again keeping the plugins section clean and concise.

<project>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>1.1</version>
        <executions>
          <execution>
            <id>add-source</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>add-source</goal>
            </goals>
            <configuration>
              <sources>
                <source>some directory</source>
                ...
              </sources>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Several threads cited from Inderjeet’s blog suggest tweaking the compiler plugin’s includes settings to add the additional source folder. While this will work in simple cases, this method does not actually add an additional sourceFolder element in the project model. This means other plugins, particularly IDE integrations such as M2eclipse won’t see these folders and will cause you trouble down the road. We therefore suggest that you use the build-helper instead.

The build-helper lets you do some other handy things such as attach additional artifacts (attachedArtifacts), purge the project’s artifacts from the local repository and add additional testSource folders.

Reactor

When we refer to the “reactor” or “reactor build”, we are usually referring to the aggregation capability of Maven. If you define a pom with <modules>, Maven will read all of these modules and build then in the correct order based on dependencies. The component that does this is also called the reactor, which is why the term is often applied to the action of aggregating the builds.

Siblings

Sibling poms are poms that are built inside the same reactor, and often share a common Parent pom.

Parent Pom

A parent pom is simply a pom from which other poms inherit via a <parent> section. Poms that inherit from a parent are sometimes referred to as “children” or “child poms.”

Corporate Pom

A Corporate Pom is a Parent Pom that sits at the top of an inheritance structure for a corporation. This is a recommended best practice for centralizing certain configuration such as Enforcer rule configuration, default Plugin versions, etc. This may also be called an “Organizational Pom.” The term “Super Pom” is often mis-applied to Corporate Poms.

Super Pom

The Super Pom is a special pom that is contained inside the maven-project jar. This is the implied parent of all poms that don’t inherit from some other Parent pom. It is analogous to Java’s Object class. The Super Pom provides the defaults that make Maven easy to use, such as the Central repository, default locations for sources, etc. The Super Pom is not user editable but through the normal inheritance behavior of Maven, all the values may be overridden. To avoid confusion, the term Super Pom should be reserved for only this specific embedded pom.

Artifact

An artifact is essentially any file that is installed in a local repository or deployed to a remote repository. It has a unique id in the Maven coordinate system of groupId:artifactId:version:classifier:type.

Dependency

A dependency is an Artifact that is specifically declared in a pom to be needed for one of the various class path scopes.

Attached Artifact

Attached artifacts are additional related artifacts that get installed and deployed along with the “main” artifact. These are most often, javadocs, sources, bundles etc, but can actually be any file. A plugin must do the actual attaching, but random files may be attached with the build-helper-maven-plugin. Attached artifacts automatically share the same groupId, ArtifactId and version as the main artifact but they are distinguished with an additional Classifier field.

Classifier

The Classifier is the least known of the Maven coordinates. A primary artifact automatically has an empty classifier and is not changeable. All attached artifacts must differ from each other with unique classifiers. The Maven2 repository layout constructs file names in the following format: [groupId]-[artifactId]-[version]-[classifier].[type]. Common classifiers are: sources, javadocs, bin and bundle.

Our book shows all these elements and more in action.

How you go about doing this actually depends on your use case because of an oversight in the Maven 4.0.0 model used by the Maven 2.0.x versions.

If you are using a plugin as a normal build plugin (as opposed to a report) then you will have it bound similar to this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-checkstyle-plugin</artifactId>
  <version>2.1</version>
  <executions>
      <execution>
        <id>check my sources</id>
        <goals>
          <goal>check</goal>
        </goals>
        <phase>compile</phase>
      </execution>
  </executions>
</plugin>

This version of the maven-checkstyle-plugin will use checkstyle 4.1 by default. If I wanted to use version 4.4 instead, I simply add a dependency block inside my plugin block like this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-checkstyle-plugin</artifactId>
  <version>2.1</version>
  <executions>
      <execution>
        <id>check my sources</id>
        <goals>
          <goal>check</goal>
        </goals>
        <phase>compile</phase>
      </execution>
  </executions>
  <dependencies>
     <dependency>
        <groupId>checkstyle</groupId>
        <artifactId>checkstyle</artifactId>
        <version>4.4</version>
     </dependency>
  </dependencies>
</plugin>

That was easy, right? As long as the new version you have introduced is api compatible with the version the plugin was linked against, you should be good.

Now, what about reports? Well unfortunately, the Model used in 2.0.x doesn’t allow dependencies to be specified inside the reporting block (MNG-1948)

In the process of creating the samples for this how-to, I discovered that the extensions don’t override the reporting plugin dependencies, so unfortunately there isn’t a way to override them. Stay tuned as we investigate how to deal with this.

Update: There is a way to make this happen with reports. In the example below, I removed the execution from the plugin block and added the plugin as a report. It seems that the dependency is inherited when the plugin is used in reporting. Not quite obvious, but here’s what it looks like:

<build>
   <plugins>
      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-checkstyle-plugin</artifactId>
         <version>2.1</version>
         <dependencies>
            <dependency>
               <groupId>checkstyle</groupId>
               <artifactId>checkstyle</artifactId>
               <version>4.4</version>
            </dependency>
         </dependencies>
      </plugin>
   </plugins>
</build>
<reporting>
   <plugins>
      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-checkstyle-plugin</artifactId>
         <version>2.1</version>
      </plugin>
   </plugins>
</reporting>

Checkstyle makes it easy to test this behavior because it writes out the checkstyle version in the results:

<?xml version="1.0" encoding="UTF-8"?>
<checkstyle version="4.4">