First, the how, what, and when of a continuous integration server. Continuous integration (CI) servers are an established fact of of modern development infrastructure. It is a server which, for the most part, waits and watches. It keeps a vigilant eye on your source control system and jumps into action every time it sees a code change. When code changes, your CI system is usually configured to run the entire build, execute all of your unit and integration tests, and send out an email to every developer if it identifies a defect or a failed test.

It does this so that you will have an easier time identifying where a particular problem was introduced to the source code. If John checks in some bad code, the CI system runs the build immediately, and about 30 minutes later, everyone in the group receives an email with the subject header “John just broke the build”. It is a great way to identify errors, and it is also a great way to motivate developers to test locally before committing to a source control system as no one likes to be the reason for a build failure email.

Running a CI server is more than “just a good idea”. Once your system reaches a certain level of complexity you can’t scale a system without commiting to continuous integration and testing. If you don’t have continuous integration, you end up having to put all development on hold each time you want to perform a release. If you don’t build, test, and deploy your system on a regular basis – if it isn’t something that is well rehearsed, integration becomes a time consuming nightmare of manual testing and builds that often leads to inconsistent builds. This is especially true if your development effort spans multiple systems and multiple development workgroups. You run a CI system because building, testing, and deploying your system should be automatic: it should be as trivial as pressing a button.

The concept of a CI server is only slightly more established than a repository manager, and very often you will see that an organization has identified the need for a CI server before they’ve identified the need for a repository manager. If you are coding a complex system, there is a very good chance that you are already running a CI server. The most popular servers out there are Hudson, Bamboo, and CruiseControl. While the connection between CI servers and repository managers might not be immediately obvious, when used together they can introduce some new possibilities for the way you develop your systems.

Continuous Publishing

When you have a system to continuously build your code, you also have a system that can continuously publish SNAPSHOT artifacts to a repository manager to enable a more granular approach to development. What do I mean by “a more granular approach to development”? To answer that question, let’s take a look at a complex multi-module project using the example of the eCommerce group from the previous post in this series.

Assume you have a new programmer starting tomorrow. Instead of throwing him at the entire forty-thousand lines of code, you would like to be able to give that developer a small, easy to digest task. You want this developer to add support for PayPal’s Adaptive Payments API in your eCommerce system. That’s it. You don’t want them to be distracted by the overwhelming scope of the project, and you certainly can’t afford for them to take a three month voyage through your project’s code before they start contributing to the effort. Deadlines are tight, and you don’t have enough people on your team. It is important that new hires start programming as soon as they walk in the door.

Without a repository manager hooked up to a continuous integration server, if you try to checkout just the ecom-paypal project, the build is going to fail because it will try to download dependencies from a repository manager. In the case of the ecom-paypal project, assume that the dependency graph looks like this.

When you have a repository manager and a continuous integration server, you can configure your continuous integration server to publish SNAPSHOT artifacts (in-progress SNAPSHOT binaries) to your repository manager. This will allow you to just check out a single, isolated portion of a much larger multi-module project.

Without a repository manager, trying to build version 1.3-SNAPSHOT of ecom-paypal in isolation is going to generate errors because you are forced to checkout the entire codebase to build and install all of the dependencies in your local repository. With a repository manager, SNAPSHOT artifacts are being continuously published because Hudson is checking you SCM every few minutes and building the latest code. When you run the ecom-paypal module’s build in isolation, Maven is going to download the most recent SNAPSHOT.

Without a repository manager, your new developer is going to have to download the entire codebase and run a large time-consuming build. With a repository manager you can work on specific components of a larger multi-module project. This ability to divide and conquer your codebase comes in very handy when you need a consultant to take a look at a specific problem, or when you need to look at a coding problem in isolation.

When you continuously publish build artifacts to a repository manager, you move away from the single monolithic project build and toward a project layout and architecture that lends itself to modularization.

In tomorrow’s post: How a Repository Manager decouples deployments from source code, and what that means for developer operations.

Another reason why we’ve seen more adoption is that most developers understand the benefits of using a tool like Maven for automatic dependency management. In 2005, it was common to see projects store binary JARs alongside source code in projects. In 2010, you rely on the repository and the metadata it contains. If you use a library like Guice, you’ll add a dependency on the artifact and let your build tool take care of the details. To do otherwise would be to commit yourself to a manual work updating JARs and testing dependencies each time a new version of an external library is released.

Despite the increasing prevalence of repository managers, I still stumble upon workgroups and organizations that haven’t heard of repository management. When you ask if they are using a repository manager, they might think you are referring to Subversion or source control. This series of posts is a high-level overview of the main benefits of repository management. If you are trying to convince someone to start using a repository manager, the next few blog posts are for you.

Repository Management: The Big Picture

Compare the diagram shown above with the diagram shown below. In the next few posts, I am going to emphasize the specific benefits of using a repository manager. Specifically, I’m going to talk about:

• How a repository manager changes the development cycle
• How continuous integration is used to continuously publish internal build artifacts
• How a repository manager simplifies the process of building and deploying systems to production
• How a repository manager can act as a gateway between vendors and external partners

When you don’t use a Repository Manager

Before I get started on the benefits of repository management, I want to talk about the realities you face when you don’t use a repository manager. Here are some common anti-patterns when you don’t use a repository manager:

• All of your developers download artifacts directly from public repositories. A new developer starts on a Monday. That developer will spend an hour downloading a massive library of dependencies from Maven Central. Worse, if Maven Central happens to be down that day, they will be out of luck entirely.
• Proprietary or Vendors libraries are passed around, from developer to developer. If you don’t use a repository manager, how do you distribute the Oracle JDBC driver? Maybe you place it in a shared file system and tell people to download it and install it in ~/.m2/repository. More likely, developers just pass this JAR around as an email attachment with some ad-hoc instructions.
• JARs are checked into source control. If you don’t use a tool like Maven, which knows how to download artifacts from a remote repository, you might be following the very common pattern of checking binary dependencies and libraries into source control. I’ve seen many instances of companies creating ad-hoc JAR repositories and checking these repositories into source control, only to version and branch these static binary files with every release.
• The source control repository is used to store everything from source code to binary builds. Because there is no repository designed to store binaries, developers start to use tools like Subversion to keep track of binaries. As time passes, the Subversion repository becomes an ad-hoc file system for files that have no business in an SCM.
• The continuous integration server depends on public repositories. When you change your build or add a new dependency, your CI system downloads dependencies from the public repo. It depends on the availability of this public resource to run builds.
• Production deployments have to run the entire build, from start to finish, to generate binaries for deployment. When a build is tested and then ultimately pushed to production, the build and deployment scripts checkout source code, run the build, and deploy the resulting binaries to production systems.
• Sharing source code with external partners means granting them access to your SCM. Since there is no established mechanism for publishing source or binary artifacts, the only way to share code with partners is to either send an archive of your source, or provide them with direct access to your SCM.

The general theme in all of these anti-patterns is that either your systems depend on public resources, or they all depend on the SCM system as a central collaboration point. In the next few posts, I’m going to detail how using a repository manager provides a solution for each of these issues. I’ll go into why each of these anti-patterns is a bad idea, and how you can use Maven, Nexus, and Hudson together to solve these problems and create a more efficient software development effort.

Stay tuned for the next post: Caching and Collaborating.

First of all, we need to know the default repository layout, which affects the repository path of each artifact. The related Maven source code is easy to understand, here it is:

  private static final char PATH_SEPARATOR = '/';
 
  private static final char GROUP_SEPARATOR = '.';
 
  private static final char ARTIFACT_SEPARATOR = '-';
 
  public String pathOf( Artifact artifact )
  {
    ArtifactHandler artifactHandler = artifact.getArtifactHandler();
 
    StringBuilder path = new StringBuilder( 128 );
 
    path.append( formatAsDirectory( artifact.getGroupId() ) ).append( PATH_SEPARATOR );
    path.append( artifact.getArtifactId() ).append( PATH_SEPARATOR );
    path.append( artifact.getBaseVersion() ).append( PATH_SEPARATOR );
    path.append( artifact.getArtifactId() ).append( ARTIFACT_SEPARATOR ).append( artifact.getVersion() );
 
    if ( artifact.hasClassifier() )
    {
      path.append( ARTIFACT_SEPARATOR ).append( artifact.getClassifier() );
    }
 
    if ( artifactHandler.getExtension() != null && artifactHandler.getExtension().length() > 0 )
    {
      path.append( GROUP_SEPARATOR ).append( artifactHandler.getExtension() );
    }
 
    return path.toString();
  }
 
  private String formatAsDirectory( String directory )
  {
    return directory.replace( GROUP_SEPARATOR, PATH_SEPARATOR );
  }

Take TestNG as an example, assume the artifact coordinate is groupId=org.testng, artifactId=testng, version=4.8, classifier=jdk15, and packaging=jar, the above code runs like this:

1. Calcuate the repository path from the groupId, replace the groupId separator dot with path separator slash, and append another path separator. So, org.testng becomes org/testng/.
2. Append the artifactId and a path separator. Since in this example the artifactId is testng, so the path becomes org/testng/testng/.
3. Append the artifact’s base version and a path separator. For release artifacts, base version is the version, for snapshots, timestamp will be converted to SNAPSHOT to get the base version. In this example, the base version is 5.8, so the path becomes org/testng/testng/5.8/.
4. Append the artifactId, a artifact separator hyphen, and the version. Now, the path becomes org/testng/testng/5.8/testng-5.8.
5. If the artifact has a classifier, append a artifact separator and the classifier. In this example the classifier is jdk15, so the path becomes org/testng/testng/5.8/testng-5.8-jdk15.
6. Finally, get the artifact extension from artifactHandler. Each packaging type has its own artifactHandler, so the extension is determined by packaging, for default packaging type jar, the extension is also jar. Append a groupId separator dot, and the extension. So the path is finally org/testng/testng/5.8/testng-5.8-jdk15.jar.

Now we know how Maven gets the repository path from artifact coordinate, but this is not the whole story. What if the artifact does not exist in local repository? what if the version of artifact is SNAPSHOT? How does Maven know where and how to get the latest SNAPSHOT?

When Maven resolves a dependency from a repository, these are the basic steps it follows:

1. If the scope of this dependency is system, simply resolve it from local file system.
2. Try to find the dependency from local repository, resolve it if it’s found.
3. If the version of the dependency is explicit and non-SNAPSHOT, iterate though the remote repositories to download it, and resolve it if it’s found.
4. (For version RELEASE or LATEST) When it’s required, iterate all the remote repositories to get groupId/artifactId/maven-metadata.xml and merge them with the local one.
5. (For version RELEASE or LATEST) Use the merged metadata to construct an explicit non-snapshot version, resolve it if it’s found locally, otherwise download it from remote repository.
6. (For SNAPSHOT version) When it’s required, iterate though the remote repositories to get groupId/artifactId/version/maven-metadata.xml and merge them with the local one.
7. (For SNAPSHOT version) Use the merged metadata to construct an explicit snapshot version, resolve it if it’s found locally, otherwise download it from remote repository.
8. (For SNAPSHOT version) If the resolved snapshot version is of timestamp style, like 1.4.1-20091104.121450-121,  copy the file to a non-timestamp style, like 1.4.1-SNAPSHOT, and use this file.

A few notes:

In step 4 and 6: ‘when it’s required’ means release/snapshot must be enabled for the repository, and repository updatePolicy is satisfied, or force update is enabled by -U options.

In step 5: In maven 2, LATEST will be constructed to the real latest version, no matter it’s snapshots or not, but in maven 3, only the latest non-snapshot version will be used.

in step 4: a typical groupId/artifactId/maven-metadata.xml is like this:

<!--?xml version="1.0" encoding="UTF-8"?-->
 
  org.sonatype.nexus
  nexus
 
    1.4.2-SNAPSHOT
    1.4.0
 
      1.3.5
      1.3.6
      1.3.7-SNAPSHOT
      1.4.0-SNAPSHOT
      1.4.0
      1.4.0.1-SNAPSHOT
      1.4.1-SNAPSHOT
      1.4.1.1-SNAPSHOT
      1.4.2-SNAPSHOT
 
    20091214221557

All the available versions in this directory are listed in order. The latest element points to the latest version, and the release element points to the latest release version. With the help of this file, Maven knows what the latest release version is.

In step 6: a typical groupId/artifactId/version/maven-metadata.xml is like this:

<!--?xml version="1.0" encoding="UTF-8"?-->
 
  org.sonatype.nexus
  nexus
  1.4.2-SNAPSHOT
 
      20091214.221414
      13
 
    20091214221558

The timestamp and buildnumber of the latest snapshot is listed, so Maven can understand what the latest snapshot artifact file is. In this case, it is nexus-1.4.2-20091214.221414-13.pom.

Now that you are familiar with the process Maven uses to resolve dependencies, calculate dependency paths, and resolve conflicts you can refer to this post whenever you have an unexpected failure resolving a dependency.