Brian Fox sat down with Ken Rimple of Chariot Solutions to talk about Nexus and to put repository management in the context of recent developments with Maven. Ken Rimple and Chariot have been long-term partners with Sonatype supporting our Maven training efforts, so Ken has a lot of background about Maven to ask some interesting questions.

Listen to Chariot TechCast, Episode #71, Brian Fox of Sonatype on Nexus 2

The Legacy of Maven: Binary Reusability

The interview leads off with a discussion about Maven, Maven’s history, and some of the recent developments surrounding the Maven ecosystem. Brian identifies binary reusability and declarative builds as the two important legacies of Maven:

“One of the unique things that Maven brought to the table, and what may in fact be the legacy for Maven years down the road, is that it introduced the concept of having binary reusability and not rebuilding the world as everybody was used to doing. The other aspect was making things more of a declarative model especially the dependencies. That was all unique at the time.”

What it takes to Run Central?

Ken and Brian discuss the introduction and development of Central from the beginning of the repository to the current iteration. From the initial efforts to create a single coordinate system for artifacts in 2001 and 2002. Brian’s very involved in the effort to maintain Central so this is your chance to hear Brian discuss some of the internals of the effort: how much bandwidth does Central consume? how much effort is involved in maintaining Central? and what are the day-to-day operations for running Central?

Key quotes about Central: “We’re approaching half a Terabyte for artifact storage” and “We’re seeing 50 to 60 new projects added every day”.

You will hear about how Nexus is used to enforce standards for artifacts added to Central from forges like Apache, JBoss, java.net, Codehaus, as well as the instance of Nexus that Sonatype provides for independent projects: http://oss.sonatype.org.

New Features in Nexus 2.0

Brian then discusses the important features we’re introducing with Nexus 2.0. Including support for .NET, the Repository Health Check, and our support for distributed proxies.

Again, if you haven’t listened to it, you should. Go over to Chariot Solutions and listen to Chariot Techcast Episode #71 – better yet, why don’t you just open up iTunes and subscribe to Chariot’s podcast? here.

Sometimes Releases are Disposable

For example, consider a company that creates a large web-based system. They may deploy new versions of components to production multiple times a day. If this seems unrealistic, know that system administrators for popular services like Facebook, Last.fm, and Flickr have talked openly about how frequently code is pushed to production systems. A few times a day isn’t odd in some of these environments, and with a large system, you’d consume terabytes of space to keep all of those old releases around.

Last time I worked at a large consumer-focused web site, pushing something to production once a day wasn’t uncommon, and even the idea of rolling back to anything other than yesterday’s build was laughably impractical. If you identified a bug in a CMS or a production database, you’d just craft the fix it and move on. This is especially true of larger, web-facing systems in which the only reality is the code that is running in production today. A site like Flickr gains nothing from being able to roll back to a release from last September.

The Other Side of the Coin: Releases are Forever

Compare this to the production release schedule of a critical, supported product and it’s like night and day. If you are coding some serious banking system you might be lucky to have a release once a month. In all likelihood you might be looking at a quarterly release.  When you are working on critical applications, ship software, or have infrequent release cycles then the ability to roll back to previously released binaries is very important. When you work somewhere fast-paced with a very short release cycle, there’s not much value in retaining older releases.

For this reason, we’re clarifying this point: if it makes sense for you to delete release artifacts from a hosted repository, go for it. Just remember to re-index the server once you’ve manipulated the storage folder.

We put together the following video to share more of our thoughts on this topic.

• The critical functions of enterprise repository managers
• The key role they play in a robust development environment
• How to greatly speed up your builds
• How the sharing of artifacts encourages collaboration between development groups
• How to protect yourself from unnecessary security, quality, and licensing risks

Webinar details:

• Tuesday, April 19
• 1:00pm EDT (GMT – 4:00)
• Duration: 30 minutes
• Presenter: Brian Fox, Sonatype Vice President of Engineering

Register Now!

Brian will also be taking questions on the exciting new features recently introduced in Nexus, the Sonatype repository manager.

All registrants will receive access to the recording after the event so if something comes up and you can’t make it, you won’t be missing out.

We’ve added a new webinar to the Sonatype series: Enterprise Repository Management.

Do you develop Java applications using open source software artifacts from Maven Central? If so, and you aren’t yet using a local repository manager, you need to attend this webinar. In this session you’ll learn how an enterprise repository manager can reduce development time, improve quality, enable greater internal collaboration, and reduce risk. Register now to learn how repository management can help your organization.

• Date: Tuesday, April 19, 2011
• Time: 1:00PM EDT (GMT – 04:00)
• Duration: 30 minutes
• Presenter: Brian Fox, Sonatype Vice President of Engineering
• To register, please click here

Learn best practices, central concepts, and complete integration for Maven, Nexus Professional, and m2eclipse. Sonatype books offer the latest content for the software development tools you depend on.

The third book in our series of books available for downloading is Repository Management with Nexus.

Nexus Professional is a next-generation repository manager that can proxy remote repositories, host internal repositories, encourage collaboration, and increase efficiency.

Repository Management with Nexus walks you through the installation process for both the Open Source version of Nexus and Nexus Professional, and it covers important topics such as:

• Installing Nexus as a Service on a Linux Machine
• Nexus Best Practices
• Managing Security with Nexus’ Fully Customizable Role-based Authentication System
• Reading Nexus System Logs
• Installing Nexus as a WAR in an Existing Application Server
• Configuring Nexus Professional’s LDAP Authentication Realm
• Configuring Nexus Professional’s Staging and Procurement Features

To download Repository Management with Nexus, please click here.

The Refcard covers:

• Repository management
• Repository coordinates
• Project dependencies
• Remote repositories
• Proxy repositories
• Hosted repositories
• and more…

Click here for an in-depth look at Maven Repository Management.

Nexus Open Source 1.4.1 Highlights

Nexus Plugin Console

Use the Nexus Plugin Console to list all installed Nexus plugins and browse REST services made available by installed Nexus Plugin. To open the Nexus Plugin Console, click on the Plugin Console link in the Administration section of the Nexus menu as shown in the following figure.

Once you open the Nexus Plugin Console, you will see a list of plugins installed in your Nexus installation. Clicking on a Nexus plugin in this list will display information about the plugin including: the plugin’s name, the plugin version, status, a description, SCM information about the plugin, and the URL of the plugin’s project web site.

Once you have selected a plugin from the list of installed plugin, you can browse the available REST interfaces by selecting the REST Services tab as shown in the following figure. Each plugin can contribute one or more REST services to Nexus.

This is just the first step in a fully interactive plugin console to install, remove and upgrade plugins on the fly.

An Improved Welcome Page

The welcome page for Nexus Open Source is now an easy-to-use search interface. Load Nexus in a browser, and start locating artifacts by class name, project name, or GAVC coordinate right away without having to wade through a complex search interface. This release of Nexus 1.4.1 brings the usability features Sonatype added to http://repository.sonatype.org to your own instance of Nexus.

Over 90 Stability and Usability Fixes

The 1.4.1 release of Nexus Open Source contains over 90 fixes and enhancements. From stability changes that affect the way Nexus responds to the unexpected to subtle changes to the way Nexus stores an index, the Nexus community continues to find ways to improve the efficiency and stability of a project that has become essential open source infrastructure for several large open source projects.

Nexus Professional 1.4.1 Highlights

Nexus Professional gives you the ability to store/edit/search custom metadata. This is part of a much larger index and searching upgrades coming down the road.

Viewing Artifact Metadata

When browsing repository storage or a repository index, clicking on an artifact will load the Artifact Information panel. Selecting the Artifact Metadata tab will display the interface shown in the following figure.

Artifact metadata consists of a key, a value, and a namespace. Existing metadata from an artifact’s POM is given a urn:maven namespace, and custom attributes are stored under the urn:nexus/user namespace.

Editing Artifact Metadata

Nexus Professional gives you the ability to add custom attributes to artifact metadata. To add a custom attribute, click on an artifact in Nexus, and select the Artifact Metadata tab. On the Artifact Metadata tab, click on the Add… button and a new row will be inserted into the list of attributes. Supply a key and value and click the Save button to update an artifact’s metadata. The following figure shows the Artifact Metadata panel with two custom attributes: “approvedBy” and “approved”.

Searching Artifact Metadata

Nexus Professional provides you with the ability to configure custom artifact metadata and search for artifacts with specific metadata. To search for artifacts using metadata, click on the Advanced Search link directly below the search field in the Nexus application menu to open the Search panel. Once in the search panel, click on the Keyword Search and click on Metadata Search in the search type dropdown as shown in the following figure.

Once you select the Metadata Search you will see two search fields and an operator dropdown. The two search fields are the key and value of the metadata you are searching for. The key corresponds to the key of the metadata you are searching for, and the value contains the value or value range you are searching for. The operator dropdown can be set to Equals, Matches, Bounded, or Not Equal.

One you locate a matching artfiact in the Metadata Search interface, click on the artifact and then select the Artifact Metadata to examine an artifacts metadata as shown in the following figure.

Uploading Metadata

You can also upload metadata from your build directly by attaching an RDF formatted file with the classifier “metadata”. Using this method, you can define your own namespaces that can be searched along with the defaults mentioned above.

Here’s an example of an RDF formatted file:

<urn:mycustomspace/artifact#test:release-deploy:1.0.1::jar> a <urn:mycustomspace#artifact> ;
    <urn:mycustomspace#repositoryId> "foo" ;
    <urn:mycustomspace#mavenVersion> "2.2.1" ;
    <urn:mycustomspace#releaseManager> "brianf" ;
    <urn:mycustomspace#codeCoverage> ".85" ;

You can use the maven-resources-plugin to filter and generate the data on the fly and attach it to your build with the buildhelper-maven-plugin:attach goal today. Once Nexus sees the file being deployed, the metadata is automatically added to the index on the fly.

We are working on a maven plugin to generate and manage the data from your build directly so stay tuned.

REST API

As always, everything in Nexus is available via a REST API (the UI is simply a REST client running in AJAX). All of the above operations are available for integration with plugins or other systems you may have.

Video Walk-through of the Custom Metadata Plugin

Searching the Central Maven Repository Just Got Easier from Sonatype on Vimeo.

To search for an library or an artifact:

1. Go to http://repository.sonatype.org
2. Type in an artifact or group identifier, a class name, or a checksum.
3. Press Enter.

Once you have located the artifact you were looking for, click on the artifact and copy the dependency XML from Nexus to your POM.   It couldn’t get any easier or intuitive.   Make sure to bookmark http://repository.sonatype.org.

New to the Professional version is OSGI Bundle Repository support. Stuart summed it up nicely in his detailed post:

…but how does this relate to Nexus Professional? Well with Nexus Pro 1.3.5 you will be able to:

1. dynamically generate OBR metadata for your existing repositories
2. proxy and cache remote OBRs, including both metadata and bundles
3. host local OBRs and deploy bundles into them using the UI or Maven
4. group all of the above types of OBRs into a single merged OBR
5. apply CRUD privileges to control access to particular bundles

Basically the same things you can do with Maven and P2 repositories you can now do with OBR!

Other than the new OBR plugin, this release was primarily to fix some bugs that cropped up since the 1.3.4 release a few weeks ago. We deem these fixes important and recommend everyone upgrade to the latest release.

We run 3 Nexus instances at Sonatype (repository.sonatype.org, oss.sonatype.org, osgi.sonatype.org) and one at Apache (repository.apache.org). This puts us in a unique position to identify and fix issues before they become an issue for our users and the orgainzations that have started to rely on Nexus for software distribution.

We have about 40 repositories in our r.s.o instance and we’ve been running with just 256mb of ram, intentionally constrained to keep an eye on any memory issues. In this case it worked as we started having oom exceptions (Nexus-2229). After some investigation, we determined that our configuration of ehcache was not optimal for large instances. We use ehcache to maintain the negative caching results for artifact lookups. The trouble is that there was a limit per repository but each repo had it’s own cache. This meant the total memory footprint of the caching was unconstrained and grew linearly with each new repository. In 1.3.5 we reduced the count and in 1.4 we’ve redesigned the caching to share a single cache across all the repositories. This way the caching can be optimized for the size of your instance, but out of the box the memory consumption is bounded.

On the Apache instance, we discovered that the snapshot remover didn’t like projects using non-unique snapshots. Generally when you have a repository manager that is able to clean your old snapshots, there’s little point in using non-unique (not timestamped) snapshots. Regardless, Nexus should support all valid Maven use cases so this was included in 1.3.5.

The most potentially serious condition we discovered (Nexus-2121)was that the Evict unused proxy artifacts task would evict items from a hosted repository if configured to run on one (or if all repositories was selected). When this was uncovered, we notified the user list of the problem to avoid any user impact. Fortunately, Nexus never deletes any files directly, and delete operation simply moves it to a trash folder, which is manually purged (or scheduled). That means no data was lost and the problem is now fixed in 1.3.5.

You can see the rest of the issues fixed in this release here and then get a copy of Nexus OSS or Nexus Pro

What is a Repository Manager?

• A proxy for remote repositories which caches artifacts saving both bandwidth and time required to retrieve a software artifact from a remote repository, and
• A host for internal artifacts providing an organization with a deployment target for software artifacts.

In addition to these two core features, a repository manager also allows you to manage binary software artifacts through the software development, quality assurance, and production release lifecycle. In addition to these core features, a repository manager can search software artifacts, audit development and release transactions, and integrate with external security systems such as LDAP. A repository manager is a powerful tool that encourages collaboration and provides visibility into the workflow which surrounds binary software artifacts.

A richer, more detailed description of the features of a repository manager include:

Management of Software Artifacts

A repository manager is able to manage packaged binary software artifacts. In Java development, this would include JARs containing bytecode, source, or javadoc. In other environments, such as Flex, this would include any SWCs or SWFs generated by a Flex build.

Management of Software Metadata

A repository manager should have some knowledge of the metadata which describes artifacts. In a Maven repository this would include project coordinates (groupId, artifactId, version, classifier) and information about a given artifact’s releases.

Proxying of External Repositories

Proxying an external repository yields more stable builds as the artifacts used in a build can be served to clients from the repository manager’s cache even if the external repository becomes unavailable. Proxying also saves bandwidth and time as checking for the presence of an artifact on a local network is often orders of magnitude faster than querying a heavily loaded public repository. Proxying an external repository such as the Central Maven repository is also an act of good citizenship; reducing the bandwidth burden on Central helps to preserve a valuable public resource.

Deployment to Hosted Repositories

Organizations which deploy internal snapshots and releases to hosted repositories have an easier time distributing software artifacts across different teams and departments. When a department or development group deploys artifacts to a hosted repository, other departments and development groups can develop systems in parallel, relying upon dependencies served from both release and snapshot repositories. Finding an efficient way to distribute the binary software artifacts during the development cycle is essential for an organization that needs to scale system complexity and number of developers. Once you start using Nexus as a sharing mechanism across development teams, each team can then focus on smaller, more manageable systems. The web application team can focus on the code that directly supports the web application while it depends on the binary software artifacts from a team managing an Enterprise Service Bus.

Searching an Index of Artifacts

When you collect software artifacts and metadata in a repository manager, you gain the ability to create indexes and allow users and systems to search for artifacts. With a Nexus index, an IDE such as Eclipse has almost instantaneous access to the contents of all proxy repositories (including the Central repository) as well as access to your own internal and 3rd party artifacts. If a user needs to search for a particular artifact, they can use the built-in auto-completion capabilities of Eclipse, and the IDE will perform a query against an index of the repository. If you need to update a library to the latest version, click on the POM editor and use the auto-complete feature in m2eclipse. If you need to search for all artifacts which contain a specific class name, you can use m2eclipse to search an index of Maven repository artifacts by class name. While the Central repository transformed the way that software is distributed, the Nexus index format brings the power of search to massive libraries of software artifacts.

Infrastructure for Artifact Management

A repository manager should also provide the appropriate infrastructure for managing software artifacts and a solid API for extension. In Nexus, Sonatype has provided a plugin API which allows developers to customize both the behavior and functionality of the tool. Here are just some of the features which are available as Nexus Plugins in Nexus Professional: Release Audits and Compliance, Support for Workflow and Process, Integration with External Security Providers.

Enterprise Repository Management

Once you adopt the core features of a repository manager, you start to view a product like Nexus Open Source or Nexus Professional as a tool which enables more efficient collaboration between development groups. Nexus Professional builds upon the foundations of a repository manager and adds capabilities such as Procurement and Staging.

Managing Project Dependencies

Many organizations require some level of oversight over the open source libraries and external artifacts that are let into an organization’s development cycle. An organization could have specific legal or regulatory constraints which require every dependency to be subjected to a rigorous legal or security audit before it is integrated into a development environment. Another organization might have an architecture group which needs to make sure that a large set of developers only have access to a well-defined list of dependencies or specific versions of dependencies. Using the Procurement features of Nexus Professional, managers and architecture groups have the ability to allow and deny specific artifacts from external repositories.

Managing a Software Release

Nexus Professional adds some essential workflow to the process of staging software to a release repository. Using Nexus Professional, developers can deploy to a staging directory which can trigger a message to a Release Manager or to someone responsible for QA. Quality assurance (or a development manager) can then test and certify a release having the option to promote a release to the release repository or to discard a release if it didn’t meet release standards. Nexus Professional’s staging features allow managers to specify which personnel are allowed to certify that a release can be promoted to a release repository giving an organization more control over what software artifacts are released and who can release them.

Integration with LDAP

Nexus Professional integrates with an LDAP directory, allowing an organization to connect Nexus to an existing directory of users and groups. Nexus authenticates users against an LDAP server and provides several mechanisms for mapping existing LDAP groups to Nexus roles.

Advanced Security

Using Nexus Professional, an organization can define a master User Password Encryption Key. Each user will be given a separate Maven settings file with an encrypted password using the Maven Nexus plugin. When users interact with Nexus, Nexus uses the User Password Encryption Key to decrypt a user’s Nexus credentials avoiding the need to send an easily compromised plain-text password over the network.

Settings Templates

Nexus Professional allows you to define Maven settings templates for developers. Developers can then automatically receive updates to Maven settings (~/.m2/settings.xml) using the Maven Nexus plugin. The ability to define Maven settings templates and to distribute customized Maven settings files to developers makes it easy for an organization to change global profiles or repository configuration without relying on developers to manually install a new settings file in a development environment.

p2 Repository Support

Nexus Professional supports the p2 repository format used by the new Eclipse provisioning platform. You can use the p2 plugin to consolidate, provision, and control the plugins that are being used in an Eclipse IDE. Using Nexus procurement, repository groups, and proxy repositories to consolidate multiple plugin repositories, an organization can use Nexus Professional to standardize the configuration of Eclipse IDE development environments.

For more Information about Sonatype Nexus: http://www.sonatype.com/products/nexus

To download your free trial of Nexus Professional: http://www.sonatype.com/products/downloads