• Search functionality that allows one to quickly track down artifacts and their dependency details when trying to resolve build problems.
• Browse functionality that aids in discovery of new artifacts to use in projects.

In the intervening months, Sonatype has focused its efforts on improving the usability of the Maven Central user interface in the hopes of making it the first place users look when trying to find an artifact.  Recently, users who have reaped the benefits of using the Maven Central website have asked about interacting programmatically with the search functionality.

If you pay attention to your web browser’s address bar when conducting searches on Maven Central, you can already see that a REST-style API exists.  For example, searching for “guice” from the main search box results in the following URL being generated (the following URL’s are NOT URL-encoded for the sake of readability):

• http://search.maven.org/#search|ga|1|guice

Translating the search request into English, that URL requests a basic search for any artifact (irrespective of version) containing the word “guice” in either the groupId or artifactId, returning only the first page of results.  Each row of the results shows the latest version of the artifact and the date the artifact was last updated as well as any classifiers associated with the artifact.

You can build up the complete library of search requests simply by paying attention to your web browser’s address field as you use the Maven Central website.  For the sake of convenience, we’ve collected all the URLs that make up Maven Central’s search API in a document available here.

Sadly, these URL’s are still only useful when requesting them via web browser.  They are links that can be bookmarked or e-mailed, but they do NOT work when using a non-browser agent like wget or curl.  The Maven Central user interface is essentially a browser-based application that uses Javascript to make asynchronous requests to yet another set of URL’s.  Once you make a request that looks like the URL above, the browser fires off the actual request to another Maven Central URL responsible for conducting the search and returning results that are formatted by the browser.

The sample request above, when converted to an actual Maven Central search request, looks like this:

• http://search.maven.org/solrsearch/select?q=guice&rows=20&wt=json

The actual text of your query goes in the appropriately named “q” parameter, the “rows” parameter restricts the results to a smaller number than the full result set, and the “wt” parameter can be either “xml” or “json,” depending on how your application prefers to handle results.

Some useful examples appear below.  Again, please refer to the API Guide for a complete listing:

• Fully-qualified classname search – http://search.maven.org/solrsearch/select?q=fc:”org.specs.runner.JUnit”&rows=20&wt=json
• GroupId and artifactId search that returns all available artifact versions – http://search.maven.org/solrsearch/select?q=g:”org.apache.maven.indexer”+AND+a:”maven-indexer”&rows=20&core=gav
• SHA1 search (you would need to pre-calculate the SHA1 before sending the request to Maven Central) – http://search.maven.org/solrsearch/select?q=1:”2d3c16092663da9041b171b8d3627cbafa8f0cb1″&rows=20&wt=json

In an upcoming post, I’ll describe the architecture behind Maven Central that makes all this functionality possible.

While you could do this via the Nexus UI, this wouldn’t help your automated nightly backup script. For this script to be easy to run, you’ll need to script Nexus via the REST interface. With the Nexus 1.6 release, we’ve released full documentation of the Nexus REST services available in every Nexus installation (REST doc). Continue reading for an example of how to interact with, authenticate, and manipulate a Nexus instance via the REST APIs.

The following script is written in Ruby and uses the REXML XML processor and the Net:HTTP library. Both of these libraries are available in the Ruby standard library. I tested these Ruby scripts using a standard Ruby 1.8.7 distribution. If you want to try these scripts out on your own, they will run in any standard Ruby interpreter without requiring any extra RubyGems.

Here’s the script, we’ll walk through some of the highlights after the source:

So, what’s going on in this script?

• Lines 7-15: Define some simple variables like host, port, the root URL of Nexus. We also define the status string “OUT_OF_SERVICE”. If we ran this script and then wanted to bring all hosted repositories back online, we would change this string to “IN_SERVICE”.
• Lines 17-21: Get a list of all repositories as an XML document and parse it. All Nexus REST services can return either XML or JSON, the default response if no “Content-type” header is present in the request is “application/xml”. This is a simple HTTP GET which does not require any authentication. You can find detailed documentation for the /service/local/repositories service here.
• Line 24: We’re only interested in manipulating hosted repositories. This line applies the XPath query “//repositories-item[repoType='hosted']” to the XML document and passed matching nodes to a large block that operates on each matching repository element.
• Lines 26-29: Just print out the id of each repository and the URL.
• Lines 31-34: Retrieve the status of a repository using the repository identifier. Again, this is a simple HTTP GET that does not require any authentication information. It returns an XML doc that describes the current status of the repository. Detailed documentation for the “/service/local/repositories/{repositoryId}/status” can be found here.
• LInes 36-41: Now here’s where it gets a bit fancy. The REST call to get the status was an HTTP GET, which returns an XML document. The REST call to set the status is an HTTP PUT, which accepts the same XML document. To set the repository status, we’re going to use the XML returned by a GET call to the status service and just change the “localStatus” element to either “IN_SERVICE” or “OUT_OF_SERVICE”. The statement on line 38, simply sets the value of this element in the XML document which was returned by the previous GET.
• Lines 43-50: Changing the status of a repository requires the proper privileges. For this call, we’ll need to supply the credentials set in the initial portion of the script. We will also need to make an HTTP PUT call and pass the XML document for repository status as the request body. While Nexus REST defaults to an XML response if no Content-Type is specified for simple HTTP GET calls, our HTTP PUT call to the repository status service must set the request’s “Content-Type” header to “application/xml” for status to be updated. The request body is set to the XML, which was modified on line 38, and the request is made on line 50.

That’s it. While this particular script isn’t much, it should provide a template for those of you who are interested in starting to automate common tasks via the Nexus REST API.

OPEN CHALLENGE: This this script is too verbose? Create your own Gist on GitHub.com and point to it in the comments. I’d love to see what other people could do with this sample. Expect a lot of follow-up posts to this, as we’re really interested in developing a large library of scripts for people to use.

Nexus OSS Core has more than 120 REST Resources

Nexus: A Core of REST Services

When we started the Nexus project, I called it a “blind” application.  A “blind” webapp is one with no UI technology whatsoever built-in.   All it publishes is a few static files, and a REST API.   All of the UI that you see in Nexus is Javascript.   Your browser executes Javascript which, in turn, interacts with a set of services.   The only presentation technology on the server side is a trivial Velocity template used to render the initial “shell” of the page.

The server-side application you call Nexus, doesn’t generate the UI, that is all Javascript.   It is a completely separate and standalone application, that runs in your browser! A self contained JavaScript application (powered by ExtJS), that communicates with Nexus server, using REST calls only.  This has one very important consequence: whatever Nexus functionality you are able to reach using the Nexus User Interface (and I do mean all), you can do with a plain REST client.     If you want to automate a task, you can use a tool like curl, or some Ruby script, or any HTTP capable client.   You can even start to implement your own custom interface if you would like a completely customized UI.

As I said, it was like this since day one. But, we always had one major issue that prevented wider acceptance of Nexus’ REST API: we had no up-to-date documentation for it.    It wasn’t a priority, we spent much of our time trying to integrate the only existing REST framework (restlet.org) with Plexus, and since we were moving so quickly to implement new features, we didn’t think it made sense to stop and document an ever-changing interface.  We did maintain a wiki page which described the REST services from the beginning of the project, it always provided stale information given our aggressive development pace.

REST Documentation: Using Enunciate

I’ve been looking at the Enunciate project for some time, and it appears to be the solution to our REST document issues.  Enunciate is a cool project hosted on Codehaus, with a bold promise:

Enunciate is designed to make your life easier. Enunciate deals with your deployment difficulties, documentation, and client-side code. By using Enunciate to publish your service interfaces, not only will you have all three Web service interfaces (SOAP, REST, and AMF), but you’ll also have full up-to-date documentation generated, strongly typed ActionScript code that you can use to invoke your remote services from your Flex code, and rich client-side libraries that can be used to access your Web services from a variety of platforms (including Mac/iPhone).

Our initial plans are to use Enunciate in a “lite” mode to generate documentation (and potentially client side code) for our current REST API. Later, we plan to integrate it more closely into new Nexus “Remoting API” implementation, and it will be the backbone of our REST services when we will implement version 2.0 of our API.

Here is just a taste of what is to come.  We’ve used Enunciate to generate documentation for our REST services, and the work is proceeding very quickly.   Here is a snapshot of the documentation we will be releasing soon.

Remaining Problems to Solve

When our REST API was concieved in Q4 of 2007 we didn’t have the rich array of options that we have today.   One major REST library was available, and that was Restlet.  JSR311 wasn’t around back then, so we were forced to code directly to the Restlet API.  Back in 2007, it was very cool to create Plexus components that were then automatically published as REST Resources in Nexus. But today, the same thing looks very cumbersome and is not what one would call a “state of the art”. Back then, we had these goals in mind:

• have complete REST API published on Nexus
• have a decoupled UI application
• provide XML/JSON serialized transport to REST Resource without any effort, with proper content negotiation (XML was kept as “Plan B”)
• make our REST Resources as Plexus components (to enjoy all the benefits of IoC, and later pluggability)

While we fulfilled much of these goals above, we did that at a high cost.  For a Nexus core developer (or Nexus plugin developer), it is very cumbersome to create a new REST Resource using our classes, compared to what REST libraries are available today. Current state-of-the-art REST libraries make our initial implementation seem somewhat naive (hence the need to move toward new libraries like Enunciate). Our 1st and biggest mistake was a very deep class hierarchy in our current API.  While all the “defaults” are properly configured, it is often difficult to create a non-trivial Resource in the current API if you are not intimately familiar with the system.

Also, in the initial design, we use XStream with a custom driver to produce JSON. Back then, it was a real pain to produce proper JSON out of Java.   Today, we have libraries like Jackson.   In the next interation of refactoring, you’ll see update the infrastructure used to produce our REST services to bring it up to speed with what is available today.

The Benefits of “enunciating” a REST API

At first, we will use Enunciate with Nexus during build time only, to generate HTML documentation of our REST API. This documentation will be available publicly, but will also be bundled with every Nexus instance!   Our work to modify our own REST infrastructure is going to require some “invasive” work, but I have to emphasize that Enunciate is not invasive if you use it on a modern JSR311 based REST applications. Enunciate is invasive for us only, since we are about to use it on a home grown, non-JSR311 REST application.

Enunciate gathers and builds your services model using your sources and all the metadata it finds (Java5 annotations, Javadoc).  To “enunciate” the Nexus REST API, we had to add JSR311 annotations to our existing REST Resources.  There was one more problem to solve: our method signatures are more “servlet like”, and they didn’t fit the model of a JSR311 REST Application. While a JSR311 application method signature completely describes the method expectations about payload, parameters, and response type, our methods – whether GET, PUT, POST or DELETE – have almost same signature: RestRequest, RestResponse, Variant. Hence, we had to add more metainformation describing what a method does: what it expects as input, if any, and what it sends out as response.

Luckily, it turned out that this was the only piece missing, and Enunciate was happy to generate documentation for us. Ryan added the new annotation that describes the “alternate REST Signature” of the method, and Enunciate got all the information it needed to generate documentation for us.

Another painful step yet to be completed is “freeing” our REST DTOs from Modello. Initially, we knew we wanted to version the DTOs used in REST. Hence, we used Modello to generate the DTOs. Later, problems with Modello and Restlet forced us to always use version 1.0 of our REST DTOs. Since Enunciate requires that DTOs have JAXB2 annotations, we decided to store our DTOs as code and ditch Modello in our builds.

All this looks like a lot of work, but it is actually pretty straightforward.   While this might seem dull to someone not developing Nexus, this work is going to bring real value once it is completed.    In addition to solving our documentation problems, Enunciate is going to open up a world of possibilities once we migrate our REST services over to the framework.

For more information about Enunciate, the Enunciate Wiki contains a nice example of how to use Enunciate in case like ours is, without JAX-RS. Example is shown here.  Examples of Enunciate generated documentation can be seen on it’s site. There is a great walk-through, with static examples.

Expect More Changes in the Next Few Months

As Nexus graduated from alphas, betas and now is at a 1.5.0 version, we realize we have to change gears to keep up. Enunciate is the 1st “newcomer” technology to Nexus stack.  There will be more to come.   Think of this as the Nexus 100,000 mile checkup.  We’re taking a look at the stack and taking this as an opportunity to upgrade where necessary.

One last note, this post is not about any deficiencies in Restlet.  Restlet is a great framework, and it served us well.  We – the Nexus Project team – are very thankful to Restlet.org project with all the help we got from them and for the great library they made.

The following scripts are in Groovy and they can be invoked from the command line using groovy.  For example, if you want to run the QuickSearch.groovy script, you would run “groovy QuickSearch.groovy activemq” to search for artfiacts that contain the string “activemq”.   I tested these Ruby scripts using the latest Groovy 1.5.7 distribution.  Go to the Groovy page, click on Download, and download the Cross-platform Installer, this will install Groovy on your machine and also tell you what you’ll need to add to your PATH.

The Nexus REST API

The UI of Nexus is written in ExtJS and it makes heavy use of AJAX callbacks to REST services.  In fact, if you load up Nexus in a tool like Firebug, an extension for Firefox which allows you to trace all network activity, you will see that almost every action in Nexus triggers a call to a REST service. For more details about the Nexus REST services and for a pointer to the documentation, read the previous blog post in this series which was focused on a similar set of Ruby scripts.

If you would like to download the sample scripts in this blog post, you can download this ZIP file which contains four Groovy scripts: http://books.sonatype.com/tutorial-files/nexus-rest-groovy.zip

Listing Repositories

The first script simply lists all of the repositories in a Nexus installation.  Here it is:

def xml = "http://repository.sonatype.org/service/local/repositories".toURL().text
 
def root = new XmlParser().parseText( xml )
root.data.'repositories-item'.each {
  println "${it.name.text()} (${it.id.text()})"
  println "\t${it.resourceURI.text()}\n"
}

This script sets the pattern for the scripts to follow.  We construct a URL in String, then we call toURL() and get the text attribute from the method call. In one line we fetch the response body. then we use the XmlParser in Groovy to print out the Repository name, id, and URL. This script could not be more straightforward. This post is simply a pointer to the service and a quick demonstration so I’m not going to dive into the meaning of every single element in the XML document returned by the repositories service.   If you are interested in see the full set of elements that are available, load up the results of this service in a web browser by click on this: http://repository.sonatype.org/service/local/repositories.

Performing a “Quick Search”

def xml = ("http://repository.sonatype.org/service/local/data_index?q=" + args[0]).toURL().text
 
def root = new XmlParser().parseText( xml )
root.data.artifact.each {
  println "${it.groupId.text()}:${it.artifactId.text()}:${it.version.text()}"
}

This script follows the pattern of the previous script to list all repositories with the exception that it reads an argument from the command line.   This script performs a quick search by hitting the data_index service and passing in the q parameter.   This script simply prints out the groupId:artifactId:version of all the artifacts located.

If you would like to see an example of the XML that this service produces click here: http://repository.sonatype.org/service/local/data_index?q=activemq. In the full results, you’ll see more information such as the number of search hits available, the resource URL for each artifact found, and the context (or repository) in which the artifact is available.

Searching by Class Name

def xml = ("http://repository.sonatype.org/service/local/data_index?cn=" + args[0]).toURL().text
 
def root = new XmlParser().parseText( xml )
root.data.artifact.each {
  println "${it.groupId.text()}:${it.artifactId.text()}:${it.version.text()}"
}

This script is almost exactly the same as the prior script that performed a quick search.  The difference in this script is that instead of passing the q parameter, this script passes the cn parameter.   Passing the cn parameter causes Nexus to search for artifacts which contain classes that match the given value.   The results are going to look the same as the quick search query script.     To see the XML yourself, search for all artifacts which contain a class named HibernateDaoSupport: http://repository.sonatype.org/service/local/data_index?cn=HibernateDaoSupport.

Performing a GAV Search

def xml = "http://repository.sonatype.org/service/local/data_index?g=${args[0]}&a=${args[1]}&v=${args[2]}".toURL().text
 
def root = new XmlParser().parseText( xml )
root.data.artifact.each {
  println "${it.groupId.text()}:${it.artifactId.text()}:${it.version.text()}"
}

This final script takes three command line arguments: groupId, artifactId, version, and it performs a GAV (groupId, artifactId, version) coordinate search over the repository.

The following scripts are in Ruby and they use the REXML XML processor and the Net:HTTP library.  Both of these libraries are available in the Ruby standard library.   I tested these Ruby scripts using the latest JRuby release.  If you want to try these scripts out on your own, they will run in any standard Ruby interpreter without requiring any extra RubyGems.

The Nexus REST API

The UI of Nexus is written in ExtJS and it makes heavy use of AJAX callbacks to REST services.  In fact, if you load up Nexus in a tool like Firebug, an extension for Firefox which allows you to trace all network activity, you will see that almost every action in Nexus triggers a call to a REST service.  We’re going to focus on just two services in this post: list repositories and searching for artifacts, but you should know that there are hundreds of things you can do to Nexus via the various REST interfaces that are available.  In fact, you could create a custom UI tailored to your own needs that interfaces with Nexus via this REST backend.    In addition to the REST services in the core Nexus installation, you can extend Nexus and add in your own REST services via a Nexus plugin.  The Nexus UI is completely decoupled from the set of services Nexus provides, and the services which Nexus provides can be extended.

The Nexus REST Services are documented here.  In this post, we’re going to be writing scripts that hit the public instance of Sonatype Nexus at http://repository.sonatype.org, our REST URLs are going to look something like this: http://repository.sonatype.org/service/local/data_index?q=maven.  If you were attempting to hit the services on your local Nexus installation, your REST URL would look more like this: http://localhost:8081/nexus/service/local/data_index?cn=HibernateDaoSupport.   These REST URLs follow the pattern:  “http://{host}:{port}/{context}/service/{instance}/{service}?{query_params}” where the {instance} will almost always be “local”.

If you would like to download the sample scripts in this blog post, you can download this ZIP file which contains four Ruby scripts: http://books.sonatype.com/tutorial-files/nexus-rest-ruby.zip

Listing Repositories

The first script simply lists all of the repositories in a Nexus installation.  Here it is:

require 'net/http'
require 'rexml/document'
include REXML
 
url = 'http://repository.sonatype.org/service/local/repositories'
resp = Net::HTTP.get_response( URI.parse( url ) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//repositories-item" ) do |r|
  puts "#{r.elements["name"].text}(#{r.elements["id"].text})"
  puts "\t" + r.elements["resourceURI"].text + "\n\n"
end

This script sets the pattern for the scripts to follow.  We construct a URL in the script (a more general script would read this from a config file), then we make the request using Net::HTTP and we parse the results with REXML by passing it to the Document constructor.  At this point, I use XPath to select all of the repositories and I print out the name, id, and the URL of the repository as served by Nexus.

This post is simply a pointer to the service and a quick demonstration so I’m not going to dive into the meaning of every single element in the XML document returned by the repositories service.   If you are interested in see the full set of elements that are available, load up the results of this service in a web browser by click on this: http://repository.sonatype.org/service/local/repositories.

Performing a “Quick Search”

require 'net/http'
require 'rexml/document'
include REXML
 
url = 'http://repository.sonatype.org/service/local/data_index?q='
resp = Net::HTTP.get_response( URI.parse( url + ARGV[0]) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//data/artifact" ) do |r|
  puts "#{r.elements["groupId"].text}:#{r.elements["artifactId"].text}:#{r.elem\
ents["version"].text}"
end

This script follows the pattern of the previous script to list all repositories with the exception that it reads an argument from the command line.   This script performs a quick search by hitting the data_index service and passing in the q parameter.   This script simply prints out the groupId:artifactId:version of all the artifacts located.

If you would like to see an example of the XML that this service produces click here: http://repository.sonatype.org/service/local/data_index?q=activemq. In the full results, you’ll see more information such as the number of search hits available, the resource URL for each artifact found, and the context (or repository) in which the artifact is available.

Searching by Class Name

require 'net/http'
require 'rexml/document'
include REXML
 
url = 'http://repository.sonatype.org/service/local/data_index?cn='
resp = Net::HTTP.get_response( URI.parse( url + ARGV[0]) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//data/artifact" ) do |r|
  puts "#{r.elements["groupId"].text}:#{r.elements["artifactId"].text}:#{r.elem\
ents["version"].text}"
end

This script is almost exactly the same as the prior script that performed a quick search.  The difference in this script is that instead of passing the q parameter, this script passes the cn parameter.   Passing the cn parameter causes Nexus to search for artifacts which contain classes that match the given value.   The results are going to look the same as the quick search query script.     To see the XML yourself, search for all artifacts which contain a class named HibernateDaoSupport: http://repository.sonatype.org/service/local/data_index?cn=HibernateDaoSupport.

Performing a GAV Search

require 'net/http'
require 'rexml/document'
include REXML
 
url = "http://repository.sonatype.org/service/local/data_index?g=#{ARGV[0]}&a=#\
{ARGV[1]}&v=#{ARGV[2]}"
resp = Net::HTTP.get_response( URI.parse( url ) )
 
doc = REXML::Document.new( resp.body )
 
XPath.each( doc, "//data/artifact" ) do |r|
  puts "#{r.elements["groupId"].text}:#{r.elements["artifactId"].text}:#{r.elem\
ents["version"].text}"
end

This final script takes three command line arguments: groupId, artifactId, version, and it performs a GAV (groupId, artifactId, version) coordinate search over the repository.