But the good news is, with a Nexus repository manager you don’t need to worry about this ever again.

This video gives a quick overview of the benefits of using a Nexus to proxy Maven Central. It also goes over how to store third party libraries, and host artifacts.

Watch the video below:

• Apache Solr, the popular, blazing fast open source enterprise search platform from the Apache Lucene project — http://lucene.apache.org/solr
• ajax-solr — http://evolvingweb.github.com/ajax-solr

In this post, I will highlight those components and describe how they were used to implement Maven Central Search.

When we started the project, we looked at a couple of options for implementing search, including Solr and the existing Nexus search capability built directly on top of Apache Lucene. The Nexus approach initially seemed compelling as we clearly have significant experience with it and Nexus search even provides a REST API for full-text search that we could have leveraged. So, why did we end up choosing Solr when we could have simply re-used the search functionality in Nexus or even crafted a web UI backed by an instance of Nexus running on top of Central? Two reasons:

• Flexibility — We discovered early on during the design phase of Central Search that we needed changes to the schemas, fields, and even field contents in the Lucene indices being used by Nexus. Making those changes to the schemas would have required other changes within the Nexus codebase. With Solr, we could simply point our Solr installation against an existing index or even have Solr build a new index from scratch by adding documents through Solr’s REST API. We could rapidly prototype schema changes (often in 1-2 lines of xml and not even requiring us to restart Solr) and see our updated search results almost immediately.
• Scalability — Solr bills itself as an “enterprise search platform.” One of the enterprise features that attracted us to Solr was its built-in support for replication. As query load increases in the future, we can simply balance that load across hardware serving multiple copies of the same data. Solr’s support for multiple indexes also leaves us a path open for sharding our index data, once it becomes so large as to be difficult to serve out of a single index on a single server.

Once we made the decision to use Solr, we quickly discovered that practically all the search functionality we needed came “out-of-the-box” with Solr’s REST API. In fact, the entire second half of the Maven Central API Guide is simply a set of URLs that are proxied to our running Solr instance. We proxy the requests so that we can do some filtering and transformation of inbound requests in order to prevent a malformed or malicious request from taking down our server.

Next, we turned our attention to the browser-based user interface to call that API. Solr provides an administrative interface out of the box that includes search functionality. However, administrative interfaces tend to be utilitarian. We wanted our interface to be clean, but also to be a little more interactive, not just at launch, but as we added new features. So, we started researching AJAX- or javascript-based UIs that could sit on top of Solr, and we found ajax-solr.

The ajax-solr website is a great resource for understanding the architecture of ajax-solr and provides an excellent tutorial for building your first Solr-powered AJAX-based website. Our developers took hold of that tutorial and very quickly fashioned a prototype version of Central Search. During development, two major benefits of ajax-solr stood out:

• MVC Pattern – The Model–view–controller (MVC) software architecture, often used for web applications, isolates “domain logic” from the user interface, permitting independent development, testing and maintenance of each. Ajax-solr applied the MVC pattern to Solr result sets within the browser which makes for a clean and easily extensible way of working with Solr result sets (the model) and ajax-solr widgets (the views). It also helped that MVC is an easy to understand pattern.
• jQuery implementation — According to the ajax-solr website, ajax-solr is designed to be Javascript framework-agnostic. Any framework that can send AJAX requests to Solr can work with ajax-solr. However, the ajax-solr tutorial and stock widgets are all implemented with jQuery. Even though we had to learn about jQuery along the way, we discovered that jQuery was fairly easy to pick up and, having done so, made extending ajax-solr to meet our requirements very easy. jQuery also gave us a decent amount of cross-browser compatibility. That’s not to say that we don’t struggle with a little bit of browser-specific tweaking here and there, but, at least anecdotally, it saved us from many hours of wondering why our pages rendered differently (or didn’t render at all) in different browsers. Finally, using jQuery opened us up to a wealth of UI plugins (http://plugins.jquery.com), which we used in adding various bits of interactivity to Maven Central.

In summary, we used several standard open source Java components to build the Maven Central search and along the way our team added several new tools to our bag of tricks. We now have a very strong foundation for continuing improvement of Maven Central. We hope you have found the new features useful and we look forward to hearing your feedback at Get Satisfaction (http://getsatisfaction.com/sonatype).

• Search functionality that allows one to quickly track down artifacts and their dependency details when trying to resolve build problems.
• Browse functionality that aids in discovery of new artifacts to use in projects.

In the intervening months, Sonatype has focused its efforts on improving the usability of the Maven Central user interface in the hopes of making it the first place users look when trying to find an artifact.  Recently, users who have reaped the benefits of using the Maven Central website have asked about interacting programmatically with the search functionality.

If you pay attention to your web browser’s address bar when conducting searches on Maven Central, you can already see that a REST-style API exists.  For example, searching for “guice” from the main search box results in the following URL being generated (the following URL’s are NOT URL-encoded for the sake of readability):

• http://search.maven.org/#search|ga|1|guice

Translating the search request into English, that URL requests a basic search for any artifact (irrespective of version) containing the word “guice” in either the groupId or artifactId, returning only the first page of results.  Each row of the results shows the latest version of the artifact and the date the artifact was last updated as well as any classifiers associated with the artifact.

You can build up the complete library of search requests simply by paying attention to your web browser’s address field as you use the Maven Central website.  For the sake of convenience, we’ve collected all the URLs that make up Maven Central’s search API in a document available here.

Sadly, these URL’s are still only useful when requesting them via web browser.  They are links that can be bookmarked or e-mailed, but they do NOT work when using a non-browser agent like wget or curl.  The Maven Central user interface is essentially a browser-based application that uses Javascript to make asynchronous requests to yet another set of URL’s.  Once you make a request that looks like the URL above, the browser fires off the actual request to another Maven Central URL responsible for conducting the search and returning results that are formatted by the browser.

The sample request above, when converted to an actual Maven Central search request, looks like this:

• http://search.maven.org/solrsearch/select?q=guice&rows=20&wt=json

The actual text of your query goes in the appropriately named “q” parameter, the “rows” parameter restricts the results to a smaller number than the full result set, and the “wt” parameter can be either “xml” or “json,” depending on how your application prefers to handle results.

Some useful examples appear below.  Again, please refer to the API Guide for a complete listing:

• Fully-qualified classname search – http://search.maven.org/solrsearch/select?q=fc:”org.specs.runner.JUnit”&rows=20&wt=json
• GroupId and artifactId search that returns all available artifact versions – http://search.maven.org/solrsearch/select?q=g:”org.apache.maven.indexer”+AND+a:”maven-indexer”&rows=20&core=gav
• SHA1 search (you would need to pre-calculate the SHA1 before sending the request to Maven Central) – http://search.maven.org/solrsearch/select?q=1:”2d3c16092663da9041b171b8d3627cbafa8f0cb1″&rows=20&wt=json

In an upcoming post, I’ll describe the architecture behind Maven Central that makes all this functionality possible.

We’re working hard and investing continued effort into making sure that Central is as available as possible. As Maven Central supports a world of developers, even a few minutes of downtime is completely unacceptable to us. In line with our previous efforts to make Maven Central as bullet-proof and available as possible, we are planning to make the US repository even more fault tolerant using a tool called Pacemaker. Once we’ve had time to evaluate the impact of the changes described in this post we will deploy similar measure for our European Union and Asia/Pacific mirrors.

As a follow up to our previous enhancements to central, we are planning to make the Maven Central US repository even more fault tolerant.

The US repository runs on two virtual machines (VMs) in a VMWare cluster with 4 physical nodes configured to use the High Availability support in VMWare. Despite having multiple levels of fault-tolerance, recovery from a misconfiguration or other catastrophic failure still requires a DNS update to a standby IP to restore Maven Central. This DNS change requires time: time to make the change and then an often unpredictable time for DNS changes to propagate over the entire Internet.

This is unacceptable to us. Millions of developers depend on Maven Central, we’ve invested in redundant virtual machines running on redundant physical hardware. If there is an unforeseen event, the problem should be addressed in a few seconds.

To achieve immediate failover in the event of failure we will be using a tool called Pacemaker to manage Maven Central’s floating IP cluster. Pacemaker monitors the repository IP address, Nginx process status, and sample content from Maven Central. If Pacemaker identifies a failure in any one of these components it will immediately failover to the backup machine. In my testing, this takes about 3-5 seconds to occur.

In a previous post I discussed the systems we have in place and how the IPs are configured:

We are aware that some users have firewall rules that are locked to the external service IP. Because of this, we strive to maintain a consistent IP for each system, however the primary mechanism for accessing the repository is by DNS for most users. At times, our failover escalation or maintenance procedures may require us to redirect the DNS for one system to another. For this reason, if you have firewall rules in place that need specific IPs, please allow this list so that you won’t be affected by any temporary transitions:

• 207.223.240.88 : US primary
• 207.223.240.92 : US staging / standby
• 89.167.251.252: UK Primary
• 89.167.251.253: UK standby

Since we declared .88 to be the primary IP in the US, I intend to use that as the new clustered ip. The migration plan is as follows:

1. At approximately 9PM CST Monday, May 9th, we will update the DNS entries for repo1/repo2.maven.org to point at 207.223.240.92 (the standby server).
2. Tuesday morning, the 207.223.240.88 ip will be added to the cluster. We’ll test the ip and failover using this address.
3. Once the testing is successful, we will revert the DNS entry back to 207.223.240.88.

Both DNS shifts should be completely transparent since both servers will be actively capable of serving artifacts, so regardless of which IP resolves for you during that transition time, the repo will respond. The Expiration TTL on the DNS data is currently 5 minutes, so normally everyone would follow over during that time, but we are going to allow 12 hours (just in case).

We will be rolling out this same setup to our UK servers in a few weeks.

I can’t thank Contegix enough for their ongoing support. They conceived of and implemented this virtual IP solution. They are playing a big role in making Maven Central more reliable.

We’ve added a new webinar to the Sonatype series: Enterprise Repository Management.

Do you develop Java applications using open source software artifacts from Maven Central? If so, and you aren’t yet using a local repository manager, you need to attend this webinar. In this session you’ll learn how an enterprise repository manager can reduce development time, improve quality, enable greater internal collaboration, and reduce risk. Register now to learn how repository management can help your organization.

• Date: Tuesday, April 19, 2011
• Time: 1:00PM EDT (GMT – 04:00)
• Duration: 30 minutes
• Presenter: Brian Fox, Sonatype Vice President of Engineering
• To register, please click here

Virtual Servers: We now have 6 nodes running in a private cloud cluster and both Central machines in the US are fully converted to virtual machines. This will allow the two systems to be completely isolated from hardware failure and provides greater flexibility for load balancing and performance tuning.

Internet Redundancy: Maven Central is now connected to Contegix’s fully managed network which provides multiple routes to the Internet. While the existing connection had been extremely stable in the past, we believe that using the managed and redundant network is the best choice for providing a reliable service to our users. The conversion to the new network is already completed and you may notice that Maven Central now has a new IP address.

We are aware that some users have firewall rules that are locked to the external service IP. Because of this, we strive to maintain a consistent IP for each system, however the primary mechanism for accessing the repository is by DNS for most users. At times, our failover escalation or maintenance procedures may require us to redirect the DNS for one system to another. For this reason, if you have firewall rules in place that need specific IPs, please allow this list so that you won’t be affected by any temporary transitions:

• 207.223.240.88 : US primary
• 207.223.240.92 : US staging / standby
• 89.167.251.252: UK Primary
• 89.167.251.253: UK standby

In this issue you’ll find information on Hudson and the Hudson survey – we want  you to tell us how you use Hudson, and what you need from Hudson moving forward. Fill out the survey for a chance to win an Apple iPad 3G!

Also in this issue: A real-time search service for Maven Central; Sonatype donates Maven indexer to the ASF; getting your developers productive in hours not days; and upcoming Sonatype’s Maven Training classes.

Subscribe to our newsletter to keep up to date with the latest information on our products and services.

And for updates on Sonatype events, software and projects the minute they happen, follow us on Twitter!

What makes this new service  different from other sites that index and search the central repository?  The new search service is not just a place for quickly and easily researching artifacts on which your project depends, but it is also the most up-to-date source of information on those artifacts other than browsing repo1.maven.org directly.

Searching Maven Central is simple. You can search for artifacts hosted in Maven Central by typing in your search terms separated by spaces. Maven Central Search will search for your terms in either the groupId or the artifactId fields. Your search results will include the latest version for each found artifact, a link to see all versions for that particular artifact, and all available downloads for that artifact.

Before you use the new service, note that the site relies heavily on Javascript. Please make sure that Javascript is enabled in your browser. Maven Central has been tested on IE7+, FF3+, Safari 3+, and Chrome 4+. The Jar Check feature downloads and executes a Java applet, so you will need to have Java 1.5+ installed and Java enabled in your browser to use this feature.

The new service also offers advanced search functionality, including:

• Search by Maven coordinates
• Search by class name
• Search by SHA-1 checksum (which Maven Central can calculate for any jar file you have on your computer using its Jar Check functionality).

Please provide any feedback you have using the GetSatisfaction feedback tab that is always available on the right-hand border of the site. You can share ideas, ask questions, or report problems with the site. This is the first of many new services we plan for Maven Central, and we very much value your opinions, and we’re interested in getting your feedback.

Earlier this year, we announced the availability of official repositories in the UK to improve performance for the users in Europe. Today we are making the artifact download statistics available to the projects whose artifacts are served by Central. This has been one of the most frequently requested features by project teams. Since the raw Central logs are larger than seven gigabytes every day, processing this data is no small undertaking.

The statistics are available to all projects hosted using Nexus at http://oss.sonatype.org, http://repository.apache.org and http://nexus.codehaus.org. These three avenues represent the majority of projects actively contributing artifacts. Nexus’ security mechanism already in place on these instances provides a mapping of repository path to project which allowed us to easily roll up the counts for each team. Read more to find out how to access your project’s statistics.

Accessing Your Project’s Statistics

Current OSSRH, Apache and Codehaus users don’t need to do anything to gain access to these statistics. If you have deployer permissions for your project, you should already be able to see the Central Statistics link in the Views/Repositories section in the left-hand menu area. NOTE: In some cases, users with early access to the plugin have reported needing to click the web browser’s Refresh button before seeing the link.

Clicking on the Central Statistics link will open a tab showing you:

• a line chart depicting the volume of raw downloads of your artifacts from Central over the past 12 months
• a pie chart breaking down the last month’s worth of downloads of your project by artifactId
• a sortable grid containing the counts that generated the pie chart

If you only have access to a single project with a single groupId, the page will be fully populated upon opening the tab. If you have access to multiple projects or a project encompassing multiple groupIds, you will need to select a project and groupId on which to report. You can also use artifactId and version combo boxes to further narrow down the reporting scope for the timeline, the pie chart, and the grid.

The pie chart and grid can be further constrained by a date range combo box that defaults to the last month, but can be used to report on the last 3, 6, 9, or even 12 months or to select a single month for reporting.

Finally, the plugin offers two types of reports. Downloads is the default and simply reports on the number of successful downloads of your artifacts. Unique Ips is the second report type and gives an idea of how many unique users are downloading your artifacts. It is also possible to export the raw data to CSV so that you can analyze it using your own favorite tool.

Frequency of Update

Currently we generate the statistics to a granularity of a single month, so you can expect to see the new results appear within a few days of each new month.

Send us Suggestions and Feedback

This is just the first release of this plugin, and we invite your feedback and recommendations, not just on how to improve the look and feel of the plugin but also for new reports you’d like to see in the future. You are welcome to create issues in the following JIRA project:

https://issues.sonatype.org/browse/CENTRALSRV

When creating issues, please assign them to the “Central Statistics Plugin” component.

While Maven Central contains a huge number of libraries, there are still some pretty important things missing. Anything covered by a non-open-source license, like the Oracle JDBC drivers or anything proprietary, isn’t going to be made available from the Maven Central repository. Very often there are going to be commercial libraries that you will need to download and install in a third party repository. While this works, it requires some manual work to upload the artifact.

Vendors and Repository Management

When your vendors embrace repository management, there’s no reason why you would have to manually install any third party JARs in your corporate repository. Instead, vendors who need to provide binaries to customers would provide them via authenticated public repositories. These vendor-run repositories, protected by authentication credentials, would allow customers to synchronize local, corporate repository managers with commercial software vendors. This would allow vendors to capitalize on a new, “always connected” model for software delivery.

Now, the situation for the customer is simple. Once they set up an authenticated proxy repository to connect to a vendor’s repository, the vendor can deliver new updates and software directly to a customer’s repository.

The situation for the vendor is equally straightforward.  Instead of having to send the customer a packaged set of binaries, they simply tell the customer to update a version number in a dependency. The build will then interrogate the customer’s repository and then automatically download the repository from the vendor.

Both the customer and the vendor have fewer moving parts to worry about, and the task of delivering software becomes very simple.

The vendor receives another benefit from this setup. They are now able to keep track of which customers had requested which artifacts. The vendor can control which artifacts or features of a product suite a customer has access to, and they could use the repository manager as a way to enforce licensing or provide metered, on-demand access to software components.

Partners and Repository Management

In today’s increasingly connected world, it is very common for one company to provide an API for partners. Sites like Twitter and Google provide APIs for the general public, and other companies provide partner-specific APIs for collaboration. If you provide an API for a partner company, you can expose libraries and interfaces using a public-facing, authenticated repository.

Suppose that you work at a financial institution that needs to partner with another financial institution. The two institutions have agreed to collaborate with one another using a simple set of REST services. While these REST services are very well documented, you also want to make sure that your partner is invoking these REST services according to a set of standards. To make it easier to consume these services, you’ve created a simple client library for your partner.

To make it even easier for your partners to use this library, you’ve published versions of this library to a public-facing, authenticated repository. Using the security features of a modern repository manager, you can create partner-specific library versions and limit access to just the intended recipients. You can also maintain detailed audits of when a particular API was delivered to a particular partner.

As more and more organizations start to adopt the best practice of running a repository manager, these same organizations will be creating ad-hoc, private relationships to emulate the ease of Maven Central for a corporate environment. Partners, vendors, and consortia of companies will create repositories to facilitate code sharing and cooperation.