If your build relies on Maven Central, and you’re not using a repository manager, you are wasting valuable time and bandwidth by continually downloading the same artifacts, over and over again.
But the good news is, with a Nexus repository manager you don’t need to worry about this ever again.
This video gives a quick overview of the benefits of using a Nexus to proxy Maven Central. It also goes over how to store third party libraries, and host artifacts.
Watch the video below:
In my last post, I explained the REST-style API that underlies Maven Central’s browser-based search UI. That API essentially comes “for free” with the main components on which Maven Central Search is built:
In this post, I will highlight those components and describe how they were used to implement Maven Central Search.
When we started the project, we looked at a couple of options for implementing search, including Solr and the existing Nexus search capability built directly on top of Apache Lucene. The Nexus approach initially seemed compelling as we clearly have significant experience with it and Nexus search even provides a REST API for full-text search that we could have leveraged. So, why did we end up choosing Solr when we could have simply re-used the search functionality in Nexus or even crafted a web UI backed by an instance of Nexus running on top of Central? Two reasons:
Since its release in January, the Maven Central website (http://search.maven.org) has provided Apache Maven users with:
In the intervening months, Sonatype has focused its efforts on improving the usability of the Maven Central user interface in the hopes of making it the first place users look when trying to find an artifact. Recently, users who have reaped the benefits of using the Maven Central website have asked about interacting programmatically with the search functionality.
If you pay attention to your web browser’s address bar when conducting searches on Maven Central, you can already see that a REST-style API exists. For example, searching for “guice” from the main search box results in the following URL being generated (the following URL’s are NOT URL-encoded for the sake of readability):
Translating the search request into English, that URL requests a basic search for any artifact (irrespective of version) containing the word “guice” in either the groupId or artifactId, returning only the first page of results. Each row of the results shows the latest version of the artifact and the date the artifact was last updated as well as any classifiers associated with the artifact.
You can build up the complete library of search requests simply by paying attention to your web browser’s address field as you use the Maven Central website. For the sake of convenience, we’ve collected all the URLs that make up Maven Central’s search API in a document available here.
Sadly, these URL’s are still only useful when requesting them via web browser. They are links that can be bookmarked or e-mailed, but they do NOT work when using a non-browser agent like wget or curl. The Maven Central user interface is essentially a browser-based application that uses Javascript to make asynchronous requests to yet another set of URL’s. Once you make a request that looks like the URL above, the browser fires off the actual request to another Maven Central URL responsible for conducting the search and returning results that are formatted by the browser.
The sample request above, when converted to an actual Maven Central search request, looks like this:
The actual text of your query goes in the appropriately named “q” parameter, the “rows” parameter restricts the results to a smaller number than the full result set, and the “wt” parameter can be either “xml” or “json,” depending on how your application prefers to handle results.
Some useful examples appear below. Again, please refer to the API Guide for a complete listing:
In an upcoming post, I’ll describe the architecture behind Maven Central that makes all this functionality possible.
Spoiler Alert! This post contains information about a change to Maven Central’s IP addresses. If your network has firewall rules in place that need specific IPs, be sure to read this post.
We’re working hard and investing continued effort into making sure that Central is as available as possible. As Maven Central supports a world of developers, even a few minutes of downtime is completely unacceptable to us. In line with our previous efforts to make Maven Central as bullet-proof and available as possible, we are planning to make the US repository even more fault tolerant using a tool called Pacemaker. Once we’ve had time to evaluate the impact of the changes described in this post we will deploy similar measure for our European Union and Asia/Pacific mirrors.
As a follow up to our previous enhancements to central, we are planning to make the Maven Central US repository even more fault tolerant.
The US repository runs on two virtual machines (VMs) in a VMWare cluster with 4 physical nodes configured to use the High Availability support in VMWare. Despite having multiple levels of fault-tolerance, recovery from a misconfiguration or other catastrophic failure still requires a DNS update to a standby IP to restore Maven Central. This DNS change requires time: time to make the change and then an often unpredictable time for DNS changes to propagate over the entire Internet.
This is unacceptable to us. Millions of developers depend on Maven Central, we’ve invested in redundant virtual machines running on redundant physical hardware. If there is an unforeseen event, the problem should be addressed in a few seconds.
To achieve immediate failover in the event of failure we will be using a tool called Pacemaker to manage Maven Central’s floating IP cluster. Pacemaker monitors the repository IP address, Nginx process status, and sample content from Maven Central. If Pacemaker identifies a failure in any one of these components it will immediately failover to the backup machine. In my testing, this takes about 3-5 seconds to occur.
In a previous post I discussed the systems we have in place and how the IPs are configured:
We are aware that some users have firewall rules that are locked to the external service IP. Because of this, we strive to maintain a consistent IP for each system, however the primary mechanism for accessing the repository is by DNS for most users. At times, our failover escalation or maintenance procedures may require us to redirect the DNS for one system to another. For this reason, if you have firewall rules in place that need specific IPs, please allow this list so that you won’t be affected by any temporary transitions:
Do you develop Java applications using open source software artifacts from Maven Central? If so, and you aren’t yet using a local repository manager, you need to attend this webinar. In this session you’ll learn how an enterprise repository manager can reduce development time, improve quality, enable greater internal collaboration, and reduce risk. Register now to learn how repository management can help your organization.
