We’re making SSL connectivity to Central available to anyone that downloads open source components regardless of the repository manager. Given the tremendous growth of Central, and the fact that modern applications are largely built from OSS components, this capability is likely to be leveraged by many organizations. SSL has become the standard mechanism for protecting web traffic – across the spectrum of Ecommerce, banking, health care, and so on. Providing SSL support for Central means that your components are no longer susceptible to man-in-the-middle attacks that could compromise the component. SSL also eliminates the potential for a hacker to gain visibility into your organization by tracking the components that you download for your development initiatives.

As of Nexus Pro 2.2 (available now), SSL is now the default connectivity option for Nexus Pro users. Because we take security of the ecosystem seriously, we aren’t stopping there, we’re making SSL connectivity to Central available to you even if you aren’t using Nexus Pro.

In order to ensure the highest level of performance for those who count on SSL, we are securing the service with a token. You can get a token for your organization simply by providing a $10 donation that will be donated to open source causes. For the first 60 days all donations will go to the Apache Software Foundation. After that, the donations will go to other open source foundations such as Eclipse. Sonatype will provide a donation on behalf of Nexus Pro customers since we’ve included SSL access to all Pro customers automatically.

If you happen to be using Nexus OSS (any version), support for the SSL token is included already. I’ve already reached out to the Artifactory and Archiva teams and they are working on the changes necessary to enable SSL to Central – we’ll let you know when that support is enabled. If you’re not using a repository manager at all, what are you waiting for?

If you are an existing Nexus Pro customer, you can download the latest release from the support page.

If you would like to make a donation to the open source community and get SSL access, you may do so here.

Only a few years ago, Central was a single Dell server running in a Contegix datacenter in St. Louis (you can see it here). From 2007 to 2011 the server was used 12 billion times by 14.3 million unique IP addresses, and since then traffic from a world of developers has only continued to increase. Over the years we’ve invested in both capacity and stability improvements, but today I’m announcing what I consider to be the biggest improvement to date.

Today we announced an agreement with EdgeCast Networks. EdgeCast Networks is a CDN with global reach that we are going to use to both accelerate the delivery and increase the availability of Central. Every Java developer who uses Maven, Gradle, or Ivy (possibly every Java developer in the world) will see immediate improvements in the speed of Central. Index downloads and artifact downloads will be served from one of EdgeCast’s 21 points of presence distributed over four continents. Your builds are going to run faster because of Sonatype’s agreement with EdgeCast.

In addition to this, Sonatype Nexus Pro customers will now have access to end-to-end Secure Socket Layer (SSL) encryption, bringing a greater level of security to their software development processes.

Two weeks ago we talked about how many of the projects hosted in Scala Tools are moving over to publish directly to Central. That process is ongoing. In this post, I want to start something new. At Sonatype we touch a lot of different technologies and communities, and I want to make sure that we’re doing all we can to help put a spotlight on some of the communities that we’re watching. Whether it is a .NET-focused open source foundation like Outercurve, a customer that contributes back to Nexus OSS or, in this post, the Scala community, I think that Sonatype can at least help introduce some of these interesting technologies to a larger audience.

So, what is Scala?

The JVM saw an explosion of “alternative” languages over the past several years. From clojure to jruby to scala, each language tends to have a specific “sweet-spot” it solves more efficiently than other languages. If you are interested in rapid web development, JRuby has become a solid option alongside Rails. Scala’s “sweet spot” is providing a functional programming language that provides an elegant way to think about concurrency and parallelism while integrating concepts of OO programming. Where languages like Ruby and Python have become logical choices for front-end web development, Scala has increasingly become the default choice for the kinds of massively parallel systems that make sites like Twitter, LinkedIn, and Google run. Scala’s applications range from large-scale scientific simulations, to high-speed, algorithmic trading applications, to systems like ad-exchanges. Here are some interesting people, projects, and companies that use Scala:

• Martin Odersky, creator of Scala, co-founder of Typesafe, discussed trends in parallelism that drive Scala at O’Reilly’s OSCON ‘11. This is a great if you are looking for some motivation behind functional programming.
• Everytime you Tweet, you Scala – Also from O’Reilly’s OSCON ‘11 Raffi Krikorian from Twitter discussed Twitter’s migration from a Ruby on Rails architecture, to an architecture that relies on Rails as a Front-end, Scala as a foundation for web-scale storage and retrieval.
• Scala at Square – Square’s a mobile payment service poised to garner some big attention this year as several major US presidential campaigns have adopted it as a mobile payment platform. Well, if you Square, you are also making use of Scala. Bob Lee, the CTO of Square mentioned Scala in his OSCON talk about a coming Java renaissance.
• Use Quora? You also use Scala. – According to this answer on Quora, Quora engineer Kah Seng Tay suggests that Quora is moving some systems to Scala because it is “fast, concise, type-safe, object-oriented yet functional, easy parallelism, extensible and built on the JVM”.
• Foursquare, Sony, Xerox, Siemens, The Guardian, Novell, LinkedIn, and more examples at scala-lang.org.

Note: What about Lift? Or Scala-tools? Why didn’t I mention X, Y, or Z? We’re not done featuring interesting Scala projects, this is the just the first installment.

When you Scala do you use Maven or SBT?

Now, you might assume that Sonatype has a definite opinion in this space. After all, didn’t we write a series of Maven books?

Yes, we did, but we’re also an increasingly “tool agnostic” company. While you’ll see a good number of our examples focus on Maven, we understand that the world uses more than one build tool. In Scalaland, it’s a choice between the maven-scala-plugin and SBT. Both seem to present a solid option with most pure Scalaist seeming to prefer SBT over Maven. There was a discussion at scala-lang.org about the relative merits of SBT vs. Maven in December of 2011, and it looks like there are a equal number of people suggesting SBT vs Maven.

Whatever you use, your build tool is still going to download from Central, and it would benefit from using a local repository manager like Nexus. In the next installment of our Scala coverage we’ll talk about how to use Scala with a repository manager.

One last thing…. The other nice thing about Scala is that Scala skills are in high demand. Every time I speak with someone running an organization that uses Scala, they are always looking for people familiar with Scala. Most companies that use Scala, end up having to just hire smart developers and train them on Scala. It’s clear that knowing Scala well will put you in an advantage for a particular type of company that is engaged in “big data” work. For example, last I heard Square was looking for Scala talent.

Two weeks ago, all Scala projects required a little bit of extra configuration to point to a custom repository for Scala artifacts hosted at scala-tools.org. Today, Scala artifacts are now available directly from Central. The contents of scala-tools.org are now integrated into the Sonatype OSS repository hosting service, and other projects have started to publish artifacts Central.

The Scala community will see immediate benefits from this move. There are no more extra repositories to configure. It just got incrementally easier to use Scala. If you are new to Scala, you don’t need to reconfigure your repository manager to proxy another remote repository. The community will benefit from Sonatype’s continued investment in the infrastructure that runs Central: a cluster of machines in both the US and the EU continuously monitored by a dynamic DNS server that can reroute traffic instantly in the event of downtime.

How did this happen? Joshua Suereth, David Bernard, and Derek Chen-Becker provided the bulk of the administrative work, and they recently decided to decommision this server and transition repository hosting to the free Sonatype OSS service. Here’s the announcement by Joshua Suereth to the user forums on scala-lang.org on January 17th:

Scala-tools.org is going down and not accepting any new OSS projects. For those of us who wish to continue release software, I recommend migrating over to Sonatype. They put a few (good practice) limitations on contributions, but scala-tools.org would have done the same before long anyway. The benefit of Sonatype hosting is that your projects will make it onto the maven-central repository and benefit from the myriads of mirrors. Here’s the link for how to get started contacting Sonatype: http://nexus.sonatype.org/oss-repository-hosting.html

Publishing Your Scala Project to Central via Sonatype OSS

If you maintain a project that previously published to the scala-tools.org repository, here are three resources that provide guidance for Scala developers looking to publish Scala artifacts to Central:

• Publishing artifacts to Sonatype instruction written by Joshua Suereth on publishing to Sonatype OSS
• Publishing SBT Projects to Sonatype OSS from Cake Solution’s Specs2 Spring Project
• PublishToSonatype.scala some Scala code from Paul Phillips to automate the process of publishing artifacts to Sonatype’s OSS

“Wait, that’s how Central works?”

There’s something so automatic about dependency management in Maven that it often takes people a few months to understand exactly where those JAR files are coming from. In an 8 hour Maven class, I get to dependencies in the third hour, and after describing Central, what it is was like before Central, how metadata is stored in a repository alongside binaries, transitive dependencies, etc…. it all falls into place, and people realize that this simple thing they’ve grown accustomed to is only easy because of a ten year effort to refine the model, the creation of a support structure for source forges at places like Oracle and Google, and a constant investment in infrastructure.

On the one hand, it’s a great success that Central is, for the most part, an invisible utility that supports developers.  On the other hand, it’s the kind of thing that people can start to take for granted very easily.

For example, a few months ago I spoke to someone who worked in an environment disconnected from the internet for security reasons.   This individual was talking about how limiting it was to have to download JARs from open source projects manually and assemble them in a project.   His words were: “It’s like programming Java in 2001 again.”

How can you help?

Imagine millions of developer spread all over the world: different time zones, different applications, but they all hit the same service: Central.    Some regions have more developers than others so we certainly see peaks in usage throughout the day, but in general, Central’s serving thousands of files throughout the world at any given time during the day.

Maybe someone just installed Maven for the first time, or maybe they blew away a local repository, with numbers like these we see a world that has a constant appetite for artifacts.   It isn’t a problem for Central, and I’m not writing this because Central is falling down on the job. Central can handle it, but it certainly isn’t the most efficient way to support millions developers.  It isn’t a good use of network bandwidth, and it isn’t a good use of energy to constantly cart around the same static JARs over and over and over again when the solution is so easy.

If everyone who used a build tool that interacted with Central adopted a repository manager such as Nexus we’d have a faster, more responsive system.   Central’s maintainers would be focused less on addressing the occasional runaway build and could spend more time and resource on increasing availability and functionality of this essential service.

Broken Builds

The other factor playing into this is that Maven builds only download releases once.   It isn’t like these build tools are repeatedly returning to Central to download release artifacts over and over again.

Well… actually… that isn’t true, we’ve seen some installations of Hudson configured to delete a local repository before every build placing a high load on Central.   Imagine a build that downloads 50 MB of dependencies running once every 5 minutes.  That’s one build consuming ~14 GB a day never mind the time wasted downloading static artifacts. While these broken builds are the exception, they do still show up from time to time.  Central can handle the load, but imagine 1000 of these broken builds running continuously and you can see the challenge.

A Simple Reminder: Please Proxy Central

We’re constantly watching the performance of the system and making sure it stays up and running for an entire world of developers.  If you use a build tool that hits Central whether it is Buildr or Maven or Gradle or Ivy, you can help us by running a Nexus instance.

Even if all of your builds work perfect, running a local Nexus instance helps preserve Central as a public, free resource and it will lead to faster, more responsive builds.

At Sonatype, we want to make synchronizing and publishing your artifacts to Central easier and to improve the quality of repository metadata for everyone at the same time. To facilitate this, we offer a dedicated instance of Sonatype Pro for Nexus at http://oss.sonatype.org specifically to host the artifacts of open source projects. In this post, I talk about the process of creating a repository for your open source projects and publishing artifacts so that they will be available from the Central Repository.

This service has been available since 2009 and includes many projects such as Plexus, Jetty, Google Guice, Spring and Ehcache (Greg wrote about his experience with migrating to oss.sonatype.org). We have tooling in place to make it easy for us to process a larger set of requests, so we invite everyone to use this resource. As of October, 2011, we have over 1,500 projects using this repository on a daily basis.

To get the process started, go here. We’ll setup a release and snapshot repository for your project, along with the appropriate configuration to allow you to use the staging features for your releases. If you have an existing repository somewhere, we can migrate that for you too. We’ll even help you add artifacts to Central that you use, but don’t necessarily own — assuming of course that it doesn’t violate the projects license.

The system allows customizable rules to be run during the staging process, which allows us to automatically check things like valid pgp signatures and correct POM parsing. This will ensure that your users have the best experience possible when using your artifacts, and relieve some of the manual validation on your side — a win for everyone.

On the technical details, this instance gets its network connection via Contegix‘s high availability network, the same one running Central, Codehaus.org and Atlassian.com. New Relic has donated monitoring services to help us monitor and tune this instance of Nexus. Since OSSRH is hosted on the same infrastructure as the Central Repository, we are able to frequently synchronize the repositories.

Next time you need to add a project to the Central Repository, you’ll know how.

While this seems like such a simple idea, the Maven ecosystem hasn’t had a standard way to search the repository for the majority of its history. For much of the last decade there was no reliable way to search for an artifact. In this post, I’m going to review this history and talk about Maven repository search and where we think search is headed. With the release of Nexus OSS 1.9 it is now a good time to summarize the results of Sonatype donation of the Nexus Indexer to the Apache Software Foundation.

In the beginning…

In the beginning there was ibiblio. That’s not entirely true. When Jason van Zyl created the first Maven repository in 2002, there were a number of different servers and mirrors involved, but after a few weeks of migrating between Apache servers, the first incarnation of Central ended up at ibiblio.

If you were depending on the Maven repository back then, you have a good appreciation for how far we’ve come in just a few years. Back then, the bandwidth was terrible and the connections were iffy. It was touch-and-go both in terms of stability but also in terms of process. You can still see some of the echos of the initial effort today in the form of old group IDs. If you are wondering why projects like log4j and commons-lang exist as top-level group IDs it is because the initial artifacts didn’t follow the same strict standards for naming. In 2002 the repository was emerging, and it would take a few years for people to agree on standards and formats.

Maven’s Structure Becomes a Standard

Within a year, people using the Maven repository had come to rely on it as an essential piece of development infrastructure. While Java had been around for almost a decade, no one had created a viable repository structure. There was no way to distribute artifacts between projects, and the Maven repository was created at a critical time in the development of open source Java. As Java really start to take off in the enterprise, as systems grew more complex, and as an explosion of open source Java libraries hit the scene from places like Apache, the Maven repository quickly became the established way to distribute artifacts. There was no going back.

Maven Search: The Dark Ages

Years passed, the format of the repository changed between Maven 1 and Maven 2. As more and more projects started to use Maven or publish to the Maven repository, there was a need to create some sort of search interface to help developers locate artifacts in the repository.

Initial efforts for repository search were disconnected and relied on multiple, independent systems running proprietary analysis on the entire repository. (I know this myself because I took at stab at writing one in 2006.) To search the repository, you had to download the entire repository and run a series of regular jobs to grab changes and update an index. There was no standard search mechanism that would facilitate tool integration, and most people discovered Maven artifacts either by clicking around in a browser or by word-of-mouth.

Earlier Repository Managers and Search

Early repository managers contained independent implementations for search indexing. Early versions of Archiva had an independent index library based on Lucene, and Artifactory relies on a JCR store. While it was clear that repositories were going to play a key role in providing an easy way to search for artifacts by metadata and class name, these earlier approaches still required people maintaining repository managers to download the entire repository and run a time consuming, CPU-intensive process just to create searchable index.

This is where the Nexus Indexer started to come into play.

The Nexus Indexer

When Sonatype created a repository manager, we also aimed to create a standard and sustainable way to search for artifacts. The Nexus Indexer defines a standard format for repositories, but, most important, it defines a portable format that captures information about a repository. This format is what your repository manager downloads from a remote repository, and it is the reason why repository managers like Nexus and Archiva can allow you to browse the contents of a remote repository without downloading “the entire internet”.

Sonatype’s open source repository manager, Nexus, was the first repository manager to define a standard format for a repository index. We didn’t just define a product with a search feature, we set out to define the standard. Using this model, servers like Maven Central and other popular open source forge repositories would periodically index repository contents and present clients with an index. Instead of downloading Terabytes of data from the internet, your Maven client, your IDE, your repository manager could download an optimized index containing all of the metadata about artifacts in the repository.

This innovation also had the effect of offloading the responsibility from your repository manager. If you want to search the repository, you don’t have to set aside a few days for your own instance to crawl Terabytes of data. Maven Central is indexed once, in a central location, and the world saves zillions of CPU cycles because of that fact.

The Nexus Indexer was an immediate success, the index format was created on a weekly schedule on Maven Central. Sonatype carved out the Nexus Indexer as a separate component, released it under a very conservative, BSD-style license, doing all we could to make sure that everyone interacting with the repository could read this format. Very quickly all of the repository managers on the market could both read from and write to a Nexus Index. For example, Archiva now relies on the NexusIndexer, and will likely move to the Maven Indexer.

Donating Nexus Indexer to the Maven Project

As this index format became more widespread through the Maven ecosystem and more generalized for other languages and systems. Sonatype thought it made perfect sense to remove any direct association with Nexus. It was becoming a part of the foundational infrastructure for the community, and, in a decision that might make some executive’s heads spin with disbelief, we donated it to the Maven community. We gave it away.

For something this low-level, this important to the community, it didn’t make any sense for Sonatype to hold on to this resource. The Nexus Indexer is now called the Maven Indexer and the code behind this index is now a part of the Apache Maven project.

At Sonatype, we understand the role we play in supporting the universal infrastructure that enables the world to develop and collaborate. We also understand how important it is to ensure that something as important as the index format be truly open: managed by an active and healthy community, free of difficult licensing questions, and unencumbered by patent concerns.

Next Up: Nexus OSS 1.9, Now with the Maven Index

To complete the circle of our successful transition of the Nexus Indexer to the Apache Maven project, we’re announcing that the next version of Nexus is one of the first projects to incorporate the newly minted Maven Indexer. Stay tuned.

To improve the experience for users in Europe, Sonatype has provisioned a new official repository in the United Kingdom. This is more than a mere mirror of Central, this system is updated in lockstep with the systems here in the US, and is managed and monitored 24×7 by Contegix, the same team watching over the US repositories. The new repository consists of two fully redundant systems running in parallel to provide complete fail-over capacity.

In addition to the new repository, we have taken several steps to improve and further secure Central itself:

• A new system has replaced Central as the inbound processing engine. On this staging system, we can now vet inbound artifacts for quality and other parameters before publishing them to repo1 and Europe. It also serves as a hot standby for the US repository.
• We’ve worked with Contegix to implement additional layered security around the repository machines themselves.
• There is a new Jira project to manage any and all concerns and issues with Central, the Mirrors, Content, etc
• We are working to setup another official Central Repository in Asia soon

The new repository is live at http://uk.maven.org/maven2/ if you’re using a repository manager, just replace references to http://repo1.maven.org/maven2 with the new url. If you’re not, you should be (Whitepapers: Intro to Repository Management / Stages of Repository Adoption), but until you get a repository manager in place, add the following to your settings.xml:

<mirrors> <mirror> <id>uk</id> <mirrorOf>central</mirrorOf> <url>http://uk.maven.org/maven2/</url> </mirror> </mirrors>

Some additional coverage on this topic can be seen at InfoWorld, BusinessWire and InfoQ

Better maven support has been frequently requested on the issue tracker and mailing list, and this is a first step in that direction. In the future, Google will publish GWT releases to maven central as part of the release process.

The GWT 2.0.4 jars currently in the repository include gwt-user, gwt-dev, and gwt-servlet.    To publish these artifacts in the Maven Central repository, Google publishes artifacts to Nexus OSS, the Open Source oss.sonatype.org repository.   You can see the Google-specific repository on this server here.   Releases are staged to this Google repository on oss.sonatype.org and then subsequently released and synchronized to the Maven Central repository.

Configuring the GWT Plugin

To start developing with GWT, take a look at the “Automatic Mode Setup” section on the GWT Maven plugin’s Setup instructions.   Before last week, the only way to develop a GWT application with the latest version of GWT was to download the SDK to your workstation and then use systemPath dependencies or a custom task to publish artifacts to your local repository.    Today, you can just point your Maven project’s pom.xml at the correct version of gwt-servlet and gwt-user and Maven will grab the necessary native libraries from Central.

This doesn’t just make GWT development easier and more straightforward for people already using the tool, it will make it much easier for developers to start using GWT.   When you publish your project’s artifacts to the Maven Central repository you make it easier for people to adopt your technology.   Maven Central is the “dial tone” for most developers, and if you put it on Central, they can access it without having to download an SDK or configure a build system.   Maven Central just works.

Nexus OSS is the fastest, most efficient way to publish artifacts to Maven Central, and Sonatype has made this service available to any open source project that needs to publish artifacts.   If you work with an open source project or a company which publishes open source libraries, read the Sonatype Nexus OSS Repository Guide to get started.