Sonatype and the Foundation of the Maven Community


January 12, 2010 By Jason van Zyl

While the Sonatype Nexus team has been focused on making Nexus an excellent repository manager, we have also been working in many different areas and on many different projects to improve the Maven ecosystem. In this post, I’m going to try to describe the ways in which we’ve tried to help set a solid foundation for continued development. For the past decade, we have tried as much as possible to create social capital and increase technical cohesion among the various projects and interests that define the Maven community. Before I can get into the specific ways that Sonatype contributes to the open source ecosystem surrounding Maven and Nexus, I will need to define those two terms: social capital and technical cohesion, and I will need to discuss Maven’s origins.

While Maven was created to meet the needs of a specific project, then a part of the Jakarta project at the ASF, it was also a concept that I had been thinking about for a few years.  I rarely enjoy foisting my own philosophical views on others, and I’m not going to start doing that in this post.   What I will do is talk about some of the ideas about architecture, technology, and progress that helped shape my own view of what problems Maven was designed to solve and how the organization and approach of the Maven community has been shaped by these ideas.

Social Capital

I came to understand the term social capital in 1995 by way of Francis Fukuyama’s excellent book Trust . The flavor of social capital which I have come to like most is that described by Jane Jacobs in The Death and Life of Great American Cities : social capital consists of networks of trust and norms of reciprocity that have emerged over time, which promote the process of voluntary exchange.   Trust, reciprocity, and voluntary exchange are all essential for the development of large-scale projects such as cities which are comprised of communities “characterized by layered complexity and seeming chaos”.   Any attempt to oversee economic development should take into account the importance of informal contact, which Jacobs describes as the “small change” that forms the basis of the fine structure.

deathandlifeSocial capital is primarily used as an economic term to describe the value of networks, but I think as a term used for groups of people developing software it’s quite apropos.  While Jacob’s writing is directly related to urban planning, her criticism of “modernist” urban planning and her embrace of dense, mixed-use urban neighborhoods provides an almost perfect analogy to the way open source works.   Good development projects resemble the sort of vibrant urban neighborhoods that Jane Jacobs advocated for such as New York’s Village which provide appropriate infrastructure to allow individuals to interact naturally and act as efficient markets for social capital.

Social capital is the currency that allows individuals to collaborate in an open source environment, and projects like Maven at the ASF are the neighborhoods in which we interact with one another. As Fukuyama writes in Trust:

“[A healthy] economy is one in which there will be sufficient social capital in the underlying society to permit businesses, corporations, networks, and the like to be self-organizing…The same propensity for spontaneous sociability that is key to building durable businesses is also indispensable for putting together effective political organizations”

Technical Cohesion

When I use the term technical cohesion, it is a combination of ideas from two influential thinkers: the historian and philosopher Lewis Mumford and the architect Christopher Alexander.

Lewis Mumford wrote The Myth of the Machine in which he posits the current mode of technology, which he calls Megatechnics, “work against technical perfection, durability, social efficiency, and overall human satisfaction.” That constant need to expand, produce a ceaseless stream of goods, and replace them endlessly does nothing to improve the human situation.  Now, if you happen to resonate with Mumford’s particular view of technical progress, it can often be difficult to operate in an industry dominated by a new fad every week.  The software industry has been intoxicated with a special brand of “Megatechnics” for some time, if a project hasn’t added several new features in the last month or added support for several new scripting languages, it is considered a failure.   This constant push for change for the sake of change works against durable technical design, it also affects technical cohesion.

technics_and_human_devChristopher Alexander is the author of Notes on the Synthesis of Form and the more well known A Pattern Language . In a nutshell he believed there are a set of healthy, common patterns at the level of infrastructure which yield “wealth” in towns and cities, where to him “wealth” had a broader definition than a simple monetary meaning.   Maven and the tools that it supports were designed to provide just this infrastructure for software development.   Alexander’s work in Notes on the Synthesis of Form relates the ideas of sustainability expressed in Mumford’s Megatechnics and echos the ideas of urban planning put forward by Jane Jacobs.  Throughout Alexander’s work is the idea that design is a continuously incremental and organic process.

In the Maven community, I have always tried to resist change for change’s sake.   This is a deliberate strategy, and I’m confident that this has helped us avoid some of the alienation and social inefficiency that Mumford ascribes to Megatechnics.   For example, even if creating a POM in yaml is a good idea for a few developers, incorporating it into the core Maven code without a long-term vision is a mistake.   With Maven 3, we’ve finally developed the underlying infrastructure to make pluggable POM interpreters a reality.  This could only have happened after some very long-term, strategic processes were worked out over many years.   Hacking on a new feature to keep up with the industry hype machine is not the way to create durable, lasting goods that will stand the test of time.   Mumford’s ideas are somewhat contrarian in a culture defined by “disposable everything”, but, in the case of the Maven community, I believe that they have helped encourage more technical cohesion between the various tools and components that are produced by the community.

Our Contribution: A Foundation of Infrastructure

Once you roll all of these various influences together and add eight years of Maven to the mix, you have an effort to create a set of tools and patterns for constructing, testing, distributing, and provisioning software that reaches an effective system equilibrium. By that I mean that much of Maven has reached the durable ideal that Mumford speaks of in Megatechnics: large parts of “the system” will likely never be replaced, and the system will be widely understood to a point where it passes into the background and people get on with the work that is important.    Think of Maven as infrastructure for a great city like Portland, you rarely have to think about details like neighborhood organization or basic infrastructure like water or power because they “just work”.

These ideas heavily influence the work Sonatype does on Maven, Tycho, Nexus, M2Eclipse, and Hudson. We want the Maven community to be stakeholders and for anyone in the community to build upon the stable foundation we are trying to arrive at.  Much of Maven has reached this equilibrium, and as we continue to progress into the new decade we’re continuing to strive for a durable, sustainable technology laying the foundation for future progress.

One of the ideas that permeates Christopher Alexander’s more recent approaches to building living neighborhoods is the idea that inhabitants, people that will live in a particular neighborhood, must themselves be active participants in the planning and construction of these neighborhoods.    Sonatype takes this idea very seriously when we participate in open source community.   Maven isn’t just something that is a means to an end for Sonatype, it isn’t just the framework that happens to support our commercial product offerings, it is very much akin to our neighborhoods.   We have been involved in this community for eight years, and we’ve been watching the culture and population evolve over time.

We are not just contractors or vendors creating derivative technologies built atop the work of others.  We are central actors collaborating with other dedicated members of the Maven ecosystem, doing messy, often challenging work to develop the social capital and technical cohesion in an open community. We are developing standards and conventions used by a diverse, mixed-used community of millions of developers and we are doing the heavy lifting to make sure that Maven evolves to become an even more solid foundation for global development. Our development and documentation benefits our customers, our community, and, at times, our competition.   This is the sort of voluntary exchange that Jane Jacobs discussed in her work.

Helping to Build a Living Neighborhood

Sonatype has been engaged in a multi-year effort to create free, foundational documentation to support the millions of developers that use the software we contribute to, and we are “daring greatly” to invest in the emerging technologies like polyglot Maven that will form the foundation of innovations yet to be imagined. The next year promises to be as revolutionary for Maven and the growing constellation of related projects as the last few years combined.

There are critics of Maven, Nexus, and the technologies we help create. As there should be. Healthy, informed criticism helps to push the community toward innovation, but I think that many people who approach Maven and encounter its rough edges should know that, like any community, there are participants and observers.   It is very easy to look at a living project such as Maven, the product of years of wide collaboration and find problems.   In any great city, there are still issues to be resolved and progress to be made, while we strive for technical durability, Maven will never be a completely “finished” product.   This year I have noticed a reduction in the number of “cheap critics”, those who might fire up Twitter or some blogging client to send an errant round into the ether. More and more people are realizing that we are here, we are listening, we are ready and waiting to help those who might have questions or issues.

I recently heard a Teddy Roosevelt quote that put Sonatype’s relationship with the community in context.

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly; who errs and comes short again and again; because there is not effort without error and shortcomings; but who does actually strive to do the deed; who knows the great enthusiasm, the great devotion, who spends himself in a worthy cause, who at the best knows in the end the triumph of high achievement and who at the worst, if he fails, at least he fails while daring greatly. So that his place shall never be with those cold and timid souls who know neither victory nor defeat.

The decade to come for Maven is going to be as interesting as the last eight years.