Reinventing Wheels and Opportunity Cost (or Why you Need to use Nexus)


June 13, 2012 By Tim O'Brien

I hear two sentences.. often. It’s either:

  • “We’re not big enough for a repository manager.”
  • Or, the increasingly popular, “We built our own repository manager. It’s just a caching HTTP Proxy.”

I can understand the first statement. People usually think “they are not big enough” for a repository manager when they haven’t understood that running a repository manager yields benefits regardless of scale.

It’s the second statement that I find problematic. (First, it’s *way* more than a caching HTTP proxy… Just saying that demonstrates that this person never bothered to try one out.) If someone has implemented a custom repository manager it suggests a certain amount of self-assuredness that is going to complicate things going forward. It suggests that whoever I’m talking to hasn’t yet understood the value of focus and making pragmatic decisions about technology.

“Look, we wrote a Custom Python Repository Manager for Java”

Here’s an example: a few weeks ago I was talking to a colleague who works at a large, high-profile technology company (which will remain nameless). He was discussing some challenges they were having with builds. I suggested that they’d likely benefit from a repository manager. Understand that I’m not in sales, I don’t get a commission, and that I wasn’t even pushing Nexus Pro, I was suggesting that he take a look at Nexus OSS. The conversation went something like this:

Him: “Well, we’re having a tough time figuring out how to manage releases and deployments We’re mostly Java, and we use Chef for deployments. Our Chef deployments run the builds as well.”

Me: “You use Hudson or Jenkins, right?”

Him: “Of course, we use Jenkins?”

Me: “Why not just create some jobs on Jenkins that push out release artifacts to a repository manager? Decouple the build from the deployment, really, it will make both the build and the deployment simpler. I’d rather walk barefoot on broken glass than have to wait for my chef build to download Maven and trigger a build.”

Him: “Yeah, I wasn’t going to mention that, but we don’t use Maven?”

Me: “What do you use?”

Him: “Our Build engineer said there was nothing out there good enough for us, so he wrote his own build system in Python and it has a built-in concept of a repository manager.”

Me: “Wait. (long pause) You run a startup that doesn’t turn a profit. You spend investor money on some Architect Astronaut to reinvent Maven and a repository manager? I mean do whatever you want to, but that sounds insane, no?”

He continued to tell me that he was trying to reward this developer by giving him a chance to work on this custom build system, but when I pressed him on the real reason it had more to do with the fact that he had larger battles to fight and didn’t feel like builds and repository managers was “the hill he wanted to die on” (his words).

Now, understand two things about this particular company: 1. You know of them, and 2. Reimplementing a build system for Java in Python has zero to do with what you know them for. You should also know that several people in the same department had expressed frustration that they couldn’t: “just use Maven”. This is a common disconnect, a single Architect Astronaut holding back an entire department from adopting the tools they understand (all because he or she has this idea that they can do a better job of builds or repository management).

Wheel Reinvention: More difficult than it looks…

I call this wheel reinvention as it would be similar to letting one of your developers just go off and reinvent something that is already well established: like “The Wheel”. If, comically, you were to reinvent the wheel, you’d quickly realize that it isn’t the concept that is difficult, it is the execution. It isn’t enough to come up with the idea of the wheel, you have to source the raw materials, you have to perfect method of manufacture, and if you are going to get into the wheel business you have to find a sustainable business model.

The same is true for builds and repository managers, while the concepts are simple, the implementation is anything but. Let’s take a look at builds. There are many build tools out there for different languages and different approaches. For Java alone there are three big players: Ant, Maven, and Gradle. (BTW, Gradle just had a 1.0 release yesterday, congrats.) All three of these tools took years and years to get to where they are now, and the people responsible for those tools will be the first to tell you that building software is a difficult challenge not because the steps are particularly complex but because of all of the edge-case conditions.

Repository management is also one of those ideas that appears to be straightforward until you find yourself in the middle of implementing one. At the start of any implementation effort you’ll hear people say, “How difficult is it to just write an HTTP proxy anyway?” or “This is just like WebDAV, easy.” But, give them a few years and you’ll understand that a repository manager (like Nexus) has to do a lot of heavy lifting: integrating with security, verifying PGP signatures, providing support for a number of build tools. Even after six years of development, Nexus isn’t “done”, there are still hundreds of features that could be developed, yet the idea that a single developer could wake up one day and just “reimplement a repository manager” from scratch is one I encounter frequently.

As an industry we encourage this “Not Invented Here” approach to development, and it isn’t just confined to build tools. The same is true for dependency injection frameworks, object relational mapping, and almost every other component. How many times have you had to sit in on a meeting where someone starts describing some custom solution for a common problem that could have easily been solved with something like Spring? As an industry we’ve elevated developers to this lofty status and I’ve seen so many organizations that just enable this idea that “nothing is good enough for us, we have to reimplement everything from scratch”.

The Limited Resource is Time, Not Money

The challenge for most IT departments these days isn’t money, it’s time. It’s developer time. It’s hard enough to find good developers, so why throw them at problems not directly related to your business? It’s a distraction from the multiple responsibilities you have to the business.

There aren’t enough developers to go around. Companies are stepping over themselves to try to hire good technical talent, and if you find a good developer you have to ask yourself whether you want that developer contributing to the bottom line of the company or sitting in some cube reinventing Maven and Nexus under the guise of “devops”. Instead of reimplementing a deployment framework using Groovy or a repo manager using Python wouldn’t you rather that developer be focused on making sure your passwords are properly salted? or that your application provides more relevant features to customers?

Developer time is the limited resource in our industry (even at the largest scale), and I’d just encourage people reading this blog post to think about whether you have a few developers sitting around reimplementing Nexus (or Chef, or Apache, or Spring). Is it worth it? Also, if you have someone working with you who is thinking about reimplementing a repository manager in Python, clearly you need to Download Nexus OSS.