Distributing Binaries: Why not just use a Shared Filesystem?

February 13, 2012 By Tim OBrien

5 minute read time

I've seen it. I've seen the builds you never want to have to reckon with. The kinds of build scripts that make you stop, take a deep breath, turn to your client and say: "This is going to take a while to untangle." Ruinous hulks of procedural build logic relying on a lot of token replacement (both pre and post compilation). These builds, especially the more difficult builds, are in love with the idea of everyone having access to the same shared filesystem, or worse, they use source code management as a way to distribute binaries.

While I've seen this pattern in Unix-based environments, the "Everything is on the Shared Filesystem" build happens most often in environments that have standardized on Windows, and I'm pretty sure it happens because it's so easy to see the shared filesystem as a cure-all for collaboration. If you share Word documents and internal files through the shared filesystem, why not build output. If everyone's already connected to it, why not just dump all your libraries on it? Isn't Central just a glorified filesystem any way?

What a "Shared Filesystem" approach turns into...

It must be easy to fall into this trap, because I've untangled so many builds that assume the presence of an E: drive or G: drive instead of integrating something like Nexus. This is so common, I want to take some time to tell people what I see at the tail-end of these projects. What happens several years after someone makes the decision to depend on a shared network filesystem instead of using Nexus.

These builds are characterized by the following anti-patterns:

  1. Need a Dependency? Just put it on the G: Drive. At the beginning of a project, it's easy. You have five external dependencies, just drop all the JARs in G:\Project\Build\lib and point your Ant build script to this directory. Now fast forward seven years and you have 40 dependencies, each with transitive dependencies. Every time you try to upgrade to a new version of a library you end up having to wade through incompatibilities and inconsistencies. Long story short, no one touches this directory after a while, it's just too difficult to change because the build is so massive, no one understand the ramifications of removing a file from this directory. You end up with 10 versions of commons-lang, seven version of Spring.
  2. We deploy on Linux, but it only builds on Windows. When your build depends on the G: drive, it's usually tough to translate all of these paths to a non-Windows environment. People have tried, but they had to hack at the build script. In general, if operations calls you up and asks for a build, tell them to just grab the latest copy from the G: drive. It's impossible to transition these builds to anyone other than a developer.
  3. Don't delete that! (and don't change it either) Remember six months ago when someone from the infrastructure group decided to delete our files to free up space? That put us back a couple of weeks. Sure, it would make sense to lock down permissions on this thing, but the organization uses it to collaborate on Word documents and budget spreadsheets. Shared filesystems rarely have strong security, if they were really secure they'd be less useful for collaboration.
  4. Can't run the build, the network's down. When everything in your build relies on a shared drive, you are dead in the water when that resource becomes unavailable. Now, if you had Nexus installed and you were using an intelligent build tool like Maven, Gradle, or Ivy, your builds would still work because these tools are designed for disconnected operation.
  5. Oh those files? I have no idea, that guy quit last week. The "Fog of War" surrounding awful builds is what amazes me. By the time a client engages me to fix a build, it's usually because the build has been fumbled by a build engineer who decided to move on. Quite often this shared filesystem contains a random collection of files that once supported one version of the build. When you rely on a shared filesystem to share binaries, it often ends up looking like someone's messy junk drawer.
  6. Right, don't run the production build at the same time as someone else.... When your builds store bits on a shared network filesystem, it is often the case that two people can clobber each other's build. Concurrency is an issue when you use shared network filesystems.

Nexus to the Rescue

Instead of using a shared filesystem to share binaries and dependencies. Use Nexus. Nexus provides a series of advantages over shared filesystems, here's an enumeration of the benefits:

  1. Structure: Nexus repositories follow a standard structure. Whether you are using a Maven 2 repository, an OSGi repository, or the upcoming support for NuGet, you'll be sharing binaries within a set structure. If you need to share a library, your tools will be able to address it using a coordinate system and you won't open yourself up to the "messy junk drawer" syndrome I've outlined above.
  2. Cross Platform Compatibility: Any tool that publishes or reads from Nexus is simply interacting with a set of REST endpoints. You don't have to standardize your build on a single operating system, and it will be easier to deconstruct massive, problematic builds. Your developers can work on Windows, OSX, or Linux, and your build scripts can run on a Hudson instance running Linux.
  3. Concurrency/Scalability: When you depend on a shared network filesystem just the right simultaneous build being run by just the right developers can wreak havoc. With Nexus, you have a web-based system that understands how to serve thousands of concurrent requests. If your team is scaling up, you should scale up to Nexus.
  4. Stability: Don't create build systems that can be held hostage by network interruptions. Tools like Maven, Gradle, and Ivy will keep a local cache of dependencies they download from Nexus. This will lead to more stable builds when compared to scripts that depend on the presence of a shared network drive.

If you recognize some of the anti-patterns in this post, it's probably time to check out Nexus OSS. To learn more about Nexus, go to http://nexus.sonatype.org

.

Tags: Nexus Repo Reel, Sonatype Says

Written by Tim OBrien

Tim is a Software Architect with experience in all aspects of software development from project inception to developing scaleable production architectures for large-scale systems during critical, high-risk events such as Black Friday. He has helped many organizations ranging from small startups to Fortune 100 companies take a more strategic approach to adopting and evaluating technology and managing the risks associated with change.