Over the past several years, the frontend JavaScript development ecosystem has been moving increasingly towards the use of dependency managers such as NPM and application bundlers such as webpack. While these are great moves for the ecosystem overall - providing automated and standardized management of third-party code along with modularity - they present some unique challenges for scanning tools such as Sonatype’s Nexus IQ Server.

These challenges were brought into sharp focus for us here at Sonatype when we recently updated one of our own codebases - the IQ Server itself, in fact - to fully use NPM and webpack for its dependency and bundling needs. Once we made this change, we were left to determine the best way to reconfigure our CI server’s IQ Policy Evaluation tasks in order to accurately evaluate the application.

The Challenge

Previously, the Nexus IQ frontend code had a JavaScript source code directory, and then within that, a directory of third-party library files - some added manually and stored in source control, others downloaded at build time using Bower. This was fairly straightforward as it more or less guaranteed that those directories only included files that we were actually using. Once we moved to NPM-managed dependencies however, things changed.

Now we have our own sources, which are still simple enough, and a node_modules folder which includes each third party dependency. There are several difficulties here.

For one thing, the node_modules folder contains both build-time dependencies (such as Webpack itself) and runtime dependencies that get bundled into and distributed with the app. We are primarily concerned about policy issues only with the latter.

Secondly, many NPM packages include extra files that aren’t really part of the library itself, and which don’t typically get used by the dependent application, at least not in a webpack setup. Take the angular-vs-repeat package, for example. In the current version (1.1.7) it includes not just the library code itself, but also its own build files, automated tests, and dependencies. Most notably it includes its own copy of Angular.js - a rather old copy at version 1.2.16.

As it turns out, this happens fairly frequently - NPM packages include their own copies of certain dependencies, even though they don’t use them at runtime.

Nexus IQ: Creating an Accurate View

Back to Nexus IQ, we had to figure out where to point the scanner in order to get an accurate view of what was getting included in the application and what policy issues might be present. Scanning the bundled output file is fruitless - with all dependencies combined and minified into one file nothing gets identified.

The other obvious option is to scan the node_modules directory. This however turns up numerous false positives for the reasons described above. We saw policy violations regarding multiple outdated instances of angular and other libraries that dependencies include in their file tree, but which IQ isn’t actually including into its bundle. We needed a way to scan only those files which Webpack was actually pulling into the bundle. We needed a way to get webpack to tell us exactly which files to scan.

The Solution

Luckily for us, Webpack has support for plugins. By writing a plugin, we could easily get Webpack to dump all of the original files that it is bundling into a separate directory, at which we could then point the IQ scanner in order to scan exactly those dependencies which we need to be concerned with. We wrote such a plugin, and it solved the problems perfectly. Our reports are no longer littered with false positives from files that aren’t truly part of the app.

Get the webpack plugin

We realize that these challenges, and the need for this workflow, are not unique to our own development. For that reason, we released the webpack plugin to our FOSS community repo: it can be found at https://github.com/sonatype-nexus-community/copy-modules-webpack-plugin. It is used much like any other webpack plugin, with a single configuration option telling it where to put the copied source files. See the README for details.

Conclusion

At Sonatype, we are not just the developers of the Nexus line of products, we are users as well. Like many of you reading this article, we combine use of those products with modern development practices, and when we hit a hurdle to that combination, we address it in order to best enable our community to succeed.