One of the reasons Sonatype is being named to best-places-to-work type lists is our culture of learning. Technical folks, not directly on the engineering team, are encouraged to explore and experiment with new ways to interact with our products. It was late fall or early winter when we were first talking internally about how we might use webhooks. Our products already have webhooks emitters but we were noodling on how we might consume them. What if we listened for webhooks from other repositories or registries? What would we do and how would that impact our customers?
One of the great things about open source is that things like webhook listeners already exist so a quick google search later and we’ve come across this project on Github. What’s even better is that we found someone else had taken the time to containerize it in DockerHub so that we could easily integrate the listener into our demo environments and play. Our version of this is on our Community page on Github
One obvious use case that came to mind was for us to listen for webhooks from cloud registries to trigger scans as containers were being published. While another team member took on some of the cloud registries like Amazon’s and Google’s I chose to explore Red Hat’s Quay. One of the things I liked about Quay was that it could automatically build the container from a GitHub repo if the project had a Dockerfile in it and, it would also run Clair to inspect the OS layer. This felt like a natural fit to our Lifecycle scans that inspect the application layer. It is worth pointing out these two scans are very different in how they look for things. OS layer scans rely on reading the manifest of the package manager which keeps track of what packages have been installed (name based). Lifecycle scans are traversing the layers and looking directly at the files to fingerprint them (hashes).
Setting up a webhook in Quay is very straight forward and well documented.
While Quay supports a variety of events like dockerfile build queued or dockerfile build complete/failed I chose the 'push' event, as that will work whether I have Quay do the build or not. The documentation on the payload the webhook delivers is straight forward and easy to use.
Here is an example of the payload sent from their documentation:
{
"name": "repository",
"repository": "mynamespace/repository",
"namespace": "mynamespace",
"docker_url": "quay.io/mynamespace/repository",
"homepage": "https://quay.io/repository/mynamespace/repository",
"updated_tags": [
"latest"
]
}
Our webhook listener makes it easy for us to associate a simple shell script with a URL we can listen on. First we create the JSON file for the hook itself. This established the endpoint that will receive the webhook, the script it will call, and which values we want to pass through - the second part if the script itself. What this code lacks in elegance, hopefully it makes up for in comments and echo statements (for troubleshooting!) . If reading bash scripts isn't your thing here it is in pseudo code:
Change directory to where the scripts are
convert the 'repository' to a valid ID for IQ
Pull the image
Save the image to a TAR file
Scan the TAR file
clean up
So now, every time Quay builds, or receives a push, the event will trigger a webhook to our listener which in turn runs a script to retrieve and scan the image. I take advantage of the automatic application creation feature in Lifecycle.
Here is what the security scan in Quay looks like.
I'm not suggesting anyone use this project as-is to be a webhook listener for your production environment, this is more of a proof of concept. We recently added a webhook listener to Nexus Repository Manager and tools like Jenkins also can be a listener.
Here's a quick demo video as well!
In Part 2, I delve into how this fits into practices and processes that I see emerging as enterprises make the move to containers.
