Why Manual Verification Still Matters

In the last few years, we’ve continuously been hearing that we should automate, automate, automate. So it might be weird to hear that manual verification still matters. Jeroen Willemsen explains to us why we still need to perform manual checks.

Most applications consist of application code and external libraries that are executed by a runtime:

You’ll potentially have issues in these places:

• third party libraries
• your application code
• your runtime

Third-Party Libraries

The first thing to do is to check your dependencies manually. It’s important to learn how to verify if an issue is actually a problem and how to fix it. Fixing can mean:

• replacing the library
• marking the issue as not being a problem
• changing application code to deal with the vulnerability

After that, you can start automating this process. But there are caveats. For example, not all ecosystems have proper dependency checkers. And even if they do, not finding a registered vulnerability doesn’t mean it’s not there.

Your Application Code

A great starting point for securing your application code is to use a static analysis tool, or SAST (Static Application Security Testing). The tooling here varies greatly among languages and frameworks.

When you decide to suppress warnings or errors, make sure you explain why you’re doing so. This will make it clear to anyone seeing the suppression once you’re no longer on the team.

If you’re looking at a commercial tool, test before you buy. What works for one team, might not work for another.

But these tools will never find everything. For example, a static analysis tool won’t tell you if authorization rules are set up correctly or if you’re disclosing information in errors.

Your Application

If you want to know how your backend is behaving in general, you can use a DAST tool (Dynamic Application Security Testing). This will help you:

• find missing headers
• detect some obvious missed XSS or SQL injection vulnerabilities

It will, of course, create false positives (issues that can be ignored) or false negatives (things that the tool thinks are okay but are in fact a problem).

And when you do find a vulnerability, through manual or automatic tests, you can write your own security test. You can even add “evil user stories” in your backlog. These tests will only run successfully if your application isn’t vulnerable to the attack.

The Container

Basically, this is what your application in a container looks like:

It’s a bunch of layers built upon a base OS layer. At the top is your application. The layers below your application might have vulnerabilities or might be wrongly configured leading to vulnerabilities.

Some tools that can help are:

• Clair
• Anchore
• Lynis
• inspec
• Dockle
• hadolint

The Container on Its Platform

As with the previous sections, each of these blocks can have security vulnerabilities. For example, your Docker daemon might be configured in an insecure way. Or the host operating system could have security issues.

Use tools like:

• Inspec
• ciscat
• Lynis
• Openscap
• OpenVAS
• Nessus

In this part, immutability is king. If your systems are dynamic, you need to scan all your systems. But if you’re using a fixed and immutable image, you can scan only that image. Also, the smaller the host, the smaller the attack surface.

General Issues

There are some general issues as well.

Leaking Secrets

One issue is that secrets may leak. You’ll have to:

• scan your source code (you can do that with trufflehog)
• check your containers (with grep and other tooling)
• verify virtual machine images
• check runtimes
• check CI/CD pipelines

These are all places that might contain hard-code secrets in plain text. Be sure to also validate the process, because maybe not everyone knows how to work with secrets correctly.

Network Issues

For the network, it’s best to deny access by default. If you’re in the cloud, most providers offer tooling. If you’re not, use tools like nmap and checkssl.sh.

And don’t forget that manual verification may still be necessary to identify unexpected infiltration or exfiltration paths.

IAM Issues

IAM issues can manifest themselves at different levels:

• application
• host
• cloud provider

Your cloud provider should have a decent set of tools to get started. Then move up the stack. It will still require manual reviews: are the rules too broad, or too fine grained? Are our rules compliant with legal obligations?

Cloud Hosting Configuration Issues

Cloud providers have their own tooling ecosystem, so use those. But something like Cloud Custodian is “cross-cloud.” If you’re using Terraform, you can look at terraform-compliance.

Should You Automate?

Does this make you feel like you need to automate more? Good! Automate as much as you can. But realize that we can’t catch every mistake with automation.

Also, take risk based actions. Assess the risk of what you see first, otherwise you might drown in a pool of to-dos.

You can check out Jeroen Willemsen’s talk “Why Manual Verification Still Matters” here.