DevOps Table Stakes: The Minimum Amount Required to Play the Game

Ever play poker? Table stakes, or the ante, is the minimum amount to play the game. So let's take a look today at DevOps table stakes—what's necessary to even come to the table and play the game these days.

The idea here is to get you to think a little differently about your approach to DevOps and DevSecOps.

Why Table Stakes?

Let's talk a little about the evolution of systems design.

If you look at these "Death Star" diagrams, you can see that our systems have evolved beyond our ability to reason about. And while this was once true of only FANG systems, it's now becoming true of everyone. With the advent of microservices architectures, any organization's graph is trending in this direction.

So without the table stakes that we'll talk about here, things will get quickly out of hand.

If you look at a lot of the terms that get thrown around these days, you can get confused. But what do all of these terms drive at?

It's really all about bringing things to market more quickly. Now, think about what's happening here: deployments, CI/CD, patterns, and the like. You're not just going to have all of this once but rather dozens or hundreds of times as architectures continue to fragment.

It's getting hard for teams to mentally model how other teams operate. So what are the table stakes for DevOps?

Table Stake #1: Software Security Supply Chain

First up, let's consider that software in the modern ecosystem isn’t written—it's assembled. This means that you're spending less time worrying about how things are built and more time worrying about how your dependencies are built.

It's easy to put poison into your software.

So offerings like Sonatype's are becoming hugely important in modern application development approaches. You need help managing the software supply chain, including with licensing, security, and managing versions.

Table Stake #2: Secrets Management

Any modern application needs credentials—usually a lot of them. Managing so-called "secrets" is becoming absolutely critical these days, lest you open yourself to all sorts of vulnerabilities.

If you want to see this in action, you can search your organization's source control repos for things like "password" and "token" and see what comes up. Hopefully, it's nothing. But for way too many organizations, it's not.

Modern table stakes for DevOps include having a good solution for managing your secrets.

Table Stake #3: Security Integrated in CI/CD

These days, security can no longer be an after-the-fact, bolt-on concern. You need to shift it as far left as humanly possible. This means that you should be thinking about and checking for security concerns right within your team's build.

Part of this involves shifting your reasoning about security, as well. It's not a cost center, but part of a modern organization's value stream.

It's also important to move away from blame culture as you do this. Take a page out of the book Site Reliability Engineering, which has a chapter about Google's blameless postmortem, and learn to conduct your postmortems this way. The key is to identify and fix issues as early as possible, and you can't do that if you're more focused on defending yourself than solving the problem.

Table Stake #4: System Observability and Software Security Logging

While this might be a modern table stake, it's also only partially solved. We as an industry lack a great solution for security logging.

We have things like stack traces and logging, but these are written by software engineers. And software engineers prioritize features and functionality, allowing them to track things from a business logic or behavioral perspective.

So because of this, the industry relies on external scanning and monitoring capabilities instead of instrumenting for security as a first class concern. We're going to need to improve when it comes to how we handle observability and logging, from a security perspective.

Table Stake #5: Continuous Security Verification

There are many different roles and mental models when it comes to building software. Team members with varied and diverse roles all contribute, and they all view security through their own lens.

This leads to disjoint reasoning about the topic, rather than a fluid, ongoing verification process.

Historically, we've just waited for something bad to happen and then assumed we'd have enough information to identify a root cause. But that can easily fail. There might not be a single root cause, and, even if you could isolate one, you might not have the information that you need.

So make sure that you're not being reactive and disjointed. Design processes that test, on an ongoing basis, that the security checks you have in place fire when they're supposed to and that they behave the way you would expect for given inputs and states.

Moving Beyond Table Stakes: Chaos Engineering

As you get to the level of maturity where you're ready to sit at the table, you’ll find that your maturation brings you to a philosophically different point. You'll be ready to think about chaos engineering.

Chaos engineering is “the discipline of experimenting on a distributed system in order to build confidence in the system’s ability to withstand turbulent conditions.”

Confidence is key here. By satisfying the table stakes, you can have confidence that when things go wrong, you're going to catch it and remediate the problem.

When you get to the point where you're gathering data accurately, reviewing and refining policies, and generally adapting to adversity in real time, you have the ability to both ask and answer important questions:

• What is the system actually doing?
• Has it done this before?
• Why is it behaving this way?
• What is it supposed to do next?
• How did it get into this state?

And doing this gives you an accurate mental model of how your system really works, meaning actual objective evidence of what it's doing and what it isn't. This is the key to building better systems and better applications.

You can view the full session on what's truly table stakes for DevOps below: