Continuous Security: Delivering Secure Apps on Kubernetes

In this post, I wanted to talk about the need to “shift left” continuous security practices with apps deployed to Kubernetes. This will touch on some popular open source tools, and how they can be used in Jenkins X and beyond.

One of the benefits of a shiny new Kubernetes setup, perhaps using something like Jenkins X , is that you have more freedom to ship rapid changes to applications. Weekly deploys become daily, daily service updates become hourly. The gap in time between someone merging a change and it starting to roll out moves closer to real time.

This is great. To achieve this rapid continuous delivery (CD), no doubt your team invested a lot of time in automated unit tests and regression tests.

So what about the security aspects of your application? Traditionally application level security would involve code audits, reviews, perhaps runtime penetration testing, approvals, code scanning and so on. The idea is to find problems before they get too far.

However, with the accelerated velocity of application deployment, these tools need to be woven into the pipeline that takes the code to production, capturing problems specific to your app as early as possible, this is what is meant by “shifting left” security practices (you may hear that term increasingly). The “left” and and “right” implied here is the pipeline of software from ideas/source code (left) to production (right).

The continuous security aspects I wanted to briefly cover here can be broken into two areas:

• Static security of your application composition (sometimes known as SAST)

• Dynamic security of our running application (sometimes known as DAST)

The first is where things like source code scanners, and “software component analysis” kick in. A common practice here is to look at how your application brings in libraries, especially open source libraries, as some of those may have vulnerabilities and need to be updated or patched.

In fact, GitHub does some analysis on dependencies and can warn you about problems quite early on.

Super! I should get right on and fix that (that demo repo is open source, but thankfully GitHub don’t shame you publicly for having really old dependencies…)

Ok, so what about the second type: Dynamic security. This involves possibly probing the running application for problems, monitoring for faults, or even deliberately trying to break it.

But wait - I said “shifting left” meant ideally catching things BEFORE they are deployed, but dynamic requires it to be running. This appears a problem, but later I want to show how Jenkins X preview apps can help with this before a change is merged or promoted.

Some popular (and open source) tools exist to help with these aspects of testing.

Firstly, the Open Web Application Security Project publishes a handy tool called ZAP. ZAP stands for Zed Attack Proxy . This is itself a project that can be used to simulate attacks on webapps, probe for weaknesses and so on. It is published in many forms (including a Jenkins plugin ), and there is a handy command called “zap-baseline.” This baseline can be used to scan a webapp for a base set of problems, and report on serious problems or warnings. Happily they make this available as a docker image that is published and updated very often. This tool is a handy complement for dynamic security testing.

For image and component (static) type security, Anchore published the Anchore Engine. The Anchore Engine (which is also helpfully published as docker images) works at the container level, looking for vulnerabilities that have made their way into your containers. This is very useful in a Kubernetes world as containers are the unit of deployment. One of the key features of Anchore is that it looks from the lowest level in the image up for problems, vs just looking at libraries that your application depends on. Typically applications built on images depend on other images that have many layers, which in turn depend on other images and so on (so it can almost seem you have no idea where things really come from).

This tool can apply to images before they make it to production, but you can also use it on stuff already running, in case there are new problems discovered that weren’t known at the time that version of the application was promoted.

Also and most importantly, Anchore has the all important nautical theme that seems to surround tools in the Kubernetes and Docker ecosystems.

As I have written about before , you can think of Jenkins X as an opinionated way to deliver apps atop Kubernetes. The benefit of this is that it allows it to have very specific hooks, or extension points, allowing security aspects to be woven in to the pipeline. In fact, it is so opinionated it makes it possible to do the following to enable image security with the Anchore engine mentioned above:

> jx create addon anchore

This will put all the machinery in place to enable the engine automatically for your team, and then you run run things like:

> jx get cve --environment=staging

Which will disclose problems in the specified environment. This also of course applies to pull requests and inprogress work. In Jenkins X each change (pull request) gets its own environment, so you can check that it is ok from a static point of view before merging it. Of course ideally you capture these at the pull request stage.

For dynamic security, you can use the ZAP tool mentioned above as simple as:

> jx create addon owasp-zap

This adds a hook that runs the “Zed Attack Proxy” on a preview app, as soon as it is deployed. Preview apps are created on every pull request in Jenkins X. Should it pick up any failures on the baseline test, it will automatically fail the build - and report to the users that the pull request/change request needs some work. Preview apps are a mini version of the app running, in an isolated environment.

Note that in both of the above cases, I didn’t have to change any pipelines, or set it up per application/project. This works at the “team” level in Jenkins X. This is a handy way to enforce a kind of governance across projects. Governance is often a dirty word with developers, but in the case of security may be it is warranted (and in both cases here, the impact is fairly minimal but the value of it preventing problems could be huge). Read more about preview apps and environments and hooks here.

This could be what your pipeline looks like:

There is clearly so much more that can be done, and a wealth of tools both commercial and open source.

The Cloud Native Computing Foundation (which is also the governing body for Kubernetes) is considering a proposal for runtime security called “Falco.” I am not sure if it has a nautical theme, but it is certainly container specific. Runtime security is another security measure that is closer to monitoring: watching for abnormal behavior (which also means learning what normal behavior is for your apps). This will be a great project to watch as it looks like it can add another layer of continuous security which has traditionally been considered quite exotic (but will hopefully be easy to apply with Kubernetes) for the benefit of all.

Learn more about Jenkins X

Read about Jenkins X and Anchore

Watch the webinar on DevSecOps