Progressive Delivery vs. Feature Flags with Jenkins X and Rollout

Recently we announced that Rollout, a feature management service, is joining the CloudBees family . In this post, we wanted to talk about how this complements the kind of advanced progressive delivery you can do with Jenkins X and how you can use it with Jenkins X.

Progressive delivery is all about getting changes to production faster, but in a safe way. By gradually rolling out new versions of applications, or slowly cutting over traffic to the new versions, you can feel confident in moving changes quicker to production. Happily, feature flags are also about the same thing! Does this mean that they conflict? No. They operate in very different ways and can be used together to great effect.

Aside: It might be worth a quick read here on what feature flags are, but in summary: they let anyone turn features on or off, without redeploying an application (could be a mobile app or a web app, anything really. But do read that link when you get time).

But let’s step back and think about the trade-offs. The closer to production we make a change, the riskier it is. Either problems for users are caused and the release has to be rolled back, or we have to go back to the development cycle and fix things up, retest (expensive). However, the faster we move into production, the sooner people get the good changes, and the more feedback we get on if people like it or not (perhaps they don’t like that new color!).

Some would go as far as calling it testing in production . Sounds slightly terrifying to me, but it is more of an attention-grabbing name, really it is more like experimenting in production (gah.. That sounds even worse). Despite the misgivings the name may cause, this is actually an important technique for continuous delivery: value is being delivered so fast, there isn’t necessarily time to slow down and stage things in a perfect replica of production. Also, it’s often simply impossible to do. Imagine Netflix or Facebook trying to stand up a replica of production!

And even if they could do that physically, then imagine then trying to replicate the scale and variability of real user load.

Let’s put this in an unnecessary graph showing how dangerous it is where a change happens:

As changes happen further to the right, the more dangerous it is. You could say (well no one has, but I am now) shifting changes to the right is a Highway to the Danger Zone. But just like the song says, “the closer to the edge, the hotter the intensity,” there is value in making changes close to production.

This is where feature flags shine: you can test ideas in production by flipping feature toggles, changing values, and even controlling what group of users see certain features, without having to update the software.

To make this less stomach ulcer inducing, rollout has a “kill switch” where you can deactivate any experiment and “put things back the way they were”:

Comforting. You can also do this with Jenkins X progressive delivery by rolling back to the previous release, but this can take time to propagate or come with its own risks. You wouldn’t want to release a new version of software to “experiment." Mobile is another area where feature flags are necessary (you can’t push out changes to phones quickly), and that is a non-trivial amount of client software these days.

Another important website resource you should check out is the most excellent page: outage party .

Sage advice, and can be helped both with feature flags kill switch, and also the ability to easily roll back to an older version of an app.

Jenkins X comes with a bunch of quickstarts for popular languages, including ones covered by the excellent Rollout SDK . Once you have a rollout account, it comes with an in-app guide for how to integrate for your language of choice. The Rollout SDK then shows how to apply feature flags to JavaScript (nodejs in that case ) - it is a fairly simple API. There are many other SDKs as well (including Golang and Java ). There is a pre-made Rollout quickstart with feature flag integration so you can get started even more quickly. In the meantime, sign up here for a demo and take a peek at the SDK docs.

Both Rollout and Jenkins X have the concept of an “environment." In Jenkins X this is a set of resources (apps, services) which are somewhat isolated. An environment may be called “staging” or “production,” but only by convention (they can be called anything and mean whatever you want).

In Rollout, you typically start with a production environment pre-made for you (Jenkins X gives you both staging and production out-of-the-box. Apps are automatically promoted to staging and then approved to get to production). This works fine as often you only need one Rollout environment as you can control feature flags as needed.

However, you may want to have separations of concerns where you have different configurations or experiments in “staging” vs “production” in Rollout (or even different people and permissions involved). Once you have this setup, it is easy to have the environment in Rollout map to the same environment in Jenkins X, read on.

Firstly, you set up the environments you want in Rollout:

Production comes by default (and is what is used by default) and I added Staging here. I have the same two environments automatically in Jenkins X.

Next, go to the Install Instructions of Rollout:

There will be a drop down for each environment you have. For each environment, grab the Setup environment ID (long random string), as you will need it later. Note which one maps to Staging, and which one maps to Production.

Finally, in your application code that Jenkins X is deploying, you can look up which environment you are by loading the file: "/var/run/secrets/kubernetes.io/serviceaccount/namespace" the contents of it will be “jx-staging” if it is running in staging, or “jx-production” if in Production, and so on (you are clever, you get the pattern here!). This way your application knows where it is running (there are also preview apps, which get their own little namespace, but we can ignore them for now). Once you have the name of the Jenkins X environment, you can initialize the rollout client for the appropriate environment using those environment ID strings we looked up above.

In pseudo code it should look something like:

if (readFile("/var/run/secrets/kubernetes.io/serviceaccount/namespace") == "jx-staging") {

Rox.setup("environment id for staging here")

} else {

Rox.setup("environment id for prod here")

}

Thus Jenkins X can default to production (or whatever you want) in Rollout. Use the IDs from the environment dropdown above in Rollout.

This way, as your applications are promoted between environments in Jenkins X, they use the appropriate flag settings (“experiments”) from Rollout in each environment, it matches up 1 to 1. Note for many users: this may be overkill, but it is nice to know you can do it easily enough for advanced cases. If you do use this, you could, for example, have more aggressive settings in staging environments. This means users trying the application will see all the features turned on (as the Jenkins X app knows it is in staging) but when the app is promoted to production (by Jenkins X) it will get the more conservative production settings.

Lastly, Rollout and feature flags let you let a range of people have control over who sees what features, and when. This isn’t just developers or engineers, but product managers too. You can have many of these features tracked and controlled at once: this is something that would be impossible to do with a different version of your app deployed for each combination, for different sets of users.

As the Rollout post on testing in production says, feature flags let you break out of this traditional trade-off, and delight your users:

So go forth, and test in production! (safely!)

• Read about gradual rollout and roll-back

• Learn about progressive delivery and canary deployements with Jenkins X

• How to create a feature flag