In this episode of our #c9d9 video podcast, our expert panel included: John Harris , Head of DevOps & Solution Architect for LeftShiftIT; Viktor Farcic , Software Architect at Everis; Flynn , solving problems at Datawire.io; and, our very own Anders Wallgren and Sam Fell. During the episode, we discussed the differences between deployments and releases, as well as advanced deployment patterns and tactics such as rolling, Blue/Green, Canary, Big Bang, feature flags and dark launches.
»Deployment vs. Release
Not having to #release all at once is necessary if you want to release at #scale , per @_flynn live on #c9d9 pic.twitter.com/B4OCdK3vvf — CloudBees (@electriccloud) September 6, 2016
"If you are doing it right, you are probably doing way more deployments than releases" - @anders_wallgren #c9d9 pic.twitter.com/IIXcxSisNk — CloudBees (@electriccloud) September 6, 2016
Pipeline is king for seeing how your deployments perform in production, per @johnharris85 on #c9d9 . pic.twitter.com/0Cr9HuDgDJ — CloudBees (@electriccloud) September 6, 2016
Flynn 
provides his definition of releases and deployments, and describes how releases are different in Continuous Delivery: “I tend to think of a release as being a chunk of software that you think is ready to go, and the deployment as the way you actually make it go…I feel that back in the days when everybody was doing big bangs, we tended to use releases as a way of collating all the dependencies together that worked. A given release was the line in the sand where everybody had to adhere to a common set of protocols and a common set of interaction patterns. In the continuous world, it feels much more like you have to arrange it such that the interaction patterns are always there, which implies that if everybody is always adhering to the same interaction patterns then you can deploy stuff whenever you need to, but it also implies that you can't do things like switch everybody from protocol version 1 to protocol version 2.”
For Harris , the difference between releases and deployments is philosophical: “For me it's kind of a philosophical split rather than a physical one, I think the release part of it is more in the business, it's a kind of disconnect – the business should be able to choose when to actually release new features and pieces to their customers, and IT should just enable that. The deployments are just rolling the whole time. If IT decides that this is a release candidate worthy of going out and this is different from a deployment.” 
Once you get to the release stage, all you can do is prepare for the worst and expect the best, says Farcic : “As you soon as you get to the point of releasing or deploying you are already in bad shape, because you don’t know what you are going to put into production. The only thing you can do at that point is put a helmet on your head and hope that nothing is going to fall on top of you.”
Wallgren advises to always do deployments before deploying to production: “Hopefully you are doing deployments before you deploy to production. Snowflake environments and snowflake deployment scripts gets us in trouble all the time. Focus more on using the same processes, even if you are going into different environments, then we are a lot more confident at the deploy to production or release time that it’s going to work.”
»Rolling Deployments
The more you can do a push-button rolling #deployments , the better, says @anders_wallgren #c9d9 pic.twitter.com/7IQuFfWCsB — CloudBees (@electriccloud) September 6, 2016
Per @samueldfell - Rehearse the deployment pattern as early on as possible so that you're not trying to create a script only for prod. #c9d9 — CloudBees (@electriccloud) September 6, 2016
Wallgren on the benefits and considerations of rolling deployments: “The one thing about rolling deployments is you get zero-downtime which is brilliant. What you have to put into is you have to architect the product in such a way that you can do this. You have to be able to have multiple versions of most things up and running. The database ends up being the problem when running multiple versions at the same time.”
If you are doing rolling deployments, make sure you are rehearsing your deployment up front, says Fell : “Rehearse the deployment pattern as early as you can in the cycle so that you are not spending hours of time creating some fancy script just for production and having one-sy/two-sy kind of deployment scripts for pre-production because then you are not testing your process.”
Farcic provides his simple definition of rolling deployments: “Rolling deployments are about running multiple instances of something and then updating parts of those instances. At any given point, at least some of the instances are always running. More likely in some period of time you are going to have a mixed release in production.”
»Blue/Green Deployments
Blue/green #deployments most often happen in silos @johnharris85 #c9d9 pic.twitter.com/xUSGFK36B1 — CloudBees (@electriccloud) September 6, 2016
A huge monolith doesn't fit into blue/green deployments, per @vfarcic on #c9d9 | #software #Release #deployment pic.twitter.com/YulG55Xg1m — CloudBees (@electriccloud) September 6, 2016
Blue/green #deployments can be expensive in terms of time, says @_flynn | #c9d9 pic.twitter.com/iVMGKdZT9d — CloudBees (@electriccloud) September 6, 2016
Harris provides a definition of Blue/Green deployments: “Blue/Green is where you have two environments which are basically identical and you route all of your customers to one, and then you deploy everything to the second environment, let's call it Green. Then you flip over everyone using load balancers, DNS pointing to Green let's say, and then Blue becomes a standby and if anything goes wrong you can quite easily flip back, you can flip the DNS back to Blue and fix whatever terrible thing happened with Green.”
Thoughts on blue/green deployments in the Cloud, per Flynn : “Even in the Cloud, with blue/green you need to completely deploy the new thing. Even though it doesn't have to be very expensive in terms of money, it can definitely arrange it so that the deployment itself is fairly expensive in terms of time. If you're going to deploy the entire second environment, then ideally you would have some way of making yourself comfortable, but your second deployment is working correctly before you switch all the production traffic to it.”
Farcic : “I really like Blue/Green deployments because of testing. It’s the only deployment that truly gives me the ability to test something. If I do a second release, it works, then I change the Proxy – it’s that easy.”
»Canary Deployments
Per @johnharris85 : Canary deployments are when you deploy a release to a small set of users to gain metrics and refactor as needed #c9d9 #IT — CloudBees (@electriccloud) September 6, 2016
Canary deployments are difficult to do in monoliths, says Flynn : “I know that Netflix has done a bunch of stuff with having the infrastructure to switch the way routing happens, it's all about being able to take a group of people to do this too and they will induce failures and make sure that the system is resilient in that case. It feels like this sort of a thing would be extremely difficult to do in monolith.”
There are a lot of moving parts in Canary deployments, so having the infrastructure to support them is critical, says Wallgren : “The more in control of your architecture you are at the application level, the easier it is. You are going to have to have an IT infrastructure that supports it because you are going to have to have a way to steer the traffic – whether it’s having the load balancer do it, or your DNS do it – you have to have some sort of way of saying I only want 1 of traffic or only people in Connecticut. Then you need to have the ability to run that stand alone application.”
»Big Bang Deployments
Those using #CD are probably more likely to do big-bang #deployments - per @vfarcic | #c9d9 pic.twitter.com/bFF6cSuDkT — CloudBees (@electriccloud) September 6, 2016
If you want to crank quality things out quickly, you have to think strategically about #deployment patterns - @anders_wallgren #c9d9 — CloudBees (@electriccloud) September 6, 2016
Harris explains Big Bang deployments: “In a Big Bang deployment, once it's gone, you've deployed, you've done it, it's out there, you can’t fix anything or worry about things after that. If anyone is interested in a specific topic, there is a fantastic talk by Gary Gruver, he used to be the head of IT at HP, and he talked about how they implemented continuous delivery for firmware, he said if we can do continuous delivery for firmware then you guys can do it right.”
You can really use any kind of deployment pattern you want if you are using Continuous delivery, but Big Bang is the most likely choice, says Farcic : “What runs in CD can be Big Bang, it can be Canary, it can be a Rolling deployment. Those using CD are probably very traditional and are probably more likely to do Big Bang, but not because it is CD.”
Wallgren relates how installers are treated to how deployment is treated, and why that needs to shift: “What we often do is assign the most junior, hated person the job of running installers. Which is dumb because running installers is really difficult and really important. The problem is anyone who wants to be an installer shouldn’t be allowed to, and I feel to some extent it’s the same way with some people’s deployment patterns – “It’s IT’s problem let them do it” – but you need to think about it better than that. If you really want to crank this stuff out quickly and with quality, then you need to worry about those things. We worry about it less than we should.”
Metrics play a major role in any deployment pattern, says Fell : “We can also look at telemetry from inside the product that’s giving us information on what’s the performance like, and all this other information that the web guys use all the time when they are doing a Canary deployment or whatever type of deployment, to make sure that things are square before they keep plowing forward.”
»Feature Flags and Dark Launches
.@_flynn advises to ask for a dark launch- How important is it to you that no one is going to see it until they have permission? | #c9d9 — CloudBees (@electriccloud) September 6, 2016
"#Software is now eating the #deployment process as well" @samueldfell #c9d9 pic.twitter.com/pAjtL8JcIN — CloudBees (@electriccloud) September 6, 2016
.@vfarcic says feature-flags are related to coding techniques as opposed to deployment techniques #c9d9 #softwaredevelopment #coding — CloudBees (@electriccloud) September 6, 2016
Feature flags can be beneficial, but understand there is a time and place for them, advises Flynn : “Feature flags gives you an opportunity to roll the stuff out, to know that the developer can enable the thing for testing and also know that you have an additional layer preventing that from being something that costumers can see before they are ready for it or before you're ready for them to see it. On the other hand, from the developers’ point of view, it also tends to involve a little bit more work, particularly for a feature that's relatively far reaching. There is a time and a place for a lot of these things, and the balance about feature flags has to do with how much hustle is it going to be for the developers to put in a runtime switch that controls this thing, and then balancing that against how important is it to you that you know that nobody is going to see it until you exclusively give them permission.”
Farcic explains: “In most cases, feature flags are more related to coding techniques than deployment techniques. You are going to deploy with one of the techniques we have been talking about, and then once you deploy it (no matter how you deploy it) then your feature flag in your code kicks in.”
Fell on software and deployments: “Software is eating the world. Well, software is also eating the deployment process. The release process is basically a feature flag, and you have to be careful with that. Your pipeline has an additional vulnerability potential.”
Facebook is an example of a company that is good at doing feature flags and dark launches, says Harris : “My favorite story about feature flags and dark launches is Facebook, one of the best companies that does this. It was two or three years ago when they said that 90 of the stuff they are going to release over the next 3 months is already in production and they are going to test it while watching under the hood. I think Facebook Chat got out six months before it was even on for users, and they were actually sending client-side calls faking chat massages to all the users through their service.”
Watch the full episode here:
Want more Continuous Discussions (#c9d9)?
We hold our #c9d9 podcast every other Tuesday at 10 a.m. PST. Each episode features expert panelists talking about DevOps, Continuous Delivery, Agile and more.
The fast-track to bullet-proof deployments
Want to get Deployment Automation right? Download the free community edition of CloudBees Flow , to easily model and deploy your applications, in a fast, predictable and safe way!