Why the Dark Launch Is the IT Director's New Best Friend

The world of physics gives us two interesting concepts: dark matter and dark energy.  Cool and mysterious as these sound, they're really just scientists' cagey way of saying they have no idea what's going on in the universe.  As an IT director, you have something equally cool-sounding at your disposal, with all the cachet but much less mystery: the dark launch. Sounds impressive, doesn't it?  It's like you're issuing some kind of top-secret software deployment, and you should probably wear sunglasses and a suit while you do it.  Okay, so that might overstate things a touch. But it certainly has a nice ring to it. But what is a dark launch? You might have heard the term tossed around in context before without fully grasping the meaning.  So let's get specific.

If you google around for the term, you'll find some rather scientific-sounding definitions.  Those can be very precise but not always very clear.  So I'll start with an example instead of a definition. Let's say that you have a new and more secure implementation of registering new users to your site.  Historically, launching this is simple in concept, if not in practice.  You make a big announcement on the feature, promote it from staging to production, cross your fingers, and buckle up, ready to act or roll back at any minute.  Such is the life of those of us that deploy software. But a dark launch takes a different approach.  You don't announce anything or buckle up at all.  Instead, you implement your new registration scheme and add a feature flag that allows you to toggle live between the new scheme and the old one.  And then you just launch it to production, with no fanfare and with the new functionality turned off. This lack of fanfare and lack of change to your application is why it's called a dark launch. Because the feature toggle is turned off, you're basically just launching a tiny, zero-risk patch to your existing code.  Except now your new functionality sits quietly in production, ready for some testing.  At any point in time, you can toggle the feature flag, try out the new scheme, and put it back quickly if need be. Dark launches, then, are these stealthy, risk-mitigated launches.  They're a process whereby you decouple deployments from production validation. And they're powerful.  Let's take a look at why this should become a prominent tool in your tool belt.

Let's start with something we've already covered implicitly and just say it explicitly.  Dark launches flatten out your group's risk profile. You develop software for periods of time where you incur very little risk as the previous version hums along in production.  Then, periodically, you summon all hands to the deck for a massively risky launch, scrambling frantically when things go wrong.  And I say "when" because things do go wrong. The practice of dark launches mostly de-risk the actual production deployment.  You can then incur the risk of turning on the new functionality at your relative leisure, picking your spots and carefully controlling the process.  The overall result is that you exist in a state where you no longer have periods of outsize risk.

At just about every enterprise with which I consult, IT and the business tend to duke it out over the idea of "technical stories." (This is what agile shops call them, anyway.) These are changes you make to the code that have no bearing on the user, such as refactorings.  To understand what I mean, consider their two perspectives:

1. IT: "We need to spend some time reworking the data access layer or we're going to be in trouble later."

2. The Business: "That's a time sink that doesn't get me what I need, so I don't want you to do that now."

I want no part of wading into the middle of this philosophical fight when I don't need to.  But I'm sure you've had it and I'm sure you can relate. What makes this particularly unpleasant for the business is the fact that it isn't just a matter of opportunity cost but also of risk.  If you spend a few weeks reworking the data access layer and then burp out an update with no new functionality, they're going to get absolutely hammered if anything goes wrong with it. Dark launches smooth over a lot of the differences here.  By de-risking the launch, you make so-called "technical stories" much more palatable to the business.

Speaking of pleasing the business side of the house, dark launches offer you another olive branch.  Traditionally, IT groups are loathe to respond to apparently frivolous business requests. "No, we're not shipping a new version of the software because you think a blue 'purchase' button would work better than a green one." But the dark launch concept lets you accommodate many more such small requests.  In the first place, you can batch these requests and work them into the next launch together. But more generally, dark launches affect your deployment process over the course of time, prompting you to make more frequent, low-risk pushes.  As you guard more and more of your code with runtime-configurable toggles, you can accommodate more and more experimentation requests from the business. And this is good for everyone, since there's no better way to know what works and what makes money than to run actual experiments with actual users.

Another benefit to the dark launch practice is simple but beautiful.  You can get out of the business of doing special beta launches. Sure, the beta launch should wind up identical to the eventual full production launch.  But does that ever really happen?  There's always special circumstances, one-off configurations, and the like.  Managing a beta run winds up just being two separate product launches where users are a little more forgiving of warts in one than in the other. But if you make dark launches, the beta run is just part of the single launch.  You launch darkly, turn the feature on for a few people as a beta, and then for everyone as a general release.  And you do it not by pushing any code anywhere but by changing a simple setting.

I've talked a lot about the app dev and operations angle on things, but there's more to the whole process than just that, obviously.  Any sophisticated shop has multiple levels of environments and personnel managing the deployments among them.  You also almost certainly have a dedicated set of folks doing a lot of testing in these environments. Dark launches won't eliminate the need for any environments besides production.  But they sure do help you deemphasize obsessive testing of every last detail in your staging environment. With lower-risk rolls to production and increased comfort testing on live users, you can realize savings.  You don't need as many man-hours devoted to pre-prod testing efforts, and you can even potentially save on environmental infrastructure.

I'll close on a more personal note.  Everything I've mentioned so far is really company-focused, and so it should be.  We make these strategic decisions for the benefit of the company -- saving money, having better quality releases, and the like. But you also can win personally.  Is it better to be part of a group that does things the old-fashioned way?  Or is it better to be part of an organization leveraging the latest techniques and the most sophisticated tools?  If you're career-minded, it's the latter. So get to know the dark launch, both as a technique and as a philosophy.  It's a win for everybody involved.