Why Having a Feature Flag Microservice Is a Bad Idea

Perhaps you have built a tight feature flag package that you can deploy. It has a Slick REST API in a microservice where you can query a flag saying, "Are you on?" It seems easy for other services to talk to this service. Perhaps it seems way better than that ad hoc approach you were taking a few months ago. It might be that you haven't built anything yet, but you know that an ad hoc approach won't work. Your properties file is sprawling with feature flags, many of which need to be removed. Perhaps someone said (or wrote), "We should probably make this into a microservice to keep the logic isolated." This sounded intriguing, so you did some research. And you landed here. I am here to warn you: don't turn your feature flags into a microservice. It will be unhealthy for your team. (If you have already gone ahead and done this, don't panic! I'll cover what to do later on in this post.)

Why shouldn't feature flags be in their own microservice? Aren't microservices meant to modularize the code? As is often the curse of good ideas that get spread throughout an industry, we may have lost some of the original intent behind microservices. So before we go any further, let's take some time to review that intent. One of the lead proponents of microservices, Sam Newman, has talked about some principles of microservices. Martin Fowler and James Lewis, leaders in microservice implementation, have a guide to microservices. I encourage you to read these articles for more information, but to save time, I've used them to compile a list of traits that make up a healthy microservice.

A good microservice is domain-centric. This means it is centered around something the business needs to get done. For example, if you are building a warehouse system, then it may make sense to separate shipping, fulfillment, and product functionality into microservices.

A healthy microservice should be able to operate for the most part without relying on any out-of-band communication. Microservices should have a small set of problems to solve and be responsible for handling all aspects of those problems, including the data, business logic, and sometimes even UI logic. The more out-of-band communication a microservice has to do, the more it has to deal with latency and network failures.

One of the main purposes of modularizing a system is to hide the stuff that is unimportant to the business processes. This has been a common idea since the early days of object orientation. Microservices are no different. We want them focused on what business problems they solve, not how they solve them.

When something fails in a microservice, the rest of the system should not fail along with it. For example, if I am on Amazon's website and I am looking at a product, it doesn't make sense to throw an error if the product recommendations service is down. I still want to see information about the product. Instead, a simple "recommendations not available" should suffice so that I can refresh the page or look up the recommendations at a later date.

Now that we have an idea of what makes a healthy microservice, let's examine how they work with feature flags. Specifically: why are feature flags an unhealthy focus for microservices?

Feature flags don't constitute a business capability unless they're your main business, like with rollout.io. They are a cross-cutting concern, like security, logging, etc. Feature flags don't care what business problems you are trying to solve; they are concerned with functions that can be used across multiple business capabilities. This goes against the above principle that we should center microservices around business capabilities.

A feature flag microservice may be autonomous in that it does not need much more than a database to run and give "enabled/disabled" responses. However, every time a consuming microservice needs to know if a feature should run, it has to talk to the feature flag service. This creates a really tight coupling between these two. These consuming services would not be autonomous, which goes against the ideal conditions for a microservice. Additionally, what happens when you check your feature flags service for a toggle and the service is down? Or it gives you an unauthorized exception due to the wrong credentials being configured? What do you do? Do you have some policy that defaults it to "disabled?" Do you retry? Without quite a bit of extra effort, the fact that you have a runtime dependency on this microservice means your consuming service is at risk of failure. This breaks the principle of isolated failures.

Microservices that use feature flags use them at a low level in the code, beneath their APIs. Therefore, feature flags are an implementation detail. We should be hiding this detail in the bowels of the application. Instead, by having a feature flag microservice, we are calling out very openly to our feature flag service.

If we structure our cross-cutting concerns (like feature flags) as microservices, we may doom ourselves to repeat the mistakes of the past. We will wind up with a highly coupled web of services that will be hard to maintain. We tried this once before, with service-oriented architecture. A concept emerged in the early 2000s called service-oriented architecture, or SOA, which devs touted as being the end to the problems associated with complex systems. At its core, SOA is the idea that distributing complex systems among different teams and creating strong boundaries will reduce costs across the organization. In such a scenario, applications would be easier to enhance and maintain. But for the most part, this ended in failure. There were many reasons for this failure, but a big one was how SOA systems communicated. When you look at the average SOA diagram, you saw a layered set of independently run services. They try to communicate with each other all the way down to the database. This layering created numerous problems. Latency was massive, causing users to abandon websites. The applications were wrought with failures from all the network calls between these layers. If one service went down, all the downstream services would not work. Different teams would mismatch contracts with each other because they were not communicating effectively.

Imagine if your microservices needed to communicate via the network to many of your cross-cutting concerns. You would have a synchronous logging microservice that you would need to call for every log message. There would be a security microservice you would call for every request. You would have a metrics service for every latency value you wanted to record. Your latency would be outrageous, and the goal of high uptime would be out of reach.

Instead, most of these cross-cutting concerns are handled by using logic inside libraries and plugging them into your system, even if they eventually report out to an external persistence store. You do the same with feature flags: put the logic into a library. That way, you get the bulk of your logic running locally and without risk of failure. You still get modularization by making only a few classes or interfaces in your library public. Or you could use a third-party solution that has already done this work for you.

If you've already built a feature flag microservice, don't worry: you can fix this. You can refactor, incrementally and safely, away from a microservice and into a library. I will quickly cover a strategy to do that. Following good refactoring guidelines, you can replace your microservice without spending weeks of risky changes to your system. Check out the steps listed below.

1. Choose a consuming microservice to be an early adopter. If you have many microservices that use your feature flags service, choose one of them to be the guinea pig. Depending on how much your team is worried about risk, you may want to choose an out-of-the-way service that will not change much.

2. Wrap calls to the feature flag service with an SDK. Your team may have already done this. If so, then we can move to the next step.

3. Extract this SDK to a separate library. We need to eventually share this SDK across all consuming services; it is the foundation of your new feature flag library.

4. Extract a core module out of your feature flag microservice. Ensure your API controllers have little logic in them by extracting out the use cases into a core module that is web independent. You may have already done this, as it is often a healthy practice.

5. In your SDK, extract the network calls once more into a set of methods matching your core methods. This will prepare you to shim in your core code underneath your SDK.

6. Normalize your core methods and your network calls into two implementations of one interface. Now you can safely swap out the network calls in your consuming service with in-process library calls.

7. Swap out the network call implementation with the core implementation. Now your service has no dependency on the old microservice!

8. Repeat step 2 for all microservices, using the core implementation. Now all microservices can be changed at your convenience.

Yes, there are a lot of steps, but don't get overwhelmed. The point I'm trying to make is that you can move incrementally and safely to a library, at the right pace for your team, instead of needing one "big bang" approach.

Eliminate your feature flag infrastructure as a maintenance headache. Spend that time thinking more deeply about business problems. That is where the money lies. When you see a feature flag microservice, an email microservice, or something similar, refactor it away. If you hear a team member suggest it as an idea, respectfully inform them of the pitfalls. Modularize your cross-cutting concerns with libraries instead of services.