Scaling Feature Flags: Security Considerations

This is the third part of a blog series that will explore what you need to consider when scaling your feature flag management tool. We’ll look at common challenges around scaling and what you can do to address them. You can catch up by reading the first post on the early stages of feature flag adoption and the second post on infrastructure. In this post, we’ll dig into security considerations.

Your first forays into feature flagging were most likely targeted at specific problems. A few individuals probably started things off, and usage began to grow from there. As more people find out about the benefits of feature flags, though, more people will also want to implement them for their own use cases. With an increasing number of users, possibly even across teams, security quickly becomes a big concern—especially at large enterprises with a lot of sensitive information at stake. An on-premise feature flag management solution can be extremely secure, but as you scale the infrastructure, your security probably can’t scale with it. This is a primary reason why many companies choose to move to a secure SaaS solution.

As remote work takes off in today’s world and teams distribute themselves around the globe, the importance of good security is making itself clear. Many organizations are focused on refreshing security hygiene and best practices. These concerns touch all points of collaboration across the enterprise, and feature flags are no different. Whether you decide to invest in scaling your own self-built feature flagging system or whether you move to an external tool, you will need to make decisions around a few key security-related points, including governance, audit logs, data privacy and enterprise security protocols.

Homegrown systems start out fairly secure. Looking at the first stages of feature flag adoption, many teams begin with configuration files or database tables. The problems only arise when the system needs to scale.

For example, database tables are great because you can use single sign-on and log on with an active directory. But they are also very locked-down, and can be too restrictive for growth. Only a few users can sign in at the same time, for example, which creates an inherent roadblock. Users with admin permissions are the only ones who can make changes—initially just one person per team. As your flags multiply, you’re also relying on specialized technical knowledge to not mess up the schema. If someone less technical needs to make a change, such as a product manager, they will need to ask someone else to do it.

While the system is secure, you’re only allowing a portion of users to actually make changes. These bottlenecks will get exponentially worse over time as flag usage grows across your organization. As you scale, consider how your system will grant secure access and permissions to a variety of different roles. You will need to implement effective role-based access control (RBAC) so that non-technical teams can get exactly the functionality they need (no more and no less). Otherwise, you run the risk of a non-technical person clicking on something they have no business accessing, potentially causing some sort of catastrophic event—or you risk developer time getting eaten away by housekeeping activities that would be easy enough for non-technical teams to manage on their own. You will also need flag approval flows, which create a system of checks and balances by enforcing a review process. Flag approval flows can also help you comply with enterprise security protocols and standards (more on those later).

When an error occurs, the last thing you want to do is play “feature flag guess who” with your team. Database audit logs are tricky. There’s often a significant overhead associated with versioning. It’s much easier to manage with config files, the other common form of early feature flag usage. You can leverage the Git audit log in config files to understand who is making changes and when.

You will always have the option to create and change your audit log schema. Still, you’re relying on specialized technical knowledge to do so, which may not always be present. Also, the bottlenecks around governance mentioned earlier are of course still in play. If you’re using database tables or config files, you will be dealing with these issues simultaneously.

Config files will be easy to read initially. As usage grows, however, config time slows down as user segmentation, split view and A/B testing are added and communicated. Data needs to be passed for user configuration—you will be transmitting personally identifiable information (PII), which means that additional encryption is required. (Otherwise, you’re putting yourself at risk for litigation. And overall, being careful with people’s data is just the right thing to do.) Most of the time, engineers are tasked with managing the added encryption process.

One way to avoid complications around data privacy is to use an external feature management tool that never touches PII in the first place, such as CloudBees Feature Management. End user data never makes its way back to CloudBees servers, because its architecture specifically prevents it from doing so. This helps CloudBees Feature Management customers more easily comply with enterprise security requirements. Large organizations should already consider these, of course, but smaller startups aiming to get acquired may also want to start making plans to meet them. 

It’s important to think ahead and figure out how your feature flag management tool—self-built or otherwise—will be able to meet enterprise security protocols and standards. 

Imagine that you’re a small startup working to build momentum in the marketplace and eventually get acquired by a larger enterprise. If you choose to rely on a tool that doesn’t conform to these protocols and standards, you’ll have to do significant work during the acquisition to bring your system up to speed. The costs will be much lower if you’re already using a tool that is compliant. (CloudBees Feature Management is SOC2 Type II compliant and fully compliant with GDPR regulations at the application level, and it also meets enterprise standards for security around customer data, instances and networks, physical data centers and EU data protection.)

As you start thinking through how you might begin to scale your feature flag management system, ask yourself questions like:

• How will you hit the right balance of flexibility and security?

• How will you track who makes changes and when? What is the process when a non-developer wants to know this information?

• How will you continue to protect end user data privacy as usage of the tool grows?

• Read how the CloudBees philosophy of “embrace don’t replace” applies to feature flags

• Learn what you need to know to manage the complexity that feature flags bring as you scale—download the whitepaper “Feature Flags Across CI/CD”