This article originally appeared in The New Stack.
Since the news first broke that SolarWinds’ Orion software had been compromised on customer installations, we’ve learned that the culprits had gained access to as many as 18,000 government and private networks.
We’re not here to criticize SolarWinds, or to tell you that our products and services would have prevented the breach. Instead, we want to share three areas where we’ve seen IT leaders overlook the potential for risk. By renewing your attention to them, you can significantly mitigate your own risk as much as possible. Our focus is supply chain integrity—it goes hand in hand with security, before and not after the fact.
1. Put Process Gates and Controls at Every Step of the Software Supply Chain
In SolarWinds’ case, the attacker gained access to one of their GitHub repositories and found a shared secret—perhaps a password—in plain text. In theory no security-conscious engineer would ever put one there, but in practice even the best people make mistakes. Could you have an automated process that checks for things that shouldn’t be in your repository?
Certainly, after the break-in there was a build process where code that had been changed was baked into the product and then shipped to customers. Automated drift detection at every step would alert engineers to changes made to their code.
Yes, we mean every single part of the pipeline
Our experience has been that even good IT leaders often focus on the most likely points of compromise, but not the entire ecosystem. What is the map of every point in the pipeline from start to finish? What is the integrity of the pipeline itself? Can bad actors or honest errors somehow impact its integrity—either intentionally, through fat-fingering or through misaligned good intentions? SolarWinds’ culprit showed that one overlooked part of the pipeline is all that a dedicated hacker may need.
Process steps and gates—which must be enforced every time—across your entire pipeline will minimize the opportunities for a bad actor to slip in malicious code. Whether you enforce them as part of CI or CD is up to you. But any step in the software build and delivery pipeline that doesn’t have automated inspection and controls to prevent bad code from going further is a point of vulnerability.
Automate and rehearse for the inevitable
While we want to focus on stopping attacks before they happen, it’s important to have processes in place to quickly mitigate any that slip through. For ourselves, we’ve added a metric in addition to the standard DORA set called Mean Time To Mitigate. It measures the length of time from discovery of a problem to its neutralization.
CloudBees provides feature flagging capabilities, which allow engineers to instantly turn off specific sections of code for customers without issuing a new download. If you’re not using feature flags, that’s OK, but do you have rehearsed and automated rollbacks? Test those processes ahead of time to reduce your mean time to mitigate, as well as reduce the chance of cascading errors that make a bad situation worse.
2. Don’t assume bad actors are only outside your organization
An actor who corrupts your software supply chain needn’t be malicious. Whoever put shared secrets in clear text in SolarWinds’ repository didn’t mean to prop the door open for attackers, but the end result was, unfortunately, the same.
Your entire process has to be secure, even from accidents and mistakes. Again, look through your entire pipeline. Who has access to it? Do you have a repository for immutable objects? If someone tries to go off-page with one of those immutable objects or to inject a phony one, do you have a way to detect that and filter it out?
At every step in the pipeline, you need to know who has access, who has privilege and who has availability. And you must have the ability to detect, approve and track any changes made by any user.
3. Think Like an Auditor
Many of SolarWinds customers were federal clients with a months-long audit process for software updates. Nonetheless, that process didn’t catch the injected code or the unauthorized access it provided once installed.
Meeting the auditors’ requirements isn’t enough. You need to think two steps ahead of them and consider what they are looking for in the first place: vulnerabilities, unauthorized changes and your response to the unexpected. More than just passing the test, you want to controller the material.
Review your automation practices - these should serve as the blueprint of your understanding of the process. How frequently do you review the assumptions that went into that automation? What part of it is still relevant or necessary? What changes to applications, environments and technology have happened since the last review? When something goes wrong, how do we deal with it, document it and make improvements?
The primary benefit of automation is consistent, repeatable outcomes for similar actions that are automatically logged and documented. It also comes with the extra benefit of the irrefutable proof that what was done is what was promised to be done – the foundation of compliance. Since DevOps spans the entire pipeline, it provides traceability from code change to release, making auditing much easier. Compliance reporting becomes a one-click effort, eliminating manual intervention or hours spent backtracking manual processes and actions.
The software supply chain and its integrity are inextricably intertwined with security. We’ve identified seven strategies for minimizing risks to that integrity that apply to any organization for which governance, security and compliance is a top business challenge. Learn more here.