Software Supply Chain Attacks: Who Owns the Risk and What Can Be Done?

This post was co-authored by Tim Johnson, Director, Product Marketing, CloudBees and John Walsh, Senior Product Marketing Manager, DevOps Security, CyberArk.

By the time you hear about a software supply chain attack, it’s a massive problem and public relations nightmare. However, they don’t start out that way. In fact, they almost always begin with a tiny gap, like an exposed Git credential or a misconfigured CI/CD tool earlier in the supply chain. And that’s exactly what cyber adversaries are looking for.

Attackers infected more than 18,000 conscientious SolarWinds customers by injecting malicious code into an otherwise legitimate software update. The April 2021 Codecov breach was traced back to a process error that enabled the bad actors to extract credentials and modify scripts. And in the massive Kaseya ransomware attack, trusted software was compromised to reach into the company’s global customer base.  

These recent headline-grabbing attacks have resulted in significant financial and reputational damage and prompted many organizations to re-examine their own development environments and delivery practices to identify vulnerabilities. And they’re getting to the heart of the issue: unprotected secrets (credentials, SQL/LDAP passwords, SSH keys and API tokens) that provide access to the organization’s most valuable data and resources. A single unprotected access point is all it takes for an attack to rip through an entire IT environment and move downstream to customers and partners.  

Jenkins—plus all the other tools, platforms and containers used by developers to automate software delivery—must be able to interact with numerous systems and applications throughout the DevOps environment to accomplish its role as the “butler.” To do this, it requires privileged access to the powerful secrets to source control and artifact deployment. 

If not configured properly and secured, these secrets become easy—and valuable—targets for attackers. CI/CD pipeline and other DevOps tools are “Tier Zero” assets: access them and you get more credentials. With compromised credentials, attackers can take confidential data, inject malware into the codebase, make significant changes to an application’s functionality or steal code and valuable IP from repositories.

It’s become clear that protecting the access embodied in these secrets and credentials across the pipeline is key to securing the software supply chain. But who ultimately bears this responsibility?

The shift to modern infrastructure and DevOps has created complexity and differences in how security needs are met. Regulations and industry standards such as Sarbanes-Oxley (SOX) require organizations to take ownership of supply chain risk, have specific DevOps security controls in place and provide greater transparency to minimize risk. Meanwhile, customer demands for security assurance, coupled with modern digital experiences, are increasing rapidly.

Security teams are charged with securing digital value chains that protect customer data, starting with who, or what, is allowed to access what within the CI/CD pipeline and production environments, when and for how long. The challenge is finding a way to achieve that goal without slowing developers down. 

Developers face their own security dilemma: with a laser-focus on bringing innovations to market fast, the DevOps culture emphasizes high velocity, intensive sharing of code, ad-hoc tooling and full-on automation. In this culture, low-security shortcuts often flourish, such as putting unprotected secrets in JenkinsFiles to save time or reusing open source code from the internet without sufficient scrutiny. The last thing they want is someone else preventing them from getting their work done, and they don’t really want to have to “own” security, yet they do not want to be held responsible for a costly breach-related outage.  

As a result, some types of DevOps security, vulnerability and code scanning are being integrated into DevOps processes, yet in many organizations, secrets management remains a basic functionality that’s fragmented across individual DevOps tools, making them difficult for developers (who don’t have time to become security experts) to manage and secure. When security issues are discovered—sometimes right before code is scheduled to go into production—developers feel the pain in terms of last-minute code changes and delayed releases.

The evolving development landscape requires security and development to work together to secure secrets across the CI/CD environment to reduce risk and minimize damage from the next supply chain threat. A centralized approach to secrets management makes it easier for developers to maintain their existing workflows, and keep moving fast. 

Most organizations already have established requirements for securing credentials with secrets management in traditional IT systems, such as automatically rotating, centrally storing and continuously monitoring credentials and secrets. However, to protect your software supply chain and secure the CI/CD pipeline, these secrets management best practices also need to be implemented throughout the Jenkins pipeline, within DevOps tools and admin consoles, on developer workstations and in applications and scripts. The most effective strategy follows these steps:

• Enforce the “least privilege principle” to limit secrets exposure. Apply role-based access control (RBAC) so the developer, tool, application, container, script or automation process only has access to the credentials it needs. This, for example, will help prevent untrusted parties from configuring jobs and help ensure valuable credentials, such as deployment tokens, are segregated by appropriate access restrictions or deployed from a second secure Jenkins server. 

• Remove all hard-coded secrets in code, DevOps tools, configuration files and scripts. It’s also important to never use default passwords. For example, some tools establish a developer default user to create projects; other tool consoles can be accessed using http or with the default password.

• Authenticate access to secrets. This extra layer of protection requires attackers to take extra steps beyond stealing a single secret, known as “secret zero.”  

A centralized secrets management platform will make these best practices far easier to implement and manage by giving your organization a complete and continuous picture of “who has access to what.”

While the notion of “shift left” to build safer and more efficient CI/CD practices and processes is widely understood, putting it into practice can be difficult. It’s important for the security team to take the lead in integrating security into the DevOps processes before poor practices become entrenched. And instead of viewing security as the “release prevention department,” developers should make it a practice to bring security in at the planning stage—and keep security tightly integrated in the CI/CD process from end to end.

Here are a few practical steps to making this collaborative approach a reality: 

• Security teams must get up to speed on DevOps tools and techniques to credibly communicate risk and best practices to their development counterparts. And by using agile and DevOps methods within their own security practices, such as automating security tasks and delivering security capabilities in smaller, more frequent increments, they can gain a deeper understanding of DevOps.

• Implement a Security-owned solution that provides critical capabilities, such as secrets management, in machine-consumable ways that can be integrated into automated processes, like security policy as code. This makes it easier for developers to get the access they need to do their jobs, while doing the “right” thing from a security perspective. 

• Teams should jointly consider how best to deploy security resources into existing or new organizational models and structures. For example, establishing centers of excellence, community leaders and security champions, and embedding security team members inside development teams.

• Offer developer training on specific attacker tactics, show how sample code modules could expose secrets and provide examples as user stories. For example, “As an attacker, I would scan the organization’s code repositories looking for secrets.” Take the team through a penetration testing exercise or engage a Red Team to demonstrate how an attacker could compromise a CI/CD pipeline.

With the right tools and strategy in place, security teams can more effectively partner with developers to establish more agile, secure and productive dev environments downstream in your software supply chains.

Tim Johnson is the Director of Product Marketing at CloudBees and focuses on the impact DevOps has on the people and the organizations adopting it. He has more than 15 years of product marketing experience with industry leaders like Electric Cloud, BMC Software, Cisco, Google and SurfControl. He holds an MBA from the University of California, Irvine and is a Scoutmaster and wood turner in his "spare" time.

John Walsh is the Senior Product Marketing manager for DevOps Security at CyberArk focused on open source secrets management, raising developer awareness around application security best practices. John has served the realm as a lord security developer, product manager and open source community manager for more than 15 years, working on cybersecurity products such as Conjur, LDAP, Firewall, JAVA Cryptography, SSH and PrivX. He has a wife, two kids and a small patch of land in the greater Boston area, which makes him ineligible to take the black and join the Knight’s Watch, but he’s still an experienced cybersecurity professional and developer who holds a bachelor's degree in Computer Science and master's degree in Management Information Systems.