Multi-tenancy with Jenkins

As your Jenkins use increases, you will likely extend your Jenkins environment to new team members and perhaps to new teams or departments altogether. It's quite a common trend, for example, to begin using Jenkins within a development team then extend Jenkins to a quality assurance team for automating tests for the applications built by the development teams. Or perhaps your company is already using Jenkins and your team (a DevOps or shared tooling kind of team) has a mission to implement Jenkins as a shared offering for a larger number of teams.

Regardless, the expansion is an indication your teams are automating more of their development process, which is a good sign. It should go without saying, organizations are seeing a lot of success automating their development tool chain with Jenkins, allowing their teams to focus on higher value, innovative work and reducing time wasted on mundane tasks.

No one wants this, after all (no dev managers or scrum controllers, anyway):

-xkcd.com/303/

At the same time, if not planned, the expansion which was meant to extend those successes to more teams could have unintended consequences and lead to bottlenecks, downtime, and pain. Besides avoiding the pain, there are also proactive steps you can take to further increase your efficiency along the way.

For the purposes of this blog post, let's define multi-tenancy for Jenkins: multi-tenancy with Jenkins means supporting multiple users, teams, or organizations within the same Jenkins environment and partitioning the environment accordingly.

You might ask --- "Jenkins is pretty easy to get up-and-running; why not just create a new Jenkins instance?" To some extent, I agree! Jenkins is as simple as java -jar jenkins.war , right? This may be true, but many teams are connected in one way or another… if two related but distinct teams or departments work on the related components, it's ideal that they have access to the same Jenkins data.

Implementing Jenkins - at least, implementing it well - takes some forethought. While it is indeed easy to spin up a new Jenkins instance, if your existing team using Jenkins already has a great monitoring strategy in place or a well-managed set of agent nodes attached to their Jenkins instance, reusing a well-managed Jenkins instance seems like a good place to start. I mean, who wants to wear a pager on the weekend for Jenkins, anyway?

Establishing an efficient strategy for Jenkins re-use in an organization can help reduce costs, increase utilization, enhance security, and ensure auditability/traceability/governance within the environment.

As you begin to scale your Jenkins use, there are a number of existing features available to help:

• Views

  • The views feature in the Jenkins core allows you to customize the lists of plugins and tabs on the home screen for better user experience when using a multi-tenant Jenkins instance.

• Folders

  • The Folders plugin, developed in-house at CloudBees, is even more powerful than views for optimizing your Jenkins environment for multi-tenancy. Unlike views, Folders actually create a new context for Jenkins.

  • This new context allows for example, creating folder-specific environment variables. From the documentation:"You can [also] create an arbitrary level of nested folders. Folders are namespace aware, so Job A in Folder A is logically different than Job A in Folder B".

• Distributed Builds

  • If you're not already using Jenkins distributed builds , you should be! With distributed builds, Jenkins can execute build jobs on remote machines (agent nodes) to preserve the performance of the Jenkins web app itself.

  • If you extend your Jenkins environment to additional teams, all the more reason to focus on preserving the controller's performance.

  • Even better, distributed builds allow you to set up build nodes capable of building the various types of applications your distributed teams will likely require (Java, .NET, iOS, etc.)

• Cleaning Up Jobs

  • When the Jenkins environment is shared, system cleanup tasks become more critical.

  • Discarding Old Builds and setting reasonable timeouts for builds will help ensure your build resources are available to your teams.

• Credentials API

  • Jenkins allows managing and sharing credentials across jobs and nodes. Credentials can be set up and secured at the Folder level, allowing team-specific security settings and data.

As you scale your Jenkins use, you will find there are some stress points where it can be... less than ideal to share a single Jenkins controller across teams:

• Global configuration for plugins

  • Some plugins support only global configuration. For example, the Maven plugin's build step default options are global. Similarly, the Subversion SCM plugin's version configuration is a global setting.

  • If two teams want to use the same plugin differently, there aren't many options (even worse: different versions of the same plugin).

• Plugin Installation and Upgrades

  • While Jenkins allows plugins to be installed without a restart, some plugins do require a restart on install. Further, all plugins require a Jenkins restart on update.

  • Some plugins have known performance, backward compatibility, and security limitations. These may be acceptable for one team, but perhaps not all your users.

• Agent Re-use

  • When multiple-teams use the same agents, they usually share access to them. As mentioned above, care must be taken to clean up agent nodes after executing jobs.

  • Securing access for sensitive jobs or data in the workspace is a challenge.

• Scale

  • Like any software application, a single Jenkins controller can only support so many builds and job configurations.

  • While determining an actual maximum configuration is heavily environment-specific (available system resources, number and nature of jobs, etc.), Jenkins tends to perform best with no more than 100-150 active, configured executors.

  • While we've seen some Jenkins instances with 30,000+ job configurations, Jenkins will need more resources and start-up times will increase as the job count increases.

• Single Point of Failure

  • If more and more teams use the same Jenkins instance, when outages occur, the impact becomes larger.

  • When Jenkins need to be restarted for plugin updates or core upgrades, more teams will be impacted.

  • As teams rely more and more on Jenkins, particularly for automating processes beyond development (e.g.: QA, security, and performance test automation), down time for Jenkins becomes less acceptable.

Hopefully this article saves you some time by laying out some of the stress points you'll encounter when setting up multi-tenancy in Jenkins. Eventually, you'll get to a tipping point where running a single, large multi-tenant Jenkins controller may not be worth it. For that reason, we recommend developing a strategy for taking your multi-tenancy strategy to the next level: creating multiple Jenkins controllers.

For each organization, the answer is a little different, but CloudBees recommends establishing a process for creating multiple Jenkins controllers. In a follow-up post, we'll highlight how CloudBees Jenkins Platform helps manage multiple Jenkins controllers. With CloudBees Jenkins Operations Center, your multi-tenancy strategy is simply expanded to controllers as well, making your Jenkins controllers part of the same Jenkins Platform . We'll also share some successful strategies (and some not so successful strategies) for determining when to split your controllers.

Andy Pemberton
Sr. Director, Solution Architecture
CloudBees