Apache Mesos and Jenkins - Elastic Build Slaves

The Mesos project is a cluster management tool for building your own elastic clouds. (see more at mesosphere.io). Here I will take you through setting up Jenkins to make use of Mesos as an elastic cloud of build agents. Mesos is fast changing project and one to watch.

The Mesos project provides an excellent Jenkins plugin to do the heavy lifting for you. Mesos can help with some of the heavy lifting of running a large pool of Jenkins build agents - something that us at CloudBees have a lot of experience with, and can appreciate (and assist with).

Say you have a bunch of hardware - virtual or otherwise - you can run Mesos across it to manage it as a cluster available for all sorts of tasks: applications, build jobs, indexing, and more.

Mesos is a multi-controller (controllers get elected) and multi-agent system - meaning that you set up a few controllers, and many agents - and then you can ask the controllers to provide you with resources - Mesos is then responsible for finding the right capacity for you. Some may think of it as a PaaS - for highly variable workloads.

(Image credit: mesosphere.io)

You can even go so far as to think of it as a toolkit for build a PaaS - which the Typesafe folk have written about here :

(Image credit: typesafe.com)

What is interesting here is that in a Mesos "cloud" - you have lots of slots available - why not use them for build agents as well. Workloads of the type Mesos appeal to are often highly elastic, and there are times where you can have a lot of spare capacity (eg at some times you want to run many apps, or many indexers, or many builds - but hopefully not all at once). Mesos lets you manage this, and make the most out of what hardware you do have.

You can read more about Mesos here .

Mesos uses the concept of "frameworks" for launching apps (a framework is like an app that can be launched on many agent). Jenkins, via the mesos plugin, will register a framework that can then be deployed (giving your controller as many agents as the the mesos cloud is able to) - as mesos agents "offer" to host the framework job.

The theory:

Let's see how it works.

I built a simple cluster to try this out - you can too.

The high level steps for getting going are:

• Install Mesos and Jenkins (I turned the finished product into an image so I could then launch many of them)

• Launch at least one controller and one agent

• Install and configure a Jenkins with the Mesos plugin, connected to a controller

• Run a test build

1. Install Mesos and Jenkins on one server

This is currently the hardest bit, if you are already using it - skip this bit. Mesos is not readily packaged so you will need probably need to build it from source.

Follow the getting started instructions - step by step.

(I used Ubuntu 13.10)

If you are building 0.18.0 - you will need to apply a patch to prevent it from trying (and failing) to unzip the agent.jar. Sorry - this is unpleasant - I am asking you to edit a .cpp file and recompile - I know. You could also get a distro from here - but it may not have the fix to make agent.jar work.

In your build directory there will be a file: src/.libs/libmesos.so - you will need this for when you run the Jenkins controller.

At this point - you have everything setup - and you can snapshot/create an image (eg if you are on ec2) so you can launch it later - if you like.

2. Launch a controller and a agent

On a server with mesos installed, run

./build/bin/mesos-controller.sh --ip=controller_IP --port=8999

I use the specific private IP address I would like it to listen on.

On another server (this will be the agent) - actually it can even be the same server as a controller, if you like:

./build/bin/mesos-agent.sh --controller=controller_IP:8999

At this point - you have a mesos cluster running.

You can even look at the web interface for the mesos controller (http://controller_IP:8999):

You can then see the the Mesos agent attached, on the agent screen:

This is what will actually do the work for Mesos - read on for how to make it do the work of a build agent.

3. Set up Jenkins

As mesos is a moving target - I recommend running the mesos jenkins plugin from the git repo - it will also have updated instructions.

Launch a server - or you can use your controller/agent server above, and run:

• apt-get install git mvn

• git clone https://github.com/jenkinsci/mesos-plugin.git && cd mesos-plugin

• Modify the pom.xml to use the Mesos version that you compiled in the steps above (in my case it was 0.18.0)

• mvn hpi:run

At this point - Jenkins is running on port 8080 - browse to it with a web browser.

Go to the /configure screen - "Add a new cloud" - pick "Mesos"

This will give you the config screen for setting up Mesos agents:

I have highlighted the important bits. Firstly - put in the full path to the mesos client binary - this will be located in build/src/.libs/libmesos.so - where you build Mesos above. This is how the plugin connects to the Mesos cloud.

Secondly - put in the controller IP:PORT - this must be the IP that the controller is listening on.

Finally - note the Label String - this defaults to "mesos" and will be how you tell build jobs to run in mesos, vs elsewhere.

4. Set up and try out a build job

Now - set up a new job in Jenkins. Then on the configure screen, check the box that says "restrict where this build can run"

Put in "mesos" (what was in the label in plugin configuration).

At this point you are good to go. If you check the Mesos console, you should see that Jenkins Scheduler is now setup as a framework - which means it is able to accept jobs:

Finally, joy of joys, you can run the job - and it will run on Mesos. You will see an executor magically appear - and then pause for a little bit (while the agent.jar is setup etc) - and then run the job, this will ask the Mesos controller to find a suitable agent (you can have multiple agent types setup) - and then set it up with the agent.jar - connected to the Jenkins controller - and run the build on the Mesos agent (so whatever tools are available on the agent, or what your build installs, can be used, as normal):

So there you go. Mesos is a fascinating project to watch which can give you, for example, elastic build agents, that can work alongside other jobs and apps running on Mesos.

You can read more about Mesos here and here .