This is part of a series of blog posts in which various CloudBees technical experts have summarized presentations from the Jenkins User Conferences (JUC). This post is written by Harpreet Singh, VP Product Management, CloudBees about a presentation given by Christopher Orr of iosphere GmbH at JUC Berlin .
Git has become the repository of choice for developers everywhere and Jenkins supports git very well. In the talk, Christopher shed light on advanced configuration options for the git plugin. Cloning extremely large repositories is an expensive proposition and he outlined a solution for speeding up builds with large repositories.
Advanced Git Options
The are three main axes for building projects: What, When and How.
What to build:
The refspec option on Jenkins lets you choose on what you need to build. By default, the plugin will build the controller branch, this option can be supplanted by wildcards to build specific features or tags. For example:
_*/feature/* will build a specific feature branch_
_*/tags/beta/* will build a beta version of a specific tag_
_+refs/pull/*:refs/remotes/origin/pull/* will build pull requests from GitHub_
The default strategy is usually to build particular branches. So for example if refspec is */release/* , branches release/1.0, release/2.0 will be built while branches feature/123, bugfix/123 will be ignored. To build feature/123/ and bugfix/123, you can flip this around by choosing the Inverse strategy.
 |
| Choosing the build strategy |
When to build:
Generally, polling should not be used and webhooks are the preferred options when configuring jobs. OTOH, if you have a project that needs to be built nightly only if a commit made it to the repository during the day, it can be easily setup as follows:
 |
How to build:
A git clone operation is performed to clone the repository before building it. The clone operation can be speeded up by using shallow clone (no history is cloned). Furthermore by using the "reference repo" during the clone operation, builds can be speeded up. In the reference repo option, the repository is cloned to a local directory and from there on, this local repository is used for subsequent clone operations. A network access is made only if the repository is unavailable. Ideally, you line these up, so shallow clone for the first clone (fast clone) and reference repo for faster builds subsequently.
 |
| Equivalent to git clone --reference option |
Working with Large Repositories
The iosphere team uses the reference repository approach to speed up builds. They have augmented this approach by inserting a proxy server (git-webhook-proxy [1]) between the actual repo and Jenkins. Thus, a clone happens to this proxy server. The agent setup plugin copies the workspace over to the agents (over NAS) and builds proceed there on. Since network access is restricted to the proxy server and each agent does a local copy, this speeds up builds considerably.
 |
| git-webhook-proxy: to speed up workspace clones |
The git-webhook-proxy option seems a compelling solution, well worth investigating if your team is trying to speed up builds.
-- Harpreet Singh
Harpreet is vice president of product management at CloudBees.