Introduction

This post is about the approach that the CloudBees DevOptics team uses to get Continuous Delivery / Deployment using Apache Maven .

If reading blog posts is not your thing, I have recorded a video on the same topic:

This is not the first post of mine on this topic, you may be interested in some of the background leading up to this post:

• A New Way to Do Continuous Delivery with Maven and Jenkins Pipeline
• A year of Continuous Deployment: Lessons learned (also available as a video )
• Our Internal Version Numbering Scheme for DevOptics (also available as a video )

Continuous Delivery/Deployment 

If you are like me, you probably get confused about the difference between Continuous Delivery and Continuous Deployment as well as how they relate to Continuous Integration. The current conventions give us these approximate definitions:

• Continuous Integration -  Every commit gets tested
• Continuous Delivery  - Every commit gets tested and, if successful , is turned into a release that may be deployed to production
• Continuous Deployment - Every commit gets tested and, if successful , is turned into a release that always gets deployed to production

Delivery leaves deployment as an optional step, but ideally the deployment itself should also be automated.

NOTE: I should mention that the above definition loses an important aspect of the original idea of Continuous Integration, namely that you would not just test the commit but the commit integrated with all the latest versions of the other things. For example, if you have multiple services in multiple repositories, you will see people say they have Continuous Integration where they just check each repositories commits in isolation. For this to really be Continuous Integration, you need to also verify the commits when interacting with the latest commits of all the sibling services. 

Apache Maven Release Plugin

If you use Apache Maven, you will likely have met the Maven Release Plugin . This is not so much of a plugin as a toolkit for building plugins that release and a sample plugin that matches the needs and requirements of the Apache Maven project for releasing. If you have different requirements the Apache Maven project expects that you would create your own release plugin that uses the Maven Release API to complete your needs.

When using the Maven Release plugin for Continuous Delivery / Deployment there are four main problems that you hit:

• Two commits for every release

  [maven-release-plugin] prepare release …

  [maven-release-plugin] prepare for next development iteration

• pom.xml gets modified all the time generating history noise and merge conflicts
• CI/CD server needs to ignore commits with [maven-release-plugin] to prevent forever loops
• Every time you build and test twice

Very often the immediate response for most people is to throw out the Maven Release Plugin. This may or may not be the correct choice, depending on what you need from a release, but I’d like to avoid throwing the baby out with the bath water

Attempted solution

One of the things that the Apache Maven Release plugin does is to store the next version number in the pom.xml . The version number itself marks each build as being a snapshot of the development of that next version. In other words a version like 1.56.2-SNAPSHOT is Maven’s way of saying “this will be released as 1.56.2 , but this is not the final 1.56.2 release”. Because the plugin is storing the version in the pom.xml , every release needs to update the pom.xml twice. First to remove the -SNAPSHOT and then to advance the version number and add back in the -SNAPSHOT .

But what would happen if we didn’t actually change the development version after a release. Nobody says we cannot run our releases like:

1-SNAPSHOT ⇒ 1.56 ⇒ 1-SNAPSHOT

or

1.x-SNAPSHOT ⇒ 1.56 ⇒ 1.x-SNAPSHOT

If we kept the development version as a constant then the merge conflicts would be greatly reduced as a diff against the pom.xml from before the release would be equally valid after the release.

There are problems with this attempted solution:

• It doesn’t remove the noise of those two commits per release. The diff just cancel each other out.
• The merge conflicts will resurface when using git rebase  - not all the time but they will be annoying
• The pom.xml now has an additional source of variability making it hard to use tools such as git blame
• We have to determine the version number for each release somehow as it is no longer stored in the pom.xml

Solution

Who says we have to ever push those [maven-release-plugin] prepare … commits, what would happen if we only pushed the tags?

The pom.xml versions stay the same because they are the same:

• Merge conflicts resulting from the Maven Release Plugin commits are eliminated - because the master branch never sees those commits
• Tags reflect releases

In the DevOptics project repositories we derive the version number from the number of commits on the master which prevents these issues.

DevOptics’ implementation

Our Jenkinsfile looks a little like this (distracting noise removed)

pipeline {
  stages {
    stage('Build') {
      when {
        not { branch 'master' }
      }
      steps {
        withMaven(maven:env.MAVEN_TOOL_ID,
            globalMavenSettingsConfig: 'maven-settings-nexus-internal') {
          sh "mvn verify"
        }
      }
    }
    stage('Release') {
      when {
        branch 'master'
      }
      environment {
        RELEASE_VERSION = getReleaseVersion()
      }
      steps {
        withMaven(maven:env.MAVEN_TOOL_ID,
            globalMavenSettingsConfig: 'maven-settings-nexus-internal') {
          sh "mvn release:prepare release:perform -DreleaseVersion=${RELEASE_VERSION}"
        }
      }
      post {
        success {
          sshagent(['github-ssh']) {
            sh "git push origin devoptics-platform-parent-${RELEASE_VERSION}"          }
        }
      }
    }
  }
}

•  Normal build for everything except master branch
• The master branch is always a release
• We only push the tags if the release passes integrations tests when deployed to a dedicated integration testing environment.

The getReleaseVersion() is a custom pipeline step that determines the version to release using the following steps:

1. Counts number of commits on master branch
2. Lists tags in Git
3. Lists display names of jobs
4. Picks next version number not already used as Git tag or Job display name

Delivery vs Deployment

We do both Continuous Delivery and Continuous Deployment

For the DevOptics Jenkins Plugin, we use a Continuous Delivery model:

The Jenkinsfile sets the build description and build status to include the downstream test results as well as the Sonatype Nexus Staging repository ID. When we need to release a version we can see the version status in Jenkins and we know what staging repository to release directly on the job screen.

For the DevOptics SaaS back-ends we use a Continuous Deployment model:

Here we deploy successful builds all the way to production (verifying against each environment: integration, staging, production before proceeding to the next) 

One key point I would make is that you need to use the right deployment model for each codebase. Some codebases are better targeting the Continuous Delivery style, while others can go all the way to Continuous Deployment.

Example:

In our case the Jenkins plugin is consumed by our end users, and we consequently have no control over when they decide to upgrade, so we have to accept a range of versions of the plugin talking to our back-end anyway. Additionally, some of our end-users have complex change control processes that regulatory bodies have mandated, so we cannot keep requesting them to update the plugin for every commit. Those two factors drove our decision to use Continuous Delivery for the DevOptics Jenkins Plugin. 

Try it yourself

If you find this approach interesting, you may want to try it out for yourself. To make this easier I have created a Maven Plugin to assist and a sample Git repository with a project set-up to work using this technique.

The sample git repository is available from our “unofficial sample code” GitHub Organisation… which has the strangely similar to CloudBees logo… though perhaps being more about enjoying a beer with the example!

GitHub Repository: https://github.com/cloudbeers/maven-continuous

If you want to take this example and apply it to your own project, here’s a summary of the necessary changes:

• Change the project version to 1.x-SNAPSHOT .

  <?xml version="1.0" encoding="UTF-8"?>
  <project xmlns="http://maven.apache.org/POM/4.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                   http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    ...
    <groupId>...</groupId>
    <artifactId>...</artifactId>
    <version>1.x-SNAPSHOT</version>
    <packaging>...</packaging>
    ...
  </project>

  This is not strictly necessary, but I prefer to indicate that the real version will be two components rather than have the version as 1-SNAPSHOT
 
• Release plugin: disable pushing back to origin

  <plugin>
    <artifactId>maven-release-plugin</artifactId>
    <configuration>
      ...
      <localCheckout>true</localCheckout>
      <pushChanges>false</pushChanges>
      ...
    </configuration>
  </plugin>

  The critical change is <pushChanges>false</pushChanges> which stops the changes being pushed back to the origin, however once we make that change we will hit issues when running the release:perform goal as it tries to checkout the remote tag that doesn’t exist yet, hence the need to also specify <localCheckout>true</localCheckout> .
   
• Build once only because bad builds will now go nowhere

  <plugin>
    <artifactId>maven-release-plugin</artifactId>
    <configuration>
      ...
      <preparationGoals>validate</preparationGoals>
      ...
    </configuration>
  </plugin>

  Normally the release plugin will run two builds, the first build is to establish if the code to be tagged will actually build, then the second build checks out the tag into a clean working copy and runs the release build. Because we are using Jenkins to run the build, we can have Jenkins clean the workspace beforehand which eliminates one of the reasons for the two builds and because we hold back the tag and artifacts until the release is confirmed OK, we can actually tolerate a bad build. By setting the <preparationGoals>validate</preparationGoals> we effectively bypass the first build and make releases time competitive with a plain build.
   
• My Git timestamp plugin (or roll your own helper plugin): select the version to release

  <plugin>
    <groupId>com.github.stephenc.continuous</groupId>
    <artifactId>git-timestamp-maven-plugin</artifactId>
    <version>1.40</version>
    <configuration>
      <snapshotText>x-SNAPSHOT</snapshotText>
      <releaseVersionFile>VERSION.txt</releaseVersionFile>
      <tagNameFile>TAG_NAME.txt</tagNameFile>
    </configuration>
  </plugin>

  My plugin will do the following
  • Counts commits on current branch
  • Determines candidate version number
  • Checks with Git Repo if tag name already used
  • Advances version number until unused tag name found
  • Sets up maven properties for release:prepare
  In this case we are requesting that the x-SNAPSHOT text be replaced by the commit count and that the resulting version will also be written to the VERSION.txt file and that the tag name will be written to the TAG_NAME.txt file. These two files will be read by the Jenkinsfile when it sets the build display name to the version number and when it needs to know the tag name to push back to GitHub.

At this point we have the Maven project set up to run this style of Continuous Delivery. We can test it using just the CLI if we want to:

1. Start with a clean checkout in a throw-away directory:

   $ cd $TMPDIR
   $ git clone git@github.com:cloudbeers/maven-continuous.git throwaway
   $ cd throwaway

2. Now we can run the release using Apache Maven:

   $ mvn git-timestamp:setup-release release:prepare release:perform

   The release will not go anywhere
    
3. Once we have verified that everything works as expected we can throw away that throw-away directory.

   $ cd ..
   $ rm -rvf throwaway

Ok, so the final step is to automate all this with Jenkins:

• Set the Jenkinsfile to run a normal build for all branches except master:

  stage('Build') {
    when {
      not { branch 'master' }
    }
    steps {
      withMaven(maven:'maven-3', jdk:'java-8', mavenLocalRepo: '.repository') {
        sh 'mvn verify'
      }
    }
  }

• When on master cut a release

  stage('Release') {
    when {
      branch 'master'
    }
    steps {
      withMaven(maven:'maven-3', jdk:'java-8', mavenLocalRepo: '.repository') {
        sh 'mvn release:clean git-timestamp:setup-release release:prepare release:perform'
      }
    }

• After release verified, push the tags if good or delete local tag if bad

  post {
    success {
      sshagent(['github-ssh']) {
        sh 'git push git@github.com:cloudbeers/maven-continuous.git $(cat TAG_NAME.txt)'
      }
    }
    failure {
      sh 'test -f TAG_NAME.txt && git tag -d $(cat TAG_NAME.txt) && rm -f TAG_NAME.txt || true'
    }
  }

• In GitHub, create a deploy key for the Jenkins job (alternatively if you are setting up lots of these, you may want to create a GitHub user that is dedicated for the Jenkins server)
  
• In Jenkins, add the deployment key as folder scoped credentials limited to the project repository
  

End result

Now every time there is a commit to the master branch, such as a Pull Request being merged, this will trigger a release build. The above screenshot shows the Jenkins view while here’s the Git view:

We have 6 builds, the first three were 1.1 , 1.2 and 1.3 . Then I triggered a new build of the code in 1.3 so this produced version 1.3.1 then for the final version I pushed two commits at the same time which is why the version jumped from 1.4 to 1.6 .

Summary

Hopefully this post has inspired you to think about how you could enable Continuous Delivery / Deployment in your codebases.

Stephen Connolly has over 25 years experience in software development. He is involved in a number of open source projects, including Jenkins . Stephen was one of the first non-Sun committers to the Jenkins project and developed the weather icons. Stephen lives in Dublin, Ireland - where the weather icons are particularly useful. Follow Stephen on Twitter , GitHub and on his blog .