Jenkins Workflow - Using the Global Library to Implement a Reusable Function to Call a Secured HTTP Endpoint

Jenkins Workflow - Using the Global Library to implement a reusable function to call a secured HTTP endpoint

This blog post will demonstrate how to access an external system using HTTP with that is protected by a session based logon. This type of integration is often required to retrieve information to be used in the workflow, or to trigger events on existing external systems, such as a deployment framework. These types of systems often have some form of authentication mechanism - it may be Basic Auth-based or, using some form of session-based security. This blog post will show how you can use the following features:

• Jenkins Workflow plugin

• Git-based Workflow Global Library repository

• curl cookie handling

1. JDK 1.7+

2. Jenkins 1.609+

3. An external system secured with session based authentication - (in this example I use a Jenkins server with security enabled as an example - although it is not recommended you do this for real as Jenkins has better token based APIs to use)

1. Download and install JDK 1.7 or higher

2. Download and install Jenkins

3. Start Jenkins

• Update plugins - Make sure you have the latest Workflow plugins by going to Manage Jenkins –> Manage Plugins -> Updates and selecting any Workflow-related updates. Restart Jenkins after the updates are complete. As of this writing the latest version is 1.8

• Global libraries repo - Jenkins exposes a Git repository for hosting global libraries meant to be reused across multiple CD pipelines managed on the controller. We will setup this repository so you can build on it to create your own custom libraries. If this is a fresh Jenkins Install and you haven’t setup this git repository, follow these instructions to setup.

Important - Before proceeding to the next steps, make sure your Jenkins instance is running

See Workflow Global Library for details on how to set up the shared library - note that if security is enabled, the ssh format works best.

If using ssh ensure that your public key has been configured in Jenkins.

To initialise the git repository:

git clone ssh://<USER>@<JENKINS_HOST>:<SSH_PORT>/workflowLibs.git

Where

• USER is a user a valid user that can authenticate

• JENKINS_HOST is the DNS name of the Jenkins server you will be running the workflow on. If running on the CloudBees Jenkins platform, this is the relevant Client controller not the Jenkins Operations Center node.

• SSH_PORT is the ssh port defined in Jenkins configuration:

Note the repository is initially empty.

To set things up after cloning, start with:

git checkout -b master

Now you may add and commit files normally. For your first push to Jenkins you will need to set up a tracking branch:

git push --set-upstream origin master

Thereafter it should suffice to run:

git push

Create the groovy script in the workflow library:

cd workflowLibs 
mkdir -p src/net/harniman/workflow/jenkins 
curl -O \ https://gist.githubusercontent.com/harniman/8f1418af794d26035171/raw/941c86041adf3e9c9bfcffaf9e650e3365b9ee55/Client.groovy 
git add * 
git commit 
git push

This will make the following script

   net.harniman.workflow.jenkins.Client

available for use by workflow scripts using this syntax:

    def jc = new net.harniman.workflow.jenkins.Client()

and methods accessed using:

     def response=jc.test(<host>, <user>, <pass>, <cmd>)

Note:

• We must NOT define an enclosing class for the script as we want to call step functions such as sh - see https://github.com/jenkinsci/workflow-plugin/tree/controller/cps-global-lib for further details on this restriction.

• scripts follow the usual Groovy package naming formats and thus need to be in the appropriate directory structure

• it is unnecessary to go out of process to access this Jenkins instance - it can be accessed from Groovy via the Jenkins model, however, this is to show how a client for form based access could be built

• The below script is built to access Jenkins as an example for demonstration

This is the actual contents of Client.groovy:

package net.harniman.workflow.jenkins 
String test (host, user, pass, cmd) {
     node("shared") {
         init="curl -s -c .cookies ${host}"
         userAttr="j_username=${user}"
         passAttr="j_password=${pass}"
     jsonAttr="json=%7B%22j_username%22%3A+%22${user}%22%2C+%22j_password%22%3A+%22$
{pass}%22%2C+%22remember_me%22%3A+false%2C+%22from%22%3A+%22%2F%22%7D"
         login="curl -i -s -b .cookies -c .cookies -d $userAttr -d $passAttr -d 'from=%2F' -d $jsonAttr -d 'Submit=log+in' $host/j_acegi_security_check"
         cmd="curl -L -s -b .cookies -c .cookies $host/$cmd"
         echo "Initialising HTTP Connection"
         sh "${init} > .init 2>&1"
         echo "Performing Login"         
         sh "${login} > .login 2>&1"
         def loginresponse = readFile '.login'
         if (loginresponse =~ /Location:.*loginError/) {
             echo "Error loging in"
             error"Unable to login. Response = $loginresponse"
         }
         echo "Invoking command"
         sh "${cmd} > .output 2>&1"
         def output = readFile '.output'
         sh "rm .init .login .output"
         return output
     }
 }

Create a new workflow job with the Workflow Definition as follows:

def jc = new net.harniman.workflow.jenkins.Client() 
def response=jc.test("http://localhost:8080”, "annie.admin", "password", "whoAmI") 
echo "=======================" 
echo response

<span style="font-family: sans-serif, Arial, Verdana, &#x27;Trebuchet MS&#x27;; line-height: 1.6;">Ensure you substitute in real values for:</span> 

• <host>

• <user>

• <pass>

• <cmd>

You can substitute in the required cmd for the query - for instance, it could be <jobname>/config.xml to retrieve a job’s config.

Time to check that it all works, so go ahead and trigger the build.

If it works successfully you should see the raw body of the response printed in the console log.

If you look closely you can see in this example, the username and password is output to the console. This is not recommended, and you can avoid this by integrating with credentials and having the workflow pass in the credentials ID to be used. See this article for more details on the syntax with a sh step using curl .

The Workflow Global Library provides the ability to share common libraries (groovy scripts) across many workflow jobs to help keep workflows DRY .

In the original version of this post, we used the Groovy built in HTTP libraries to make the HTTP call. On further investigation and testing of failure scenarios we found this to be brittle when the controller restarts, The underlying reason for this behaviour is that ALL the workflow logic runs within the process space of the controller, this includes any logic operating within a node(){} block.

To ensure survivability of the workflow that is executing during a controller restart, it is necessary to perform the bulk of the work inside workflow steps . These are designed to be passed to a separate executor for execution rather than executed within the controller’s process space. We have therefore moved the HTTP call, which may take an amount of time to run, to be called within a sh step using curl .

We have still enabled the packaging of this into a shared library module, thus simplifying the workflow script and promoting re-use. It is necessary to ensure subsequent curl requests share the cookiejar. Should you want to leverage the power of groovy to perform such a call, this should be done by calling a separate groovy script from an sh or bat step.

Jenkins Workflow - Getting Started by Udaypal Aarkoti

​Nigel Harniman
​Senior Solutions Architect
CloudBees