Browser-testing with Sauce OnDemand and Pipeline

Testing web applications across multiple browsers on different platforms can be challenging even for smaller applications. With Jenkins and the Sauce OnDemand Plugin, you can wrangle that complexity by defining your Pipeline as Code.

Pipeline ♥ UI Testing, Too

I recently started looking for a way to do browser UI testing for an open-source JavaScript project to which I contribute. The project is targeted primarily at Node.js but we're committed to maintaining browser-client compatibility as well. That means we should run tests on a matrix of browsers. Sauce Labs has an "open-sauce" program that provides free test instances to open-source projects. I decided to try using the Sauce OnDemand Plugin and Nightwatch.js to run Selenium tests on a sample project first, before trying a full-blown suite of tests.

Starting from Framework

I started off by following Sauce Labs' instructions on "Setting up Sauce Labs with Jenkins" as far as I could. I installed the JUnit and Sauce OnDemand plugins, created an account with Sauce Labs, and added my Sauce Labs credentials to Jenkins. From there I started to get a little lost. I'm new to Selenium and I had trouble understanding how to translate the instructions to my situation. I needed a working example that I could play with.

Happily, there's a whole range of sample projects in "saucelabs-sample-test-frameworks" on GitHub, which show how to integrate Sauce Labs with various test frameworks, including Nightwatch.js. I forked the Nightwatch.js sample to bitwiseman/JS-Nightwatch.js and set to writing my Jenkinsfile. Between the sample and the Sauce Labs instructions, I was able to write a pipeline that ran five tests on one browser via Sauce Connect:

node {
    stage "Build"
    checkout scm

    sh 'npm install' // <1>

    stage "Test"
    sauce('f0a6b8ad-ce30-4cba-bf9a-95afbc470a8a') { // <2>
        sauceconnect(options: '', useGeneratedTunnelIdentifier: false, verboseLogging: false) { // <3>
            sh './node_modules/.bin/nightwatch -e chrome --test tests/guineaPig.js || true' // <4>
            junit 'reports/**' // <5>
            step([$class: 'SauceOnDemandTestPublisher']) // <6>
        }
    }
}

• 1: Install dependencies
• 2: Use my https://wiki.saucelabs.com/display/DOCS/Installing+and+Configuring+the+Sauce+OnDemand+Plugin+for+Jenkins[previously added sauce credentials]
• 3: Start up the Sauce Connect tunnel to Sauce Labs
• 4: Run Nightwatch.js
• 5: Use JUnit to track results and show a trend graph
• 6: Link result details from Sauce Labs

NOTE: This pipeline expects to be run from a Jenkinsfile in SCM. To copy and paste it directly into a Jenkins Pipeline job, replace the checkout scm step with git url:'https://github.com/bitwiseman/JS-Nightwatch.js', branch: 'sauce-pipeline'.

I ran this job a few times to get the JUnit report to show a trend graph.

This sample app generates the SauceOnDemandSessionID for each test, enabling the Jenkins Sauce OnDemand Plugin's result publisher to link results to details Sauce Labs captured during the run.

Adding Platforms

Next I wanted to add a few more platforms to my matrix. This would require changing both the test framework configuration and the pipeline. I'd need to add new named combinations of platform, browser, and browser version (called "environments") to the Nightwatch.js configuration file, and modify the pipeline to run tests in those new environments.

This is a perfect example of the power of pipeline as code. If I were working with a separately configured pipeline, I'd have to make the change to the test framework, then change the pipeline manually. With my pipeline checked in as code, I could change both in one commit, preventing errors resulting from pipeline configurations going out of sync from the rest of the project.

I added three new environments to nightwatch.json:

"test_settings" : {
  "default": { /*---8<---8<---8<---*/ },
  "chrome": { /*---8<---8<---8<---*/ },

  "firefox": {
    "desiredCapabilities": {
      "platform": "linux",
      "browserName": "firefox",
      "version": "latest"
    }
  },
  "ie": {
    "desiredCapabilities": {
      "platform": "Windows 10",
      "browserName": "internet explorer",
      "version": "latest"
    }
  },
  "edge": {
    "desiredCapabilities": {
      "platform": "Windows 10",
      "browserName": "MicrosoftEdge",
      "version": "latest"
    }
  }
}

And I modified my Jenkinsfile to call them:

//---8<---8<---8<---8<---8<---8<---
sauceconnect(options: '', useGeneratedTunnelIdentifier: false, verboseLogging: false) {
    def configs = [ // <1>
        'chrome',
        'firefox',
        'ie',
        'edge'
    ].join(',')
    // Run selenium tests using Nightwatch.js
    sh "./node_modules/.bin/nightwatch -e ${configs} --test tests/guineaPig.js" // <2>
} //---8<---8<---8<---8<---8<---8<---

• 1: Using an array to improve readability and make it easy to add more platforms later.
• 2: Changed from single-quoted string to double-quoted to support variable substitution.

WARNING: Test frameworks have bugs too. Nightwatch.js (v0.9.8) generates incomplete JUnit files, reporting results without enough information in them to distinguish between platforms. I implemented a fix for it and submitted a PR to Nightwatch.js. This blog shows output with that fix applied locally.

As expected, Jenkins picked up the new pipeline and ran Nightwatch.js on four platforms. Sauce Labs of course recorded the results and correctly linked them into this build. Nightwatch.js was already configured to use multiple worker threads to run tests against those platforms in parallel, and my Sauce Labs account supported running them all at the same time, letting me cover four configurations in less that twice the time, and that added time was most due to individual new environments taking longer to complete. When I move to the actual project, this will let me run broad acceptance passes quickly.

Conclusion: To Awesome and Beyond

Considering the complexity of the system, I was impressed with how easy it was to integrate Jenkins with Sauce OnDemand to start testing on multiple browsers. The plugin worked flawlessly with Jenkins Pipeline. I went ahead and ran some additional tests to show that failure reporting also behaved as expected.

//---8<---8<---8<---8<---8<---8<---
    sh "./node_modules/.bin/nightwatch -e ${configs}" // <1>
//---8<---8<---8<---8<---8<---8<---

• 1: Removed --test filter to run all tests

Epilogue: Pipeline vs. Freestyle

Just for comparison here's the final state of this job in Freestyle UI versus fully-commented pipeline code:

NOTE: This includes the AnsiColor Plugin to support Nightwatch.js' default ANSI color output.

Freestyle

Pipeline

node {
    stage "Build"
    checkout scm

    // Install dependencies
    sh 'npm install'

    stage "Test"

    // Add sauce credentials
    sauce('f0a6b8ad-ce30-4cba-bf9a-95afbc470a8a') {
        // Start sauce connect
        sauceconnect(options: '', useGeneratedTunnelIdentifier: false, verboseLogging: false) {

            // List of browser configs we'll be testing against.
            def platform_configs = [
                'chrome',
                'firefox',
                'ie',
                'edge'
            ].join(',')

            // Nightwatch.js supports color ouput, so wrap this step for ansi color
            wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'XTerm']) {

                // Run selenium tests using Nightwatch.js
                // Ignore error codes. The junit publisher will cover setting build status.
                sh "./node_modules/.bin/nightwatch -e ${platform_configs} || true"
            }

            junit 'reports/**'

            step([$class: 'SauceOnDemandTestPublisher'])
        }
    }
}

NOTE: This pipeline expects to be run from a Jenkinsfile in SCM. To copy and paste it directly into a Jenkins Pipeline job, replace the checkout scm step with git url:'https://github.com/bitwiseman/JS-Nightwatch.js', branch: 'sauce-pipeline'.

Not only is the pipeline as code more compact, it also allows for comments to further clarify what is being done. And as I noted earlier, changes to this pipeline code are committed the same as changes to the rest of the project, keeping everything synchronized, reviewable, and testable at any commit. In fact, you can view the full set of commits for this blog post in the sauce-pipeline branch of the bitwiseman/JS-Nightwatch.js repository.

Links

• Sauce OnDemand Plugin
• bitwiseman/JS-Nightwatch.js
• saucelabs-sample-test-frameworks

This is the fourth in a series of posts showing ways to use Jenkins Pipeline. Follow along with this entire series through the links below:

• Faster Pipelines with the Parallel Test Executor Plugin
• Publishing HTML Reports in Pipeline
• Sending Notifications in Pipeline
• Browser-testing with Sauce OnDemand and Pipeline
• Continuous Security for Rails apps with Pipeline and Brakeman
• xUnit and Pipeline