Add Post-Deploy Smoke Tests to Any Codeship Pipeline

One of the biggest goals of a continuous development workflow is to find and fix bugs earlier in the development cycle. Automatically smoke testing every deployment helps developers quickly validate new changes and catch bugs before they make it to production.

Codeship's deployment pipelines and Assertible's web app testing tools make it easy to add an extra layer of automated testing after every build.

In this post, I'll walk through setting up automated post-deployment smoke tests in a Codeship pipeline using Assertible. I've put together an example Ruby app that is deployed to staging and production environments on Heroku and validated with Assertible.

Codeship is a continuous integration and delivery service that allows you to build, test, and deploy code after every commit. Codeship makes it easy to get started by removing the need for yaml configurations and complex scripts. This, combined with the Assertible deployments API, gives developers an easy way to set up test automation for any web app.

Let's start by outlining the steps of the pipeline:

• Push code and build it with Codeship

• Deploy the application to Heroku

• Smoke test the app with Assertible

The primary goal of this process is to create a completely automated build, deployment, and smoke-testing pipeline that will catch application errors during development by testing the app on a live staging environment.

The first step is to connect Codeship to a repository on GitHub. To do this, create a new project in your Codeship account and enter the Git clone URL of your repo:

Now every time code is pushed to the connected repository, the changes will be pulled and built by Codeship. Learn more about connecting repos in the docs.

Now that the repo is connected to Codeship, the next step is to deploy the application. For deploying to Heroku, Codeship's deployment pipelines are a perfect fit.

A deployment pipeline is specific to a branch in your repository, like controller. Multiple pipelines can be set up to handle deployments from different branches; for example, the Ruby app is deployed to production from the controller branch and to staging from any wip-* branch. The deployment pipeline is initiated only after all of the build and test steps have completed.

The example Ruby application is set up with the following pipelines:

1. For commits on the controller branch, deploy the app to production.

1. For builds on a branch that start with wip-*, deploy the app to a staging environment.

In Codeship, this is called [Wildcard Branch Matching](https://documentation.codeship.com/basic/getting-started/deployment-pipelines/#wildcard-branch-deployment-pipelines) and is extremely useful for handling deployment of the same app to multiple environments.

Testing a web app on staging environments is an effective first line of defense to find bugs early in the development cycle, before they make it to production. The process outlined above provides automatic deployments to an isolated staging environment where the release can quickly be checked and validated.

Learn more about Managing Multiple Environments for an App in the Heroku docs.

!Sign up for a free Codeship Account

With a deployment pipeline in place, you can set up Assertible to smoke test the application immediately after it's deployed. This approach has several benefits:

• Quickly validate potential releases

• Catch bugs earlier in the development cycle

• Tighten the feedback loop between QA and developers

• Prevent wasting time on testing broken applications

The Assertible Deployments API can be integrated with the deployment pipelines to run an automated test suite against both the staging and production environments.

The first step is to define your web service in Assertible. Log in to your Assertible Dashboard and enter the URL of your web service or import your Swagger Spec. A test will automatically be created for the URL, and you can add more tests for other critical endpoints.

For the Ruby API example, I've created a test for each endpoint that will validate the core functionality of the app and fail if anything is critically wrong.

Ideally, the smoke tests are fast and check core behavior to identify a faulty deployment. The assertions made against the Ruby API include:

• Validating the response against a JSON schema

• Verifying the HTTP headers

• Testing the response time

If any of these assertions fail, you will be immediately notified of a bad release.

Now that the web service is defined, we can go back to the Codeship deployment pipelines and add a second step to run the smoke tests. The Assertible deployments API makes this possible by tracking releases of an application and initiating tests against new deployments.

Add a Custom Script step to the deployment pipeline that makes a POST request to Assertible:

This step is called after the first step has successfully deployed the application to Heroku. You can use a simple cURL command to send the details of the release to Assertible, which will then initiate the tests for that service.

Environment variables

To execute the custom script, you need to set a few environment variables in your Codeship environment. They can be retrieved from the Assertible dashboard:

• ASSERTIBLE_TOKEN

• ASSERTIBLE_SERVICE_ID

• CI_COMMIT_ID (provided by Codeship)

Assertible supports the concept of environments for a web service. An environment is a separate URL where the same application may be deployed, like a staging or testing environment.

The example Ruby API has two environments:

• staging at https://ruby-example-staging.herkuapp.com

The example application is deployed to both staging and production. Tests can be run against a specific environment by specifying the environmentName in the POST body.

When Assertible receives deployment events, it will look up the environment and run tests against that URL. Learn more about POSTing deployments to Assertible and how environments work here.

Everything is now set up for Codeship to build the code, deploy to Heroku, and run smoke tests against the application. As code is pushed to the repository, you will hopefully see passing tests in Assertible and a green build in Codeship:

Once your continuous delivery and post-deployment testing pipeline is set up, you'll want to show off how all your builds and tests are passing.

Codeship and Assertible both offer status badges you can use in a README, documentation, or anywhere else to let your team and users know the status of your web services.

• Codeship status badges

• Assertible status badges

There are many great resources to learn more about setting up and extending this pipeline. Check out some of the articles below:

• The Assertible blog

• Codeship Getting Started

• Assertible Quick Start

Do you have any questions, comments, or corrections? Hit me up on Twitter. I'd love to hear about your deployment and testing workflow. To get started building and testing your code, sign up for a free Assertible account and Codeship account.