Testing Your CI Procedures with ElectricCommander

As more and more of our customers are moving from a waterfall process to a lean, agile one, the common question I get is “How do I test my ElectricCommander changes so I can reduce my errors and therefore downtime on my production server?” As a matter of fact, a lot of companies are using the ElectricCommander platform to ensure that what their developers build is thoroughly tested across multiple environments before being deployed in production. But how do you ensure the same precautions are taken so a bad change made to an ElectricCommander procedure does not result in the downtime of your internal development pipeline. After all, breaking the build CI workflow has a cost in term of lost development time, even if it does not necessarily impact your external customers. Our Services team runs into this request often, and so we are making available our own internal test framework for your usage. You can find it in our community GitHub repository . While the projects we publish on GitHub are not officially supported, we will do our best to fix any problems you encounter. The framework is composed of the following parts:

1. The “ntest” Perl script that will read and execute your tests. It is compatible with ec-perl 5.8.8 provided with ElectricCommander.
2. An “Assert.pm” module that makes it easy to test different assertions
3. A helper module named ECTest.pm to make it easier to write tests.
4. A directory with some examples. Additional real life example can be found in the EC-Admin repository under src/systemtest

How to write a test:

The README.md will give you detailed information on how to run and write tests but here is a very short example to show you how easy it is to write a simple test. In this example I have a factory procedure that is used to create a project and ensure naming conventions. So my test will reflect if the procedure “createSafeProject " does it job correctly:

1. Create the project as requested
2. Reject projects whose name start with a number
3. Reject projects whose name contains spaces

So here are some excerpts from the createSafeProject.ntest. The first test simply verifies the procedure has been created as expected by using the factory procedure called “createSafeProject”:

#
# Global Variables
my $PROJ="Ntest";           # Name of the project to test
my $PROC="createSafeProject"; # Name of the procedure to test

my $pid=$$;                # process id to generate semi random names
my $goodName="foo$pid";     # good project name
my $badName=$pid."foo";     # bad project name
my $spaceName ="foo $pid"; # bad name with space

#
# here we are calling the procedure to test
ntest "$PROC: project with correct name", {}, sub {
            my $xpath = $::N->runProcedure("$PROJ", {
       procedureName=>"$PROC",
       actualParameter=>
     });
            assertOK($xpath, "run $PROC procedure");

   # make sure it finished
   assertDef($xpath, "runProcedure finished without a timeout - goodName");

    if ($xpath) {
       # get new job ID
       my $jobId = $xpath->findvalue("//jobId");
       waitForJob($jobId);

       # tag the newly created project to make it easier to find
       $::N->setProperty("ec_tags", {projectName => $goodName, value => " ntest "});

        # now verify the project exists
       my $projXML = $::N->getProject($goodName);
     my $projectName=$projXML->findvalue("//projectName");
       assertEq($goodName, $projectName, "project $projectName created");

   # CLEAN
   $::N->deleteProject($goodName);
   }
};

The second test is to verify that a project whose name starts with number is not created:

ntest "$PROC: project with name starting with number", {}, sub {
    my $xpath = $::N->runProcedure("$PROJ", {
        procedureName=>"$PROC",
        actualParameter=>
      });
    assertOK($xpath, "run createSafeProject procedure - badName");
    # make sure it was created properly
    # make sure it finished
    assertDef($xpath, "runProcedure finished without a timeout"); 

    if ($xpath) {
        # get new job ID
        my $jobId  = $xpath->findvalue("//jobId");
        waitForJob($jobId);

        my $jobData = $::N->getJobDetails($jobId);
        my $outcome = $jobData->findvalue("//job/outcome");
        my $exitCode = $jobData->findvalue("//job/jobStep/exitCode");
        
        # testing outcome=error and exitCode=1
        assertEq($outcome, "error", "project $badName created despite bad name");
        assert($exitCode, "==", 1,  "exit code incorrect for project starting with number");

    # CLEAN
    $::N->deleteProject($badName);
    }
};

The third example is to check that the project does not contain spaces

ntest "$PROC: project with name with space", {}, sub {
    my $xpath = $::N->runProcedure("$PROJ", {
        procedureName=>"$PROC",
        actualParameter=>

      });
    assertOK($xpath, "run createSafeProject procedure - badName");
    # make sure it was created properly
    # make sure it finished
    assertDef($xpath, "runProcedure finished without a timeout"); 

    if ($xpath) {
        # get new job ID
        my $jobId  = $xpath->findvalue("//jobId");
        waitForJob($jobId);

        my $jobData = $::N->getJobDetails($jobId);
        my $outcome = $jobData->findvalue("//job/outcome");
        my $exitCode = $jobData->findvalue("//job/jobStep/exitCode");
        
        # testing outcome=error and exitCode=1
        assertEq($outcome, "error", "project $spaceName created despite space in the name");
        assert($exitCode, "==", 2,  "exit code incorrect for project with a space");

    # CLEAN
    $::N->deleteProject($spaceName);
    }
};

To invoke ntest, use the “--target” option to point to your commander server and simply add the directory containing your *.ntest files (or a list of files)

ntest --target=ECSERVER:443 examples

It should produce something like:

$ ./ntest --target=ecmaster:443 examples

Mon Dec  8 17:25:18 PST 2014: examples/createProject.ntest:
create project ................................................... PASSED

Mon Dec  8 17:25:18 PST 2014: examples/createSafeProject.ntest:
Check project "Ntest" exists ..................................... PASSED
Check procedure "createSafeProject" exists ....................... PASSED
createSafeProject: project with correct name ..................... PASSED
createSafeProject: project with name starting with number ........ PASSED
createSafeProject: project with name with space .................. PASSED

Total test duration: 3 seconds

Total tests: 6   Passed: 6   Failed: 0   Skipped: 0

So now you can get to it :) Hit me up if you have any questions or comments!

---

 

Want to see more cool things you can do with ElectricCommander?

Check out my talk at the recent DevOps Enterprise Summit  for best practices for achieving Continuous Delivery with ElectricCommander.