Author’s Note: This post is a summary of the chapter “Orchestrating Quality Across DevOps Pipelines” from the new book on optimizing quality and embedding testing and test practices in DevOps, Continuous Testing for DevOps Professionals . The new book launches at DevOps World | Jenkins World 2018 on September 17.
If you are a tester who interacts with engineering, a software engineer who tests, or a tester who lives in harmony in engineering, you are, or have, adapted to life in an agile world. This raises questions about whether test orchestration is different than overall delivery automation.
The answer is yes and no.
Yes, because of task and processes specific to testing. No, because in continuous delivery orchestration, testing “stuff” is just part of the larger set of “stuff” to consider .
Three challenges to integrating testing with DevOps
1. Methodological
To include appropriate levels of testing into efficient delivery pipelines requires the entire team agree to what tests are possible and necessary.
Key aspects to consider are:
Functional tests as a core pipeline function. If it’s an automated test, embed it in a Jenkins pipeline and run it, period. Consider how impactful adding these tests to pipeline can be.
Integration testing. While this is the “I” of continuous integration, many teams don’t do it well.
System testing. Typically, a “black box” test of the complete end-to-end application or system, this non-functional testing can be done well with automation.
User acceptance testing. Notification, dispatch of test plans to users, automation and relaying of results are all parts of the manual process that can be managed within an automated pipeline.
Make non-functional testing non-negotiable.
Load and performance testing . A team implementing automation with Jenkins and a cloud computing account can have a job to create a test environment, execute “load” by simultaneously running unit tests, report results back and tear down the environment almost as easily as executing a single unit test.
Security testing . Another quick win for test orchestration is to have security team members added to agile teams, and begin to plug in security tests as pipeline components.
2. Technical
Applying technology to help with streamlining testing across multiple, complex pipelines is a requirement to optimization.
Test automation is mature . There is no excuse for teams to skimp out on building great automated functional tests to plug into their continuous integration and continuous delivery (CI/CD).
Deployment options are a revolution . Containerization is likely the largest boost to testing since web recording made creating automated functional tests as easy as clicking a mouse.
3. Organizational
Driving change in thoughts and approaches to continuous testing can be difficult among organizations with a history of siloed QA teams as well as engineering-centric groups who don’t value the holistic and comprehensive range of testing available beyond the basics of unit and integration tests. The best way to face these challenges is with education.
Ideas to spark action
Kubernetes for quality . Learn the basics of Kubernetes and play with setting up a dev and test environment.
Mobile testing . Nearly every organization has a mobile app, and testing is usually an afterthought. Try out test services that enable you to replay tests on devices.
Testing pipeline . Create a testing pipeline and introduce continuous delivery CI/CD.
CI/CD . Choose a CI/CD as a Service or download Jenkins to learn how to build your own automated pipelines. Many definitions of CD start after the development tests end. So, if CI/CD is new to your organization, be the catalyst and start with CD!
Keys to Success
Start strategically . If the entire DevOps plan bakes test automation into the entire pipeline from start to finish, and treats the test code (functional, non-functional) as production code – that is a perfect strategic step toward success
Start small . To build trust, there needs to be a small and proven smoke or sanity test suite that runs automatically within CI (Jenkins) upon each code commit.
Start NOW. The entire industry is implementing Agile development practices and aiming to scale DevOps and shift to CI/CD, the best time to start is today!
To read the new book on continuous testing, come to DevOps World | Jenkins World 2018 in San Francisco and have it signed by the authors on September 17. Special conference pricing will be available in the conference bookstore. You can also order the book on Amazon beginning September 13 and be notified when the book is available through Perfecto.