Engineers don’t always follow standard processes or use the same tools—it is just how they tend to work. That behavior can pose a challenge to engineering leaders who then need to track a set of metrics or KPIs across disparate teams to get visibility into overall performance.
CloudBees wanted to understand how engineering leaders accomplished this and if there was any way to improve on it. We conducted a survey asking 156 engineering leaders how they currently aggregate multisystem data, and how well these setups fit their needs. We also asked what kind of metrics they use to understand their teams’ performance and what kind of insights they’d like to see.
What the Research Reveals
The results of the survey were intriguing. We learned that one of the greatest struggles for engineering leaders is finding a tool that can track standard metrics across teams that follow different processes. Ninety-two percent of respondents have used a third-party DevOps tool and 69 percent have used more than one. Survey participants haven’t found a tool that provides sufficient data-driven insights to transform their organizations, but they keep investing in multiple tools to do their best.
The survey also found that engineering leaders welcome any insights into their DevOps pipelines, although they most trusted the metrics identified by the DevOps Research and Assessment (DORA) group to accurately measure their teams’ performance. DORA metrics are a set of data points that have been proven effective at measuring companies’ success with adopting DevOps practices. They are the result of a six-year research program conducted by the DORA team, founded by well-respected DevOps thought leaders Gene Kim, Nicole Forsgren and Jez Humble. DORA metrics measure Deployment Frequency (DF), Mean Lead Time for changes (MLT), Mean Time To Recover (MTTR) and Change Failure Rate (CFR).
Survey participants also want to improve their development organizations, but still face challenges standardizing and improving the metrics that can help them achieve this goal. Engineering leaders are looking for a tool that can do this without forcing them to conform to a single process.
What more did this survey reveal? And how can CloudBees help engineering leaders empower their teams to perform at an elite level? Let’s dive into what we found and what it means.
For more insight into this research, join our webinar on May 12: Aligning High-Performing Engineering Teams to Business Value
Gaps in Current Analytics Tools
We learned that engineering leaders typically need to invest in a tangle of tools for team metrics. Sixty-nine percent of engineering managers who responded to the survey said they have used more than one third-party tool to track and measure engineering performance—but out of all the options available on the market today, none received an average rating higher than approximately three out of five (with five being the highest rating). None of the current tools used can meet all of an engineering manager’s needs on their own, and when they’re stitched together, they aren’t all that satisfying, either.
*Of these tools / solutions, which have you personally used, either currently or in the past, to track and measure engineering performance?
†Please rate how satisfied you are with the level of insights available to you today in these areas.
We interpreted these results to mean that engineering leaders need a more comprehensive and configurable engineering analytics tool to help them see and understand their teams’ performance at a glance.
What Metrics to Use When Measuring Engineering Teams
Engineering leaders need to track standard metrics across a range of teams engaged in different processes. Only about half of survey participants had heard of DORA metrics, but among those who were familiar, 88 percent of those participants said they trust that DORA accurately demonstrates organizational performance.
To us, this finding means that we can help spread the word about DORA metrics. Check out our blog post that delves into the subject in more detail, and stay tuned for more resources as well.
Availability of Insights
The survey asked participants to rank the importance of various DevOps-related insights: the predictability of delivery timelines; the team’s ability to invest in new business value versus maintenance, technical debt or rework; individual developer productivity; development and delivery process efficiency and effectiveness; team effectiveness in delivering on features and initiatives and the cost of development and running applications.
Non-tech companies and those with $501M-$1B in annual revenue placed “Development process efficiency and effectiveness in support of fast and high quality software delivery” slightly higher priority than other insights, but the difference was not significantly greater than other categories. From this data, we learned that participants consider all of these insights important. The ideal tool for measuring DevOps performance should measure all of these areas.