It's a no-brainer that some DevOps teams will accomplish more, with higher quality results, than others. But according to the 2017 State of DevOps Report released on June 6, the difference between high and low performing groups is much wider than might be expected.
For this year's report, the open source company Puppet and DORA, a DevOps research and assessment company, surveyed over 3,200 IT professionals, developers and executives from around the world. The bottom line is that the DevOps model continues to grow, with 27 percent of those responding to the survey working in DevOps -- up from 16 percent just three years ago. Most of the report focuses on the performance of DevOps teams, however, breaking them down into three performance-based categories: high, medium and low.
Although the medium and low performing groups show some gains over last year, in all areas covered in the report they lag far behind the high performers who have:
- 46 times more frequent code deployments.
- 440 times faster lead time from commit to deploy.
- 96 times faster mean time to recover from downtime.
- 5 times lower change failure rate (changes are 1/5 as likely to fail).
Those numbers should be more than big enough to prod any IT department to make sure their DevOps teams are up to speed -- and to work to get them there if they're not.
As high as those numbers are, however, they represent a considerable narrowing from last year for the top two items on the list. In the 2016 report, high performing groups deployment frequency was 200 times higher and lead time was a whopping 2,555 faster. The gap for the other two items has widened, however, especially for the all important "time to recover from downtime," where last year high performing groups were 24 times faster. In 2016, the high performers had a three times lower change failure rate.
The report points out that the areas in which the gap between high and low performers is narrowing seems to represent a positive trend:
"High performers are still shipping code as the business demands, while low performers went from shipping between once per month and once every six months in 2016, to shipping between once per week and once per month in 2017. Low performers in 2017 have also reduced their lead time for changes: from between one month and six months in 2016 to between one week and one month [this year]."
In other words, it's not that high performers have fallen down on the job, but that low performing teams are improving.
For the record, high performers indicate they routinely deploy code on demand, performing multiple deployments per day, with a "code committed to code deployed" time of less than an hour. High performers also indicated that on average it takes them less than an hour to recover from a system outage, compared to between one day and one week for low performers. The "change failure rate" was between zero and 15 percent for high performers and between 31-45 percent for low performers.
The data suggests that perhaps the best way to turn medium and low performing groups into high performers is by integrating more automation into the mix.
"When we compared high performers to low performers, we found that high performers are doing significantly less manual work," the report states. When compared to the other groups, high performers automate:
- 33 percent more of their configuration management.
- 27 percent more of their testing.
- 30 percent more of their deployments.
- 27 percent more of their change approval processes
"What we've also learned, in our work with teams and in conducting research, is that people are not very good at estimating how automated they are. What they are good at is estimating the percentage of their work that is still done manually. That's not surprising: Manual work is painful, so people are highly aware of it. Once work is automated, it's no longer painful, and tends to disappear from people's attention."