Cloud migration has pushed data mining beyond the current analytics capabilities of most companies, with organizations forced to rely on multiple, disparate monitoring and data analytics solutions to maintain observability and security.
These were among the findings of a Coleman Parkes global survey of 1,300 CIOs and senior IT practitioners, commissioned by software company Dynatrace, which found on average, organizations capture just 10% of observability data for querying and analytics.
The report, "How to Tame the Data Explosion and Overcome the Complexity of the Cloud," also found CIOs' teams are using an average of 10 monitoring tools across their technology stacks but have observability into only 9% of their environment.
"Modern enterprises are complex and ever-expanding, often comprised of multiple teams —DevOps, SRE [site reliability engineers], ITOps, and so on — who are using different solutions for cloud monitoring and analytics," explained Adrian Phillips, Dynatrace's director of product marketing.
Multiple Tools Lead to Multiple 'Versions of Truth'
Teams gravitate toward tools they are comfortable with and are often hesitant to move away from legacy systems, he said, but leveraging different tools leads to a lack of consistency and visibility, creates silos between teams, and results in multiple "versions of the truth."
"Digital transformation is all about bringing teams together to work in a more simple and automated way, and shifting to a more unified approach to monitoring and analytics is a crucial step," Phillips adds.
Nearly six in 10 of CIOs surveyed said without a more automated approach to IT operations, their teams could soon become overloaded by the increasing complexity of their technology stack.
"Teams need platforms and processes that will quickly and automatically provide them with precise insights so they can better understand infrastructure threats and respond effectively," Phillips said. "They need a platform that enables continuous automation across BizDevSecOps processes in order to obtain the exact answers IT teams need from their data."
CIOs need to think long-term, which means investing in a unified platform that provides end-to-end observability, security, and business analytics with advanced AIOps, machine learning (ML), and automation to deliver precise answers versus relying on statistical guesses, he said.
"AIOps and automation can save teams time and allow them to focus on high-value initiatives like developing new solutions to drive innovation," Phillips said. "Without these tools, teams are stuck manually managing a data explosion brought on by modern cloud-native and multi-cloud environments."
However, ML-based approaches to data analytics are only as good as the data available.
"Causal AI is another approach which requires continuously updated topology to deliver precise answers and intelligent automation," Phillips pointed out.
Getting the C-suite on board with any new approach requires BizDevSecOps teams to trust the insights delivered by these processes while being able to increase focus on other high-value initiatives, according to Phillips.
In addition, more than three-quarters (77%) of CIOs surveyed said their IT environment changes every minute or less.
Phillips said this speaks to the challenges SREs and operations teams are facing to achieve complete observability across their IT environments and understand what is impacting the end user.
"Between the sheer volume of data and the ever-changing environment, it is not surprising teams are feeling underwater," he added.
Cloud Complexity Causing Burnout
CIOs are also expected to face the pressure of tightened budgets and talent shortages moving into 2023, at a time when 43% of survey respondents are convinced current approaches to observability data ingestion and storage won't support their future needs.
"Even now, 64% of CIOs say it has become harder to attract and retain enough skilled ITOps and DevOps professionals to manage and maintain their cloud-native stack," Phillips says. "That's because having teams chase manual tasks impacts morale and retention more than most understand."
The survey found that 93% of CIOs believe AIOps and automation are increasingly vital to alleviating the shortage of skilled IT, development, and security professionals and reducing the risk of teams becoming burned out by the complexity of modern cloud and development environments.
To further combat challenges brought on by mounting economic headwinds, CIOs should also look for ways to modernize tool sets, Phillips said.
"They must leverage modern platforms to drive greater efficiencies from DevOps, SRE, and application teams," he explained. "This can drive better business outcomes and experiences for users, enabling organizations to deliver value faster."
About the author
Nathan Eddy is a freelance writer for ITPro Today. He has written for Popular Mechanics, Sales & Marketing Management Magazine, FierceMarkets, and CRN, among others. In 2012 he made his first documentary film, The Absent Column. He currently lives in Berlin.
