Explosion of Data in the Cloud Era Leading to Observability Complexity

As organizations continue to adopt cloud-native technologies, the complexity of their technology stacks has increased significantly, but what is the actual impact? A new report from Dynatrace aims to provide that answer.

The report titled, "The state of observability 2024," is based on a survey of 1,300 CIOs and technology leaders and reveals that while multicloud environments and cloud-native architectures enable rapid transformation and innovation, they also create an explosion of data that is becoming increasingly difficult for IT teams to manage.

The report found that 88% of organizations have experienced an increase in complexity over the past 12 months, with the average multicloud environment spanning 12 different platforms and services. This complexity not only makes it more challenging to deliver outstanding customer experiences but also makes applications more difficult to protect.

Key findings include:

Related:What Is CloudOps?

"Over half of organizations — 51% — said that the complexity of their technology stack will continue to increase, which was surprising because it highlights the education that needs to be done around adopting a platform-centric approach," Jay Livens, senior director of product marketing at Dynatrace, told ITPro Today.

The Struggle to Keep Up with Data

One of the most significant challenges organizations face is the sheer volume of data generated by cloud-native technology stacks.

According to the report, an overwhelming 86% of technology leaders say that these stacks produce an explosion of data that is beyond humans' ability to manage effectively. To cope with this data deluge, organizations use an average of 10 different monitoring and observability tools to manage applications, infrastructure, and user experience. However, 85% of technology leaders say that the number of tools, platforms, dashboards, and applications they rely on only adds to the complexity of managing a multicloud environment.

As organizations grapple with the increasing complexity and data volumes, traditional approaches to log management and analytics are proving inadequate. The report found that 81% of technology leaders say manual approaches cannot keep up with the rate of change in their technology stack and the volumes of data it produces. This not only hinders their ability to gain valuable insights from their data but also takes time away from innovation, with 81% of technology leaders saying that the time their teams spend maintaining monitoring tools and preparing data for analysis steals time from innovation.

Related:Who Needs – or Doesn’t Need – a CloudOps Engineer?

Best Practices to Improve Observability

There are several things that organizations can implement to manage the increasing complexity of their multicloud environments.

Livens noted that organizations are starting to unify monitoring tools and automate some tasks in order to adapt to the complexity of multicloud environments, but the research reveals there are still significant challenges in managing this complexity. To address these challenges and drive business value, advanced AI, analytics, and automation capabilities are essential.

To overcome the challenges posed by cloud-native technologies, organizations are turning to AIOps. 

The report found that 72% of organizations have adopted AIOps to reduce the complexity of managing their multicloud environment. However, 97% of technology leaders say that probabilistic machine learning approaches have limited the value AIOps delivers due to the manual effort needed to gain reliable insights.

"While AIOps is a crucial practice, it relies on probabilistic methods, which can be imprecise and time-consuming to implement," Livens said. "I hope to see more organizations adopting mature AI, analytics, and automation strategies to overcome the complexity of modern technology stacks. By doing so, they can reduce manual efforts and streamline operations, empowering teams to spend more time on innovation and less time on routine maintenance tasks."