The Cloud Imperative: Balancing Innovation and Cost Control

This article was written by Nik Acheson, field chief data officer at Dremio.

Cloud spending has increasingly become a complicated web that organizations struggle to untangle. Forty-nine percent of cloud-based businesses cannot manage their cloud expenses, and 44% say that at least a third of their cloud spend is wasted, according to CloudZero research. How did we get here? And what can we do to unravel the complexity?

The answer is simple. Previously, when it came to embarking on cloud strategies, there was little focus on cost. Instead, organizations were driven by the need to speed up innovation, self-service, and scale as fast as possible. This is why we are now seeing these same companies, who were early adopters, operating in variations of hybrid models and private cloud solutions through more mature partnerships. They can enjoy increased negotiating power when it comes to securing a better deal from cloud providers hungry to continue winning business, as well as enjoying heightened flexibility without fear of being locked into a single vendor. But both of these things come at a cost (literally).

Yet, there is hope. Budgets are now generally easier to track against and billing, chargeback, and similar models are easier to attribute. Cloud forecasting and consumption models have also dramatically improved accuracy, helping organizations build a solid foundation to review their cloud strategy while controlling spending.

Related:5 Simple Ways to Reduce Kubernetes Costs

Operation: Cloud Strategy

For businesses that want to review their cloud strategy, the most obvious goal is to understand how to optimize storage and manage data efficiently, whether through combining data lake and warehouse approaches or using cloud data warehouses. As an industry, we are seeing the scaling up of computing resources to handle data flow and events, whether choosing specialized tools or enhancing existing cloud services.

But when it comes to reducing cost – before organizations can run in the cloud, they must learn to walk. This means understanding what’s happening in their environment.

Historically, teams have had to provision for storage and compute – with costs being managed, mitigated, or minimally understood – for easier forecasting. In fact, even as companies shifted to the cloud, central IT teams were still able to manage the provisioning of resources through service management tooling and processing. However, without the proper instruments, these teams often lose visibility of all the data events and associated cost-control measures across the business.

Related:7 Myths About Hybrid Cloud

The Strategy With MALT-itude of Benefits

Such an instrument has centralized MALT (metric, alerting, logging, and telemetry) capabilities. This provides a unified approach to monitoring and optimizing system performance and operations. Having a proper strategy helps grow organizational visibility – not just at the spending level but also on the health of the systems, data quality improvements, and overall stability. Additionally, it opens the door to finding cost-saving opportunities. It does so by balancing load during scheduled high traffic and demand periods that can easily shift when data is processed without impacting performance or stability. Nevertheless, for those who have a strategy in place, advancing to an intentional platform would continue to improve insights and recommendations for cloud savings. 

There are also built-for-purpose commercial tools available to central teams that provide resource visibility, forecasting, and attribution capabilities for full resource usage and budgeting. Using native tools like Cloudwatch inside AWS or a tool like Chronosphere across platforms helps to create a more focused MALT capability with cost-based reporting at its heart – to support the balance between innovation and value realization with cost efficiency.

The Sweet Role of the C-Suite

Cloud spending is not solely an IT or finance problem but rather an organizational one. C-suite responsibilities are vital in ensuring cloud provisions are costed accurately and can deliver what the business needs.

For example, most organizations engage in regular reviews and ongoing product- and portfolio-level incremental planning. During these planning sessions, the deliverables are aligned to and associatively mapped into budget allocation and spending. As teams run over (or under) budget, there should be ongoing touchpoints to review how and why costs are above or below expected ranges. During these times, deliverables should be aggressively tracked and committed to. Additionally, Individual product teams should have ongoing reviews with product leaders to ensure the scope and deliverables are on track.  

This can take various forms of planning, from scaled-down agile to more classic project planning approaches that include earned value metrics, like actual cost of work performed and holding check-ins at predefined intervals. Most mature organizations will now forecast and fund monthly or quarterly increments with at least yearly review processes in which all projects, programs, products, and portfolios are accepted or canceled, and financed for the next – ideally, balancing between driving innovation through the cloud while keeping a tight control on spending.

It is a critical challenge, but it is not insurmountable. Having a holistic view of the business and centralized MALT capabilities enhancing visibility and cost management is key. In combination with C-suite involvement, cloud provisions can be accurately costed and aligned with business needs, creating a well-structured cloud strategy that incorporates cost efficiency while paving the way for sustainable growth and success.