Table of Contents:
1. Reduce Cloud Logging Data Ingestion
2. Make Sure Cloud Logging Is Turned On Where Needed
3. Don't Settle for Cloud Vendors' Logging Tools
4. Set Cloud Logging Retention Policies
5. Correlate Cloud Log Data
Getting More From Cloud Logging
You probably understand why you should use cloud logging. Logs are a vital source of visibility into the health, performance, and security of your cloud resources.
But how do you make the most of cloud logging? That's a more complicated question. This article helps answer it by discussing best practices for getting the most out of cloud logs and logging tools. Specifically, we'll cover strategies that help minimize cloud logging costs, maximize visibility, and make your logging data as actionable as possible.
Best Practice 1: Reduce Cloud Logging Data Ingestion
Your first instinct when it comes to cloud logging might be to log and analyze as much data as possible. After all, the more data, the better, right?
Theoretically, yes. But the problem with logging every potential data point produced by your cloud resources is that you'll end up spending more. Most cloud logging tools charge based in part on how much data or how many events you ingest into them, so more data means a higher price.
On top of this, having more data than you can realistically analyze can undercut visibility. It's better to draw actionable conclusions based on a smaller amount of cloud logging data than to try to analyze more information than you can make sense of.
So, avoid logging events that aren't necessary. You can also consider practices like data sampling or removing redundant entries from log files to reduce cloud log size and, by extension, cloud logging costs.
Best Practice 2: Make Sure Cloud Logging Is Turned On Where Needed
We just told you to avoid logging unnecessary data, so this next point of advice may sound counterintuitive: Make sure that logging is enabled wherever you need it across your cloud.
We're not saying you should turn on logging for resources (like VMs hosting applications that you are testing) that you don't need to track. That would be a waste of effort and money, and it would violate the cloud logging best practice we described above.
What we are saying is that you need to make sure that logging is enabled wherever you do need it. This is important because some cloud services don't generate logs by default, so you'll want to make sure that you turn logging on if you need to collect data from them.
Best Practice 3: Don't Settle for Cloud Vendors' Logging Tools
All of the major cloud vendors offer built-in cloud logging tools and services. These solutions are easy to use because they integrate with public cloud platforms by default. Using them is usually as simple as turning them on, then accessing logs through your cloud provider's portal.
However, there are limitations to cloud vendors' own logging tools. A big one is that most of these tools only work with a single cloud, which is problematic if you have multiple clouds to manage. The built-in logging tools may also cost more in some cases to ingest, analyze, and store data than third-party solutions, and their visualization and analytics features may be more limited.
So, before deciding to manage all of your cloud logging using the built-in tools that your cloud provider offers, explore alternative options. You may decide to use the built-in tools alongside third-party solutions, or to rely on third-party tooling alone to manage cloud logs.
Best Practice 4: Set Cloud Logging Retention Policies
In addition to charging based on the volume of data they ingest, most cloud logging tools also charge based on how much log data you retain and how long you retain it.
For that reason, it's important to set retention policies for your logs. Rather than planning to store all log data indefinitely, determine how long you'll need to keep different types of logs on hand based on factors such as compliance requirements. Then, set up auto-deletion rules accordingly within your cloud logging tools.
Best Practice 5: Correlate Cloud Log Data
Getting the most out of cloud logs requires more data than you'll find in the logs themselves. Ideally, you'll also correlate log data with other sources of information, such as application performance metrics that aren't recorded in cloud logs or authentication events that are stored in separate systems.
So, strive to ensure that analytics operations are able to draw on data from your cloud logs as well as other sources. That way, you get the fullest possible context on events or trends that appear within cloud logs.
Getting More From Cloud Logging
Cloud logs are a crucial source of visibility into cloud workload performance, cost, and security. But getting the insights you need in the most actionable form possible requires more than just collecting whichever data your cloud provider logs by default and calling it a day. Be intentional in choosing what to log and make sure you optimize your logging and analytics processes to minimize costs and maximize actionability.
About the author
Christopher Tozzi is a technology analyst with subject matter expertise in cloud computing, application development, open source software, virtualization, containers and more. He also lectures at a major university in the Albany, New York, area. His book, “For Fun and Profit: A History of the Free and Open Source Software Revolution,” was published by MIT Press.
