There was a time when IT monitoring had a relative simple definition. That day is gone. Now, IT monitoring is a multi-faceted activity that involves a suite of different tools, processes and strategies. In fact, it's not really helpful anymore to talk about "monitoring" tout court, because there are now so many distinct flavors and types of monitoring--moving far beyond infrastructure and application monitoring.
Just how complex is the IT monitoring landscape today? And how can DevOps teams cope with monitoring challenges? Let's explore.
A Brief History of IT Monitoring
Back in the day, applications and the infrastructure that hosted them were relatively simple. Everything ran on-premise, microservices were not yet a thing, and load balancing was primitive at best. It was all about application and infrastructure monitoring, and essentially making sure everything was up. That was easy to do using basic tools that you could write yourself; a simple script that pinged servers to see if they would respond was often enough to keep track of issues.
Later, monitoring tools like Nagios came along. They helped automate the monitoring processes described above. They could also do some simple load checks to notify you when the capacity of a given infrastructure component had passed a certain threshold. Still, these tools were premised on the basic principles of uptime monitoring: They detected when something ceased to be available, and that was about it.
Modern IT Monitoring
Fast forward to the present, and the way we deploy applications looks very different. Infrastructure often consists largely of software-defined components whose mapping to physical infrastructure is constantly changing. A single application could be deployed on-premise, in the cloud, in multiple clouds or in a mix of all of the above. Containers, microservices and serverless environments have significantly increased the number of moving parts that DevOps teams need to monitor.
These changes help to explain why IT monitoring now breaks down into several distinct disciplines and categories:
- Uptime monitoring. It's still important to make sure your applications and services are available, although uptime monitoring is rarely enough on its own to achieve all of your goals.
- Infrastructure monitoring. Modern infrastructure monitoring has assumed a new level of complexity because much of the infrastructure that you have to monitor -- like virtual machines and containers -- is software-defined. Modern infrastructure monitoring tools also typically have to be able to monitor on-premise and cloud-based infrastructure at the same time, since many organizations deploy workloads in both types of environments.
- Network monitoring. Network monitoring has evolved into a distinct discipline. It now involves making sure that sufficient bandwidth is available, as well as optimizing and load balancing the network. Network monitoring typically requires keeping tabs on software-defined networks whose configuration changes constantly. And it overlaps in some ways with security monitoring, since the network is the most common vector of attack for intruders.
- Application performance monitoring (APM). APM helps you to identify and fix performance problems. It is founded on the idea that you need to keep applications and services operating at peak efficiency; simply keeping them up is not enough.
- Security monitoring. Arguably, security monitoring could be broken down into several sub-categories of its own. Most security monitoring tools use data analytics to detect anomalies that could signal an intrusion, but security monitoring is about more than that. Penetration testing also falls within this category, as do tools that parse configurations in an effort to identify vulnerabilities.
- API monitoring. The concept of API monitoring is newer than many of the other types of monitoring on this list, but several vendors now offer monitoring tools designed specifically for this category of monitoring. That's no surprise, given the importance that APIs have assumed in helping microservices communicate and allowing applications to access remote resources, such as cloud storage.
- Data monitoring. Data monitoring is also somewhat less well known within the DevOps world as a whole. But as data admins will tell you, monitoring data sets for data quality errors is important -- particularly because we now rely so much on data to drive insights and automation.
- Disaster recovery monitoring. Discovery recovery monitoring tools help to ensure that data and applications are properly backed up and can be restored quickly when the unexpected happens.
- Compliance monitoring. It's hard to automate compliance monitoring entirely, but that is not stopping vendors from offering compliance monitoring software designed to help detect compliance issues. Expect these types of IT monitoring tools to become increasingly important as digital compliance rules grow even more complex.
Help! Monitoring Is Too Complicated
Having so many different types of monitoring to worry about makes it challenging to effectively monitor infrastructure and applications. That is especially true given that, in the age of DevOps, every engineer is supposed to have familiarity with everything that the IT organization does. If you think that, say, networking monitoring should be the job of network admins alone, or that only security engineers have to worry about security monitoring, you're not thinking in a DevOps mindset. You're instead embracing the concept of silos.
How can DevOps teams cope with the diverse and complex monitoring needs of modern IT environments? Obviously, automation is an important part of the answer. Platforms like PagerDuty, VictorOps and OpsGenie, which are essentially monitoring software for your monitoring software, come in handy here.
But you can only automate so much. At a certain point, DevOps teams need to admit that monitoring is hard, and requires a real investment of time and resources. In fact, given that monitoring processes and tools have grown so complex in recent years, and are likely to get even more complicated going forward, the idea that you can automate monitoring entirely is unrealistic. You should probably be spending even more time and effort on monitoring today than you did in the past.
That may be a difficult reality to accept. Monitoring is not on most IT professionals' lists of exciting job responsibilities. It can be tedious and feel like a distraction from activities that create new value for the organization.
But the fact remains that monitoring is important -- and hard. Admit it, accept it and make sure that all of your monitoring needs are supported with the right personnel, processes and tools.
