How to Fix OOMKilled Errors in Kubernetes

At first glance, OOMKilled might sound like a song title from a band trying to fuse the style of "MMMBop" with death metal.

In fact, though, OOMKilled has nothing to do with either the Hanson Brothers or Marilyn Manson. Instead, it's a common type of error on Kubernetes that can cause applications to fail.

That's why understanding what causes OOMKilled events and how to troubleshoot them — the topics this article covers — is important for any IT pro who works with cloud-native technology like Kubernetes.

What Is OOMKilled in Kubernetes?

OOMKilled is an error in Kubernetes that means containers were terminated (or "killed") to save memory. It's also sometimes called exit code 137 because 137 is the code number that Kubernetes reports when containers stop running due to OOMKilled errors.

OOM stands for out-of-memory in this context — so when a container stops running due to an OOMKilled error, it's because it was killed due to an out-of-memory event.

How Process Termination Works on Linux

To understand fully what that means, let's talk about how processes work, and how they can be "killed," on Linux (which is the operating system that usually powers the nodes, or host servers, within a Kubernetes cluster).

Any process running on a Linux server can be terminated if the operating system sends a termination, or "kill," signal to it. You can send that signal manually using a command like the following:

Related:How to Simplify Kubernetes Management

kill 1234

This command tells Linux to terminate a process whose process ID is 1234. (You can find process IDs on Linux using the ps -e command.)

However, it's not only through manual kill commands that processes can be terminated on Linux. The operating system can also terminate a process automatically if it believes doing so is necessary to keep the system running stably.

This may happen in the event that a system runs critically low on memory. If memory becomes totally maxed out, a system is likely to crash because processes will run out of space to store temporary data. To prevent this from occurring, Linux can decide to terminate specific processes.

Linux determines which processes to kill during periods of low memory using an algorithm that assesses factors such as how much memory each process is using and how many processes are running. The selection is made automatically, and there's not much you can do to control it.

So, if a Linux server happens to decide that the process that runs a container is at the top of the list for termination during an out-of-memory event, the operating system will kill that process, resulting in an OOMKilled error for the container.

Related:4 Reasons to Run Kubernetes On-Prem

How to Diagnose and Fix OOMKilled on Kubernetes

Determining whether a Kubernetes application has experienced an OOMKilled error is easy. Simply run the command:

kubectl describe pod <podname>

If an OOMKilled error has occurred, you'll see the following lines in the output of that command:

Last State:  Terminated
Reason:      OOMKilled

But this doesn't tell you much about why the error occurred. To determine this, you'll need to work through some additional steps:

What Causes OOMKilled Errors?

Once you know the underlying cause of the OOMKilled event, you can take steps to fix it based on the guide above.

Here's a look at common causes of OOMKilled and how to fix each one: