When a Linux system faces severe memory pressure, the kernel must decide which process to terminate to free memory.
This is where the Linux Out-Of-Memory (OOM) killer comes into play.
The OOM killer is a last-resort mechanism. When the kernel cannot satisfy memory allocations after reclaiming available memory and taking other measures, it may terminate one or more processes to recover memory and keep the system running.
But how does Linux decide which process to kill?
It doesn’t simply choose the process using the most RAM. Linux uses a scoring mechanism that considers factors such as memory usage and administrator-defined adjustments.
Understanding OOM scores can help you investigate unexpected process termination and configure critical services more carefully.
What Is the OOM Killer?
Linux normally tries several approaches before reaching an OOM condition.
The kernel can reclaim filesystem caches, reclaim other memory, and potentially use swap. If memory pressure becomes severe and a required allocation still cannot be satisfied, the kernel may invoke the OOM killer.
Its basic goal is straightforward:
Free enough memory to allow the system to continue operating.
When the OOM killer selects a process, that process receives a termination signal, typically SIGKILL. The process cannot catch or ignore SIGKILL, so it is forcibly terminated.
You may see a message similar to this in kernel logs:
Out of memory: Killed process 1234 (myapp)
The interesting question is why myapp was selected instead of another process.
What Is oom_score?
Linux exposes an OOM score for processes through /proc.
You can inspect a process with:
cat /proc/1234/oom_score
Replace 1234 with the process ID.
The value is an integer representing how strongly the process is considered a candidate for termination by the OOM killer.
In general, a higher OOM score means the process is more likely to be selected when the kernel needs to choose a victim.
You can inspect multiple processes with:
for pid in $(pgrep -d ‘ ‘ -f .); do
echo “$pid $(cat /proc/$pid/oom_score 2>/dev/null)”
done
For practical troubleshooting, however, you will often want to combine the score with memory usage and process information rather than looking at the score by itself.
OOM Score Is Not Simply a Memory Ranking
One common misconception is:
“When memory resources become limited, the process with the highest RAM usage is selected for termination.”
That’s not necessarily true.
Memory consumption is an important factor in the OOM selection process, but the kernel also considers other information.
For example, administrators can influence a process’s OOM preference using oom_score_adj.
Check it with:
cat /proc/1234/oom_score_adj
The value can range from:
-1000
to:
1000
A more negative value makes a process less likely to be selected, while a more positive value makes it more likely to be selected.
This gives administrators a way to influence OOM victim selection.
Understanding oom_score_adj
Suppose you have two processes:
database
application
You might want the application process to be sacrificed before a critical database process.
An administrator can adjust the database process’s OOM preference:
echo -500 > /proc/1234/oom_score_adj
The exact value should be chosen carefully based on the system’s requirements.
At the extreme end:
-1000
makes the process effectively immune to the OOM killer’s normal selection mechanism.
This should be used cautiously. Protecting too many processes can leave the kernel with fewer viable processes to terminate during severe memory pressure.
Why Does Linux Need a Scoring System?
Imagine a server with 16 GB of RAM running:
SSH
database
web server
monitoring agent
application workers
background jobs
A severe memory shortage occurs.
Killing an arbitrary process could make the situation worse. Terminating an important system component might cause additional failures, while terminating a less critical workload could recover enough memory for the server to continue operating.
The scoring mechanism gives the kernel a way to make a more informed decision.
The general principle is:
Recover useful memory while minimizing the damage caused by process termination.
Viewing OOM Information for a Process
For a specific PID, inspect:
cat /proc/1234/oom_score
and:
cat /proc/1234/oom_score_adj
You can also examine the process itself:
ps -p 1234 -o pid,ppid,%mem,rss,vsz,cmd
This gives you context around the OOM score.
For example:
PID %MEM RSS COMMAND
1234 18.2 3.0G java -jar application.jar
If this process also has a relatively high OOM score, it may be an important candidate during an OOM event.
However, don’t assume that the score alone predicts exactly what the kernel will do. OOM selection depends on the memory context and other kernel decisions at the time of the event.
OOM Scores Can Be Different in Containers
Containers introduce another important layer.
A container may have a memory limit imposed through Linux cgroups. If the container reaches its configured memory limit, an OOM event can occur within that memory-controlled environment even if the host still has available RAM.
This means you may encounter:
Host memory: available
Container memory: exhausted
The relevant OOM decision may therefore happen within the container’s cgroup rather than as a simple system-wide “the server ran out of RAM” event.
When investigating container OOM events, inspect both:
- Host memory usage
- Container memory usage
- Container memory limits
- Cgroup memory events
- The processes running inside the container
Don’t Automatically Protect Every Important Process
It can be tempting to protect critical services by assigning very negative oom_score_adj values.
For example:
echo -1000 > /proc/1234/oom_score_adj
But doing this indiscriminately can create another problem.
If every important process is protected, the kernel may have very few processes available to terminate when memory becomes critically low.
OOM configuration should therefore be treated as part of a broader memory-management strategy rather than a replacement for fixing memory leaks, incorrect limits, or insufficient capacity.
How to Investigate an OOM Kill
When a process unexpectedly disappears, start with the kernel logs:
journalctl -k | grep -i -E “oom|out of memory|killed process”
You can also use:
dmesg | grep -i -E “oom|out of memory|killed process”
Look for:
- The process that was killed
- Its PID
- Memory usage information
- The memory-constrained environment
- Other processes involved in the event
Then inspect the application’s logs around the same timestamp.
The goal isn’t simply to identify the victim. You also want to determine why the system reached the point where a victim had to be selected.
OOM Killer vs. Application-Level Memory Management
It’s important to remember that the OOM killer is not designed to fix the underlying cause of excessive memory consumption.
If a Java application has a memory leak, for example, the OOM killer may eventually terminate the process, but that doesn’t solve the leak.
Similarly, if a container has an unnecessarily low memory limit, repeatedly restarting it after OOM kills won’t address the underlying configuration issue.
The correct response usually involves investigating:
Application memory usage
↓
Memory limits
↓
Workload changes
↓
Swap and reclaim activity
↓
Cgroup constraints
↓
OOM victim selection
Conclusion
The Linux OOM killer exists to protect the system when memory allocation becomes impossible under the relevant constraints. When it needs to select a process, Linux uses an OOM scoring mechanism rather than simply killing a random process.
You can inspect a process’s score with:
cat /proc/<PID>/oom_score
and its administrator-adjustable preference with:
cat /proc/<PID>/oom_score_adj
A higher OOM score generally makes a process a more attractive candidate, while oom_score_adj allows administrators to influence that preference.
Facing issues?
Our technical support
engineers can solve it.
