Root Cause Diagnosis
The Linux kernel’s Out of Memory (OOM) killer selects target processes to terminate based on a calculated oom_score, which correlates directly with the percentage of system RAM consumed by a process and its child tasks. High-priority system daemons (such as database engines or API gateways) that consume significant memory during normal operation often yield high base scores, making them primary targets over short-lived or low-priority background workers. Without explicit score adjustments, the kernel lacks awareness of process criticality and prioritizes raw memory reclamation over service importance.
Quick Fix
Instantly immunize a critical daemon process (e.g., PID 1234) from the OOM killer by writing -1000 directly into its tuning interface in /proc:
# Completely protect PID 1234 from OOM killer termination
echo -1000 | sudo tee /proc/1234/oom_score_adj
Step-by-Step Resolution
Step 1: Inspect Current OOM Scores and Adjustments
Check the current heuristic score and adjustment factor for your targeted daemon:
# Replace <PID> with your process ID
cat /proc/<PID>/oom_score
cat /proc/<PID>/oom_score_adj
The value of oom_score_adj ranges from -1000 (completely disables OOM killing) to +1000 (makes the process the primary target).
Step 2: Configure Systemd Service OOM Protection (Persistent)
Most modern Linux distributions manage system daemons through systemd. To ensure protection persists across service restarts and system reboots, configure the OOMScoreAdjust directive in the service override file.
# Create or edit an override file for your daemon (e.g., mysqld)
sudo systemctl edit mysqld
Add the following block to set the adjustment factor:
[Service]
# Value range: -1000 (least likely to be killed) to 1000 (most likely)
OOMScoreAdjust=-900
Note: Avoid setting -1000 for general services unless absolutely necessary; -900 keeps the daemon highly protected while allowing extreme safeguard fallbacks if the kernel panics.
Step 3: Apply and Verify the Configuration
Reload systemd units, restart the service, and verify that the kernel reflects the update:
# Reload systemd manager configuration
sudo systemctl daemon-reload
# Restart the service to apply changes
sudo systemctl restart mysqld
# Verify the adjusted score on the newly spawned process
cat /proc/$(pgrep -f mysqld | head -n1)/oom_score_adj
Why This Fails for Some Users
- Systemd Memory Cgroup Limits Superceding OOM Scores: If a service exceeds a hard cgroup threshold set by
MemoryMaxorMemoryHighin its systemd unit, the kernel’s cgroup-level OOM killer triggers locally within that container/slice. The local OOM killer will terminate processes within that cgroup regardless of a negative globaloom_score_adj. - Child Process Inheritance and Privilege Escalation: Non-root worker threads or spawned child processes cannot decrease their own
oom_score_adjvalue unless running with theCAP_SYS_RESOURCEcapability. If a daemon drops privileges after startup without setting the score beforehand, runtime adjustment attempts will fail with aPermission deniederror. - Containerized Environments (Docker/Kubernetes): In containerized runtimes, setting
-1000inside a unprivileged container is blocked by default security profiles (AppArmor/Seccomp). Adjustments must instead be declared at the container engine level (e.g.,--oom-score-adjin Docker orresources.requests/limitsin Kubernetes Quality of Service classes). - Kernel Panic On Global Memory Exhaustion: Disabling the OOM killer completely (
-1000) across all large memory processes causes the kernel to run out of allocatable memory entirely without recourse, leading to an immediate kernel panic or total system hang instead of selective process termination.