Linux systemd service status: main process exited, code=killed, status=9/KILL

Root Cause Diagnosis

The status code=killed, status=9/KILL in systemd indicates that the main process was abruptly terminated by an uncatchable SIGKILL signal issued either by the Linux kernel Out-Of-Memory (OOM) killer or an explicit system invocation (such as systemd-oomd, an administrator, or explicit resource timeout enforced by systemd itself). Unlike SIGTERM (signal 15), SIGKILL cannot be handled or ignored by the application, resulting in immediate process termination without cleanup. This issue typically occurs when a service breaches cgroup memory limits, exhausts host physical RAM and swap, or exceeds systemd watchdog timeouts.

Quick Fix

Check kernel ring buffers immediately to verify if the kernel OOM killer terminated the service process, and identify the exact timestamp and memory state:

sudo dmesg -T | grep -E -i 'oom|killed process' | tail -n 20

If memory limits are restricting the unit, temporarily raise memory caps and restart the service:

sudo systemctl set-property <service_name>.service MemoryMax=2G
sudo systemctl restart <service_name>.service

Step-by-Step Resolution

Step 1: Inspect Detailed Systemd Logs and OOM Diagnostics

Retrieve the recent journal logs specifically for the failing unit to verify whether systemd-oomd or the Linux kernel invoked the kill signal:

sudo journalctl -u <service_name>.service -n 100 --no-pager

To specifically isolate kernel-level memory invocations:

sudo journalctl -k | grep -i "oom"

Step 2: Configure Systemd Memory Limits and OOM Policies

Modify the service unit file to grant sufficient memory headroom or customize how systemd handles low-memory events.

1. Open a systemd drop-in override for the affected service:sudo systemctl edit <service_name>.service
2. Add or adjust memory allocation parameters and disable aggressive systemd-oomd killing if appropriate:[Service] # Increase memory limit allowed for the unit's cgroup MemoryMax=4G MemoryHigh=3.5G # Set policy for systemd-oomd (options: cancel, kill, continue) ManagedOOMMemoryPressure=kill ManagedOOMMemoryPressureLimit=90% # Prevent kernel OOM killer from favoring this process (range: -1000 to 1000) OOMScoreAdjust=-500
3. Save the file, reload systemd configurations, and restart the service:sudo systemctl daemon-reload sudo systemctl restart <service_name>.service

Step 3: Rule Out Systemd Timeout Executions

If the service fails during startup or shutdown and receives SIGKILL after a delay, systemd may be forcibly terminating the process after TimeoutStartSec or TimeoutStopSec expires.

1. Edit the service override file (sudo systemctl edit .service):[Service] TimeoutStartSec=300s TimeoutStopSec=60s
2. Alternatively, disable SIGKILL sending on timeout and allow graceful termination:[Service] SendSIGKILL=no

Step 4: Configure Host-Level Swap Protection

If system RAM is fully exhausted and swap is disabled, configure persistent swap space to prevent kernel SIGKILL actions.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Ensure persistence in /etc/fstab:

/swapfile swap swap defaults 0 0

Why This Fails for Some Users

• cgroup v2 systemd-oomd Aggressive Kills: On modern Linux distributions (Ubuntu 22.04+, Fedora 34+, Debian 12+), systemd-oomd operates at the cgroup slice level rather than individual PIDs. It may issue SIGKILL to entire service slices based on swap or memory pressure trends long before traditional kernel OOM kicks in. Disabling the unit daemon (sudo systemctl disable --now systemd-oomd) or setting ManagedOOMPreference=omit in the unit override is required to stop this behavior.
• Hard Hardware/Container Limits (LXC/cgroups/Kubernetes): If the service runs inside an unprivileged LXC container, systemd unit adjustments inside the guest cannot override host-enforced cgroup boundaries. The host kernel will send SIGKILL directly to the container process regardless of local systemd configurations.
• External Process Watchdogs / Security Agents: Third-party monitoring daemons, custom health checks, or security tools (such as fail2ban, custom cron scripts, or security agents) configured with kill -9 or killall -9 can terminate the process externally without logging reason codes inside systemd’s direct scope.
• Execution of KillMode=control-group Default: By default, systemd terminates all child processes in a control group when the main process stops or fails. If a child process breaches memory limits, systemd may kill the entire group via SIGKILL, reporting status=9/KILL on the main process wrapper even if the primary binary was not the root consumer.