How to Fix journalctl Logs Consuming 100% Disk Space on Ubuntu Servers

Root Cause Diagnosis

Systemd-journald continuously collects log data from kernel events, system services, and stdout/stderr streams, storing them in persistent binary files under /var/log/journal/. By default, systemd caps journal storage at up to 10% of the root file system’s total volume capacity, which can equate to dozens of gigabytes on large disks. When an unhandled application loop, noisy daemon, or kernel failure floods stdout, the journal service rapidly expands to this limit and starves the root partition of available disk space.

Quick Fix

Flush archived journal logs immediately to reclaim disk space, keeping only the most recent 500 megabytes:

# Vacuum journal logs immediately to a max size threshold
sudo journalctl --vacuum-size=500M

Step-by-Step Resolution

Step 1: Measure Current Journal Disk Usage

Determine how much storage the systemd journal files currently occupy on your file system:

# Display disk usage of active and archived journal logs
journalctl --disk-usage

Step 2: Perform Immediate Emergency Log Vacuuming

Truncate older binary log files using either size-based or time-based retainment flags:

# Retain only the last 2 days of log entries
sudo journalctl --vacuum-time=2d

# OR enforce a hard size cap on total journal storage
sudo journalctl --vacuum-size=500M

Step 3: Configure Permanent Storage Limits in `/etc/systemd/journald.conf`

To prevent systemd-journald from filling up the disk in the future, edit the central configuration file:

sudo nano /etc/systemd/journald.conf

Uncomment and modify the following parameters under the [Journal] block:

[Journal]
# Enforce a maximum cap for total disk space consumed by persistent journals
SystemMaxUse=1G

# Enforce a maximum size for individual journal files before rotation
SystemMaxFileSize=100M

# Limit maximum log retention time
MaxRetentionSec=1month

# Enable compression for log files larger than 512 bytes
Compress=yes

Step 4: Restart the Journal Daemon and Verify Configuration

Apply the configuration updates without rebooting the server:

# Restart the systemd-journald service
sudo systemctl restart systemd-journald

# Re-check current disk usage to confirm limits took effect
journalctl --disk-usage

Why This Fails for Some Users

Active High-Volume Log Flooding (Unchecked I/O Rate): Vacuuming logs frees space temporarily, but if an active crash loop or run-away application writes thousands of log lines per second, journalctl will immediately re-fill the disk within minutes. You must identify and stop the root process generating log noise via journalctl -f or journalctl -p err.
Unmodified Active Journal Files: The --vacuum command only deletes archived (rotated) journal files. If a single active, open binary log file has already grown to multi-gigabyte scale, running --vacuum-size will not delete it until systemd-journald rotates it via sudo journalctl --rotate.
Forwarding to Secondary Loggers (rsyslog / syslog Redundancy): Configuring limits in /etc/systemd/journald.conf only restricts binary files under /var/log/journal/. If ForwardToSyslog=yes is enabled, duplicate logs are concurrently written in plain text to /var/log/syslog and /var/log/messages, which are managed separately by logrotate and can independently consume 100% disk space.
Read-Only / Corrupted Journal Filesystem States: If the disk hits absolute 0 bytes remaining, systemd may lock the journal file handles or remount the storage array in read-only mode, causing journalctl --vacuum-size commands to fail with I/O errors until space is manually cleared elsewhere (e.g., clearing /tmp or apt caches) or the file system is repaired.