Fix: Apache graceful restart failing due to lingering defunct workers

Root Cause Diagnosis

An Apache graceful restart (apachectl graceful or systemctl reload apache2) sends a SIGWINCH signal instructing parent and worker processes to finish active requests before exiting. When worker threads are blocked on non-interruptible I/O operations, deadlocked database connections, or unmanaged C-extension subprocesses (e.g., PHP extensions or mod_wsgi threads), they ignore the graceful shutdown signal and hang indefinitely. Because these lingering worker processes maintain bindings to port 80/443 or hold IPC semaphores, the new Apache parent process cannot bind sockets or allocate shared memory, causing graceful reloads to stall or fail completely.

Quick Fix

Forcefully terminate all orphaned Apache worker processes and release held semaphores before executing a hard service restart:

killall -9 httpd apache2 2>/dev/null; ipcs -s | awk '$3=="www-data" || $3=="apache" {print $2}' | xargs -r ipcrm -s; systemctl restart apache2 || systemctl restart httpd

Step-by-Step Resolution

1. Identify Hung Apache Worker Processes

Check for defunct (<defunct>) processes or long-running worker threads that survived the graceful reload signal.

ps aux | grep -E 'httpd|apache2' | grep -v grep

Look for processes in D (uninterruptible sleep) or Z (zombie) status.

2. Configure Process Timeout Limits

Prevent workers from hanging indefinitely on stalled requests by enforcing hard socket and execution timeouts in /etc/apache2/apache2.conf or /etc/httpd/conf/httpd.conf.

# Reduce maximum time Apache waits for I/O operations
Timeout 30

# Enable GracefulShutdownTimeout to force-kill workers that do not exit cleanly
GracefulShutdownTimeout 15

3. Tweak MPM Worker/Event Termination Directives

If using mpm_event or mpm_worker, configure worker lifecycle boundaries so processes refresh predictably rather than accumulating leaked resources.

<IfModule mpm_event_module>
    StartServers             3
    MinSpareThreads         25
    MaxSpareThreads         75
    ThreadLimit             64
    ThreadsPerChild         25
    MaxRequestWorkers      150
    MaxConnectionsPerChild 1000
</IfModule>

Setting MaxConnectionsPerChild to a non-zero value (e.g., 1000) forces child processes to terminate and re-fork periodically, preventing permanent process hangs.

4. Clean Held IPC Semaphores and Shared Memory Blocks

When worker processes crash ungracefully, leaked IPC semaphores prevent new workers from spawning.

# List IPC semaphores owned by the web server user
ipcs -s | grep -E 'apache|www-data'

# Clear all semaphores for the specific user
for id in $(ipcs -s | awk -v user="$(whoami)" '$3==user {print $2}'); do ipcrm -s $id; done

5. Verify and Reload Service

Validate Apache syntax, stop remaining processes, and start a fresh daemon instance.

apachectl configtest
systemctl stop apache2 httpd 2>/dev/null
systemctl start apache2 || systemctl start httpd

Why This Fails for Some Users

Uninterruptible I/O Sleep State (D state): If Apache workers hang while waiting for an NFS mount, a unresponsive remote database connection, or slow disk I/O, the kernel puts the thread into an uninterruptible sleep state. These processes cannot process SIGWINCH or even SIGKILL until the underlying system I/O call completes or times out.
Persistent Database Connection Pools: Modules like mod_php using persistent connections (mysqli.allow_persistent or PDO persistence) or backend Python WSGI applications with improper socket lock handling often hold open TCP sockets during a reload, blocking the graceful drain phase.
Systemd Service Type Mismatch: On systemd-managed platforms (e.g., RHEL/CentOS/Ubuntu), if the systemd unit file is set to Type=forking without a proper KillMode=mixed or explicit ExecReload command using apachectl -k graceful, systemd may misinterpret process tracking and leave orphaned worker PIDs outside its control group.
Kernel Semaphore Exhaustion: High-traffic servers running prefork MPM can hit system-level semaphore limits (/proc/sys/kernel/sem). When defunct processes leak semaphore arrays, new Apache parent processes fail to start even after all PIDs are terminated.