Fixing LiteSpeed OpenLiteSpeed cache crawler 403 Forbidden loop

by

The OpenLiteSpeed and LiteSpeed Web Server (LSWS) cache crawler encounters a 403 Forbidden loop when the crawler’s automated user-agent triggers internal security throttling, IP access restrictions, or ModSecurity/WAF rules. Because the crawler originates locally or from a configured system loopback IP while sending rapid HTTP/HTTPS requests, default security policies misidentify the process as a Layer 7 DoS attack. This causes LiteSpeed to reject the requests with a 403 status, which the crawler logs and retries, creating an infinite loop.

Quick Fix

Force-allow loopback requests in the LiteSpeed Server Control Panel under Server Configuration > Access Control, or dynamically add your server’s primary public IP and local loopback (127.0.0.1) to the allowed list using iptables or standard server access configs.

If running OpenLiteSpeed with lsctl or systemd, restart the server after temporarily granting unlimited connections to loopback:

/usr/local/lsws/bin/lswsctrl restart

Step-by-Step Resolution

  1. Verify Crawler User-Agent & Allowed IPs in OpenLiteSpeed WebAdmin
  • Log into the OLS WebAdmin Console (https://your-server-ip:7080).
  • Navigate to Server Configuration > Access Control.
  • In the Allowed List, add your server’s public IP and loopback range followed by a trailing T (Trusted):
127.0.0.1, ::1, YOUR_SERVER_PUBLIC_IP T
  • Click Save and perform a Graceful Restart.
  1. Adjust Per-Client Throttling Rules
  • Go to Server Configuration > Security > Per Client Throttling.
  • Locate Connection Soft Limit and Connection Hard Limit.
  • Set Block Bad QoS to No or raise the limits for trusted IPs so the crawler’s request rate is not flagged as aggressive traffic.
  • Verify that Static/Dynamic Requests/sec limits are high enough to accommodate the crawler’s concurrent threads.
  1. Whitelist the Crawler User-Agent in LSCache Plugin
  • If using WordPress LSCache, log into the WordPress Admin Dashboard.
  • Navigate to LiteSpeed Cache > Crawler > General Settings.
  • Ensure Crawler Position is enabled.
  • Under LiteSpeed Cache > Crawler > Sitemap Settings, verify that the custom User-Agent matches the server-level whitelisted user-agent (default: lscache_runner).
  1. Exclude Crawler from ModSecurity / Comodo / Imunify360
  • Open your ModSecurity configuration file (typically located at /usr/local/lsws/conf/modsec.conf or managed via cPanel/DirectAdmin/CyberPanel).
  • Add a rule at the top of your custom ModSecurity rules file to bypass inspection for local loopback requests:
SecRule REMOTE_ADDR "@ipMatch 127.0.0.1" "id:999001,phase:1,nolog,allow"
SecRule HTTP_USER_AGENT "@contains lscache_runner" "id:999002,phase:1,nolog,allow"
  • Restart the web server to apply rule changes.
  1. Test and Confirm Resolution
  • Run the crawler manually via CLI or through the LSCache dashboard.
  • Tail the OpenLiteSpeed error log to verify that 403 status codes have stopped:
tail -f /usr/local/lsws/logs/error.log | grep "403"
  • Verify that the step was successful: The log output should show continuous 200 OK status responses for crawler requests instead of repeated 403 Forbidden errors.
See also  aaPanel Docker manager bridge network conflicting with local host IP subnets

Why This Fails for Some Users

  • Reverse Proxy / Cloudflare Layering: If the server is behind Cloudflare or an external reverse proxy (like Nginx), the crawler’s requests might loop externally and hit Cloudflare first. If Cloudflare blocks the server’s originating IP via Bot Management or Web Application Firewall, the request returns a Cloudflare-generated 403 error before reaching LiteSpeed.
  • CyberPanel / Control Panel Rule Overrides: Control panels like CyberPanel or cPanel often overwrite /usr/local/lsws/conf/httpd_config.conf or manage ModSecurity via panel-specific database flags. Direct edits to .htaccess or LiteSpeed config files can be silently overwritten during automated panel syncs.
  • IPv6 Loopback Mismatch: The server environment may resolve localhost to the IPv6 loopback address ::1, while the access control whitelist only includes the IPv4 address 127.0.0.1. If ::1 is not explicitly marked as trusted (T), the crawler will continue to hit 403 blocks over IPv6.

Leave a Comment