3 posts tagged with "Troubleshooting"

Every day around 20:08 the cPanel server rotates its domain logs (domlogs) — and the freshly created log file stays at 0 bytes for hours: the access-log evidence surface goes blind, and every fail2ban / traffic-analysis rule starves — while the panel shows everything as normal.

Encountered this while running a fully-managed cPanel/WHM hosting environment for a manufacturing-industry client on Alibaba Cloud China — recording the two-layer root cause and the permanent fix. Read the full engagement story in the managed-hosting case study.

TL;DR​

Two root causes stacked: cPanel's domlog rotation never reloads Apache, and the anti-blackout cron itself was written wrong — failing silently every night.

• Immediate recovery: apachectl graceful — domlog writing resumes within seconds
• Permanent fix: a two-line post-rotation cron — apachectl graceful (restore writing) + fail2ban-client -q reload (rebind jails to the new log file)
• Detection: ls -la /etc/apache2/logs/domlogs/<domain>-ssl_log — still 0 bytes one hour after rotation means you are hit

Symptoms​

After each daily rotation (~20:08 on this host), the site log (domlogs/<domain>-ssl_log) stays at 0 bytes for hours:

• First confirmed 09-27: new file at 0 bytes after rotation, until manual intervention
• Recurred 09-28: rotation at 20:07, still 0 bytes at 21:08 — the safety-net cron deployed for exactly this had no effect

The danger is the silence: no errors, no panel alerts — but fail2ban, traffic analysis, and intrusion detection all lose their data source. For a managed environment, that is a quiet collapse of the evidence surface.

Diagnosis: The Rotation-Window journalctl Is the Watershed​

Step 1: check what Apache did during the rotation window.

journalctl -u httpd --since "20:00" --until "21:30"

Zero activity — no reload, no restart around the rotation. That pins the mechanism: the rotation moved the old file and created a new one, but Apache was never told to reopen its log files.

Step 2: find out why the safety-net cron did not catch it. The anti-blackout cron fires nightly at 21:00. Running its command by hand exposed everything on the spot:

$ fail2ban-client reload -q
ERROR  No section: '-q'

Two mistakes surfaced at once:

1. Syntax: -q is a global option of fail2ban-client and must come before the subcommand (fail2ban-client -q reload); placed after reload, it is parsed as a jail name and the command errors out
2. Wrong target: even with correct syntax, fail2ban-client reload reloads fail2ban itself — it does not restore Apache's log writing. The breakage lives in Apache; what it needs is apachectl graceful

In other words: since the day it was deployed, this cron ran a command that was guaranteed to fail, never worked once, and raised no alert — the 09-28 blackout recurring as usual was its direct consequence.

Root Cause: Rotation Doesn't Reload, and the Net Was Tied Wrong​

The two layers, separated:

Root cause 1: cPanel domlog rotation does not reload Apache. The rotation = move the old file + create a new one, nothing more. Apache opens each log file once and keeps writing through its held file descriptor — which points at the old file's inode. After rotation, writes follow the old inode into the archive, and the new file sits unwritten from birth. This is not a malfunction; it is descriptor semantics — which is why something external must tell Apache to reopen its files (a reload) after every rotation.

Root cause 2: the safety-net cron got the command wrong twice over. The intent was right (reload after rotation), but the command was fail2ban-client reload -q: the misplacement of -q broke the syntax, and even fixed syntax would have reloaded the wrong daemon (fail2ban manages jails, not Apache's log handles). The combined effect: a safety net that silently did nothing — cron failures notify no one, the second silent point in this story.

The Fix: A Two-Line Cron, One Job Each​

Rewrite the safety-net cron (a custom file under /etc/cron.d/), split into two lines with separate responsibilities:

# Nightly 21:00 — restore domlog writing (after the ~20:08 rotation)
0 21 * * * root /usr/sbin/apachectl graceful
# Nightly 21:05 — rebind fail2ban jails to the new log files (-q is a global option; it precedes the subcommand)
5 21 * * * root /usr/bin/fail2ban-client -q reload

Design notes:

• 21:00 graceful: after the rotation (~20:08) and before the log-consumption peak, restores Apache's writes to the new file
• 21:05 fail2ban reload: deliberately 5 minutes apart — fail2ban's watched paths point at the rotated file, and a reload rebinds the jails; the offset avoids colliding with the Apache reload
• Manual verification (same night): after graceful, domlog resumed writing (922 bytes and climbing); fail2ban-client -q reload returned clean

The other failure classes from this machine's provisioning phase (TFA, resource 404s, domain mounting) are covered in AlmaLinux 10 cPanel: TFA Ineffective, Resource 404s, Domain Mount Refused — together the two posts form a full provisioning-troubleshooting picture.

FAQ​

Why is my cPanel domlog 0 bytes after daily rotation?​

Because cPanel's domlog rotation never reloads Apache. The server keeps its file descriptor on the old inode — new writes follow the rotated file into the archive — and nobody opens the freshly created log. Run apachectl graceful to restore writing immediately, then add a nightly post-rotation graceful cron as the permanent fix.

Where does the log content go after Apache log rotation?​

Old content lands in the rotation archive. The "empty new file" effect comes from the descriptor semantics: Apache holds one open file handle and keeps writing to the old inode no matter what happens to the filename. A graceful reload makes Apache reopen its log files — switching to the new inode — which is exactly why rotation must be followed by a reload.

How do I troubleshoot a crontab job that never runs?​

Three steps: crontab -l to confirm the entry exists; run the command by hand to see the real error (in this case fail2ban-client reload -q instantly failed with ERROR No section: '-q' — -q is a global option that must precede the subcommand); then check cron logs for trigger records. A syntactically broken command fails silently under cron — manual execution is the only reliable verification.

An AI model config in .env was changed — provider, key, and model name all swapped. Service restarted. The application's behavior did not move by an inch: still the old provider, the old model. The new config sits right there in the file, looking ready to apply at any moment. It never does.

Encountered this while maintaining a client-delivered AI content-processing application — recording the detour and the final diagnosis.

TL;DR​

When an app supports editing config through an admin UI, the database fully overrides .env — same-named entries in .env are a fallback at best, dead config at worst.

One sentence for the triage order: check the app's runtime config store (the DB config table) before checking .env. This investigation wasted two steps on .env and the process environment snapshot before the DB config table revealed the truth: it held a complete set of values from a different provider, shadowing .env in full.

Symptoms​

The server's .env:

# /path/to/backend/.env — looks "in use"
AI_API_KEY=sk-xxxx...xxxx
AI_BASE_URL=https://old-provider-compatible-endpoint
AI_MODEL=old-model-name

Actual behavior: model calls go to a different provider entirely (a real OpenAI-protocol endpoint, different key, different model). Edit .env, restart, no effect. Edit again, restart again, still no effect.

The Detour: Two Checks That Paid Nothing​

The investigation itself is worth recording — two checks that looked professional and accomplished nothing.

Detour 1: staring at the .env file. The config was complete, well-formatted, in the right directory (the systemd unit genuinely points there). Everything about it says "in use". But a config being in a file and a config being in effect are different things — the app's loader has a precedence chain, and .env is only one candidate source.

Detour 2: checking /proc/<pid>/environ. The standard move to see what environment a process actually received:

cat /proc/<pid>/environ | tr '\0' '\n' | grep AI_

Empty. And the check itself was flawed: dotenv injects into os.environ at runtime, after the process started — /proc/<pid>/environ is a snapshot from launch time and can never show dotenv-injected values. Absence here proves nothing about the process, and presence here proves nothing about effectiveness. For a dotenv app, this snapshot supports no conclusion in either direction.

The third step finally landed: querying the DB config table. The app supports editing AI config through an admin UI, and its loader is "DB first, env as fallback" — the DB config table held a full set of the other provider's values, outranking .env. Case closed.

Root Cause: A Config UI Requires DB > env​

Why must this class of app rank DB above env? Reverse-engineer from the feature requirement:

The app promises "admin edits AI config in the backend; it applies on save". If .env outranked the DB, every UI edit would be crushed by .env and the feature would be dead on arrival. So any app with this feature has a loader shaped like:

def _ensure_client():
    cfg = load_db_config()          # 1. DB config table first
    if cfg is None:                 # 2. fall back to env only when DB is empty
        cfg = from_env()
    return build_client(cfg)

DB wins whenever it has values — and the moment anyone saves a config in the backend once, the DB has values forever. From that day on, .env is dead config: nothing written there gets read, unless the DB config is cleared.

The trap is its invisibility: .env sits right there on the server, complete and tidy, and the natural operator reflex is "edit this". Nothing ever warns "your edit was overridden by the DB" — the config system silently follows its precedence chain, and only someone who knows the chain can predict the outcome.

The Fix: Check the DB First, Then Clean Up​

Step 1: confirm the real source of effective config. Query the app's config table (an app_config-style key-value table in this case):

SELECT key, value FROM app_config;

Match the DB values against observed behavior (e.g., which endpoint requests actually hit). Once they line up, the conclusion is nailed: DB fully overrides; the same-named .env block is leftover dead config.

Step 2: before deleting dead config, verify the DB really covers everything. Compare DB entries against .env item by item: if DB covers all critical fields, removing the .env leftovers should be harmless in theory — but restart and verify before deleting, in case some field in the loader still falls back to env. Verify, then delete: that is the safe order for dead-config cleanup.

Step 3: write the precedence chain into the ops doc. The two wasted steps trace back to one gap: nobody knew this app had config precedence at all. One line in the handover doc saves the next person two hours: "config lives in the admin UI (DB); .env is fallback only — check the DB before touching .env."

FAQ​

What is the config precedence between database and environment variables?​

In apps with an admin config UI, from lowest to highest: system-level env vars → process startup env → dotenv-injected values → the app's runtime config store (a DB or config file). The DB layer wins — UI edits must take effect immediately, which is impossible if .env outranks it. When troubleshooting, check the app's config store before the env files and you skip most of the detour.

Do environment variable changes require a restart?​

By layer: system-level variables need a shell or service restart; dotenv reads .env once at process start, so file edits need a process restart too; apps with a DB config layer pick up backend edits instantly — the loader reads the DB on every client build. Mixing the three timing models is how you end up concluding "changes do nothing".

Why can't I see dotenv variables in /proc/pid/environ?​

Because /proc/pid/environ is a snapshot from the instant the process started, while dotenv injects into os.environ after the process is already running. To verify dotenv's values, print them inside the process or read the .env file directly — and remember that a value being in the file still doesn't mean it's in effect; a higher-precedence layer may be overriding it.

While maintaining a 2-core 7GB cloud server for a client, I discovered disk usage at 93% (only 3GB free) and Airflow scheduler consistently consuming 160% CPU. Here's the complete diagnosis and fix.