10 posts tagged with "DevOps"

While debugging a workspace-mcp local service inside Claude Code, curl and lsof both came back looking perfectly "normal" — yet the service stayed unreachable. The later discovery was worse: binaries carrying the quarantine attribute hang forever inside the sandbox, which means every conclusion I'd drawn from sandboxed checks was untrustworthy.

TL;DR​

• On macOS, the Claude Code sandbox doesn't fail loudly — it silently returns plausible-looking empty results: curl connects to nothing without an error, lsof misses real listeners, and binaries with the quarantine attribute hang forever (0 CPU, no output).
• When diagnosing local services, negative results inside the sandbox (no listener, no process, hung command) are worthless. Rerun the same batch of commands with dangerouslyDisableSandbox to get conclusions you can trust.

Symptoms: a "clean" debug session, all wrong​

The contradiction: every check "returned normally" yet the service stayed down — the failure had two layers.

Layer one: regular diagnostics went quietly wrong. The workspace-mcp service was unreachable, so I ran the standard checks inside the session: curl 127.0.0.1:8000 showed no error, lsof -iTCP showed nothing unusual. By experience, none of the loud signals — connection refused, no listener — appeared. Every command "returned normally." But the service stayed down. I retried inside the sandbox several times; same result every time.

Layer two surfaced later and cut deeper: while looking into Syncthing, its core binary (carrying the com.apple.quarantine attribute) hung forever when executed inside the sandbox — the process existed, sat at 0 CPU, and printed nothing at all.

Both layers point at the same essence: diagnostic commands inside the sandbox lie. Errors make you alert; plausible negative results make you comfortable — and that's the dangerous form.

Why are Claude Code sandbox diagnostics unreliable on macOS?​

Root cause: the sandbox disguises blocking as normal. It distorts commands in three ways, in increasing order of severity.

Network probes go blind. curl 127.0.0.1:8000 inside the sandbox can't reach the port, but it doesn't throw a connection error either — it "returns normally." What you see is not connection refused but a clean empty result, and the wrong conclusion ("nothing is listening") gets written into your notes without a trace. The same applies to the phrasings "connection refused" or "localhost unreachable" — inside the sandbox, those symptoms get swallowed.

Process enumeration goes blind. lsof -iTCP inside the sandbox cannot see other processes' sockets on the host, so it prints nothing. The port is listening; you just can't see it. "No listener found" does not mean "nothing is listening," and any "the service isn't running" verdict built on it is wrong.

Binary execution hangs. The subtlest one. When you execute a binary carrying the com.apple.quarantine attribute (the Syncthing core), the sandbox policy stacks with the Gatekeeper check and the process stops dead at 0 CPU — no return, no crash, no clue.

The obvious escape hatch is welded shut, too. I tried xattr -dr com.apple.quarantine to strip the attribute; on every app under /Applications it returned EPERM — starting with macOS 27, the system locks the quarantine attribute on installed apps. That routine self-help path no longer exists on current macOS.

Fix: rerun the same commands outside the sandbox​

Conclusion first: on macOS, always verify local-service diagnostics (port listeners, processes, Gatekeeper, unified log, quarantined binaries) outside the sandbox.

The fix is mechanical — rerun the same batch of commands with dangerouslyDisableSandbox. Side-by-side:

Once outside, the earlier "everything looks fine" illusion collapsed — ports, processes and the unified log all came back with real data, and nearly every "negative" check from the first round got overturned.

The rule I now keep: negative results inside the sandbox don't count. This is not "sandbox output is garbage" — positive findings (a listener you did see, a process you did find) remain valid. What's broken is the empty results: they cannot distinguish "genuinely absent" from "the sandbox won't let you see." And sandbox-style guardrails distorting commands is a pattern, not a one-off — Claude Code's Hook interception falsely blocks normal commands the same way; your criteria need to anchor on execution semantics, not surface output.

Boundaries and caveats​

If you're building your own tooling around local MCP services like workspace-mcp, the approach in building a custom MCP toolkit with FastMCP pairs well with this — and when you wrap diagnostic commands as MCP tools, keep the sandbox layer in mind.

FAQ​

curl to 127.0.0.1 inside the Claude Code sandbox shows no error but never connects — is my service down?​

Probably not — the sandbox disguises blocking as clean empty results. On macOS, curl inside the sandbox fails to connect without an error, lsof returns nothing while the port is actually listening, and binaries with the quarantine attribute can hang forever (0 CPU, no message). Bottom line: never trust negative results inside the sandbox; rerun the same commands with dangerouslyDisableSandbox before concluding anything.

What does Claude Code sandbox mode silently block on macOS?​

Three things observed in practice: network probes (curl to localhost fails silently), process enumeration (lsof -iTCP can't see other processes' sockets), and binary execution (a binary with the com.apple.quarantine attribute, like the Syncthing core, hangs at 0 CPU with no error). You can't strip the quarantine attribute either — since macOS 27, xattr -dr returns EPERM for installed apps. These binaries only run outside the sandbox.

How do I safely bypass the Claude Code sandbox for local diagnostics?​

Rerun the same diagnostic commands with dangerouslyDisableSandbox. In my run everything came back real: syncthing --version returned in seconds, and ports, processes and the unified log were all visible. Bottom line: bypass for read-only diagnostics, keep everyday builds and execution inside the sandbox.

The .py file is gone from the codebase and deployed — yet the DAG still sits in Airflow's list. You clean the metadata tables by hand, and minutes later the rows are back.

Encountered this while building AI Ops — LLM-powered analytics that surfaces market trends, user behavior, and sales data for precise operational strategy. Retiring old analysis types in the DAG pipeline left zombie entries piling up in the list page.

Symptom: the file is gone, the metadata refuses to leave​

Two symptoms, depending on the order you did things:

• File deleted first, tables checked after: rows for the dag_id remain in dag, serialized_dag, dag_code and dag_version, and the UI list still shows the DAG;
• Tables cleaned first, file deleted after: stranger still — freshly DELETEd rows reappear within minutes. The DAG "resurrects".

If you searched for "Airflow delete DAG still shows in UI" or "dag metadata comes back", same story.

Root cause: dag-processor registers whatever it finds; reserialize never cleans​

The resurrection is designed behavior, not a haunting:

• dag-processor periodically scans the DAG directory. As long as the .py file exists, the scan parses it and re-registers metadata rows — which is exactly why cleaning tables before deleting the file resurrects the DAG: at clean time the file was still there, and the next scan rebuilt everything.
• airflow dags reserialize only registers upward. It serializes "files currently in the directory" into the tables, but never removes rows whose files have disappeared — run reserialize after deleting a file and the leftovers don't budge.

So the order of "delete file" vs "clean metadata" is inherently one-directional: while the file exists, cleaning is futile; once it's gone, one clean is final.

The fix: delete file → clean tables in FK order → reserialize to verify​

Three steps, order not negotiable:

1. Delete the .py file first. In production the DAG directory is a volume mount — sync via git pull on the host rather than deleting inside the container:

ssh <host> "cd /root/workspace/ai_dag && git pull"
# the DAG directory (mounted as /opt/airflow/dags) no longer has the .py

2. Clean the metadata in foreign-key order (task_instance goes with dag_run via cascade):

DELETE FROM dag_run        WHERE dag_id = 'old_dag';      -- cascades task_instance
DELETE FROM serialized_dag WHERE dag_id = 'old_dag';
DELETE FROM dag_code       WHERE dag_id = 'old_dag';
DELETE FROM dag_version    WHERE dag_id = 'old_dag';
DELETE FROM dag            WHERE dag_id = 'old_dag';

3. Run reserialize as the verification: after airflow dags reserialize, query the dag table — if the dag_id doesn't reappear, the cleanup is final.

The whole thing takes under a minute; with the file gone from the directory, dag-processor's scans have nothing to re-register.

Boundary cases​

• Make sure no scheduler/processor is mid-parse of that DAG before cleaning — avoid racing the registrar; off-peak windows keep it simple.
• If run history matters as evidence, export it with SELECT first — run history is the only record of "what actually happened"; once deleted it's gone.
• Same dag_id moving directories/platform folders follows the same logic: make the old file disappear from the directory first, then do the metadata surgery.
• Metadata schemas differ across Airflow versions (dag_code/dag_version are relatively recent) — \d your own tables and foreign keys before running DELETEs; don't copy table names blind.

FAQ​

Why does a deleted DAG still show in the Airflow UI?​

Its metadata rows remain in dag, serialized_dag, dag_code and dag_version, and the UI renders from tables. reserialize only upserts files that exist — it never removes rows for vanished files, so the tables need a manual DELETE in foreign-key order.

Why did the metadata I cleaned come back?​

The DAG file was still on disk when the tables were cleaned: dag-processor periodically scans the DAG directory and re-registers any file it finds. Delete the file first (git pull into the mounted directory), then clean tables, then reserialize to confirm.

What is the right order to fully remove an Airflow DAG?​

1. delete the .py file and sync the DAG directory; 2) DELETE in foreign-key order: dag_run (cascades task_instance) → serialized_dag → dag_code → dag_version → dag; 3) run airflow dags reserialize and confirm the dag table stays clean.

After updating a user password on Airflow 3, requests to /auth/token with the new password still return 401 — while the password hash in the database verifiably holds the new value.

Encountered this while building AI Ops — LLM-powered analytics that surfaces market trends, user behavior, and sales data for precise operational strategy. The platform's DAG trigger chain rides on this Airflow JWT auth, so a stuck password rotation left the pipeline authorization dangling.

Symptom: password changed, /auth/token still answers with the old one​

The new password gets 401 from /auth/token while the old one still gets 201 — every modification step "succeeded", yet the password in effect never changed. This particular rotation was incident response: the old credential had leaked, so every minute of "changed but not effective" was exposure.

The first reflex is the official CLI:

$ airflow users reset-password -u apiuser -p <new>
AttributeError: 'AirflowSecurityManagerV2' object has no attribute 'find_user'

The error points into the FAB (Flask-AppBuilder) auth stack, which smells like a 3.x version bug. CLI broken? Fine — bypass it and update the database directly. That detour set up an even deeper trap.

Root cause: SimpleAuthManager never reads the database — the password lives in passwords.json​

This deployment's auth manager is not the FAB the error implies, but SimpleAuthManager — one command reveals it:

airflow config get-value core auth_manager
# airflow.api_fastapi.auth.managers.simple.simple_auth_manager.SimpleAuthManager

Three layers, and the symptom explains itself:

• The CLI error is a red herring. The airflow users CLI family belongs to the FAB auth stack; on a SimpleAuthManager deployment its code path crashes with AttributeError. The AirflowSecurityManagerV2 class in the message does exist (as a leftover component), but it has nothing to do with the manager actually handling auth.
• ab_user is a leftover table. Deployments upgraded from Airflow 2.x carry FAB's user tables. Writing directly — UPDATE ab_user SET password=... with a werkzeug generate_password_hash scrypt hash — returned rowcount=1, and check_password_hash confirmed the stored hash matched the new password. Yet /auth/token kept answering 401: the table genuinely changed; the auth flow simply never reads it.
• The real source is a JSON file. SimpleAuthManager takes user passwords from the file pointed to by AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_PASSWORDS_FILE — a flat mapping:

{
  "apiuser": "<password>",
  "admin": "<password>"
}

The username-to-role mapping is declared separately via AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS (e.g. apiuser:admin,admin:admin). The first move in incidents like this should be asking "which manager owns auth?", not patching the symptom "auth is broken".

The fix: edit passwords.json, then force-recreate the container​

Three steps:

1. Edit the host-side passwords.json (the source file mounted into the container):

python3 - <<'EOF'
import json
d = json.load(open('/root/workspace/ai_dag/deploy/passwords.json'))
d['apiuser'] = '<new password>'
json.dump(d, open('/root/workspace/ai_dag/deploy/passwords.json', 'w'), indent=2)
EOF

2. Recreate the api-server container. This step is not optional — the password file is read only at startup, so editing it does nothing to the running process:

cd /root/workspace/ai_dag/deploy
docker compose up -d --force-recreate airflow-api-server

3. Once the service is up, verify in both directions — the new password must pass AND the old one must fail; both assertions matter:

# health: back to 200 within ~30s of the recreate
curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/api/v2/monitor/health

# new password → expect 201
curl -s -o /dev/null -w '%{http_code}' -X POST http://localhost:8080/auth/token \
  -H 'Content-Type: application/json' \
  -d '{"username":"apiuser","password":"<new>"}'

# old password → expect 401
curl -s -o /dev/null -w '%{http_code}' -X POST http://localhost:8080/auth/token \
  -H 'Content-Type: application/json' \
  -d '{"username":"apiuser","password":"<old>"}'

On this incident the results were 201 for the new password, 401 for the old — rotation closed. Verifying only "the new password works" without "the old one fails" is the most common loose end in credential rotation.

FAQ​

Why does the reset-password command fail after upgrading to Airflow 3?​

On 3.x deployments running SimpleAuthManager, the FAB airflow users CLI is gone for good — it crashes with AttributeError: 'AirflowSecurityManagerV2' object has no attribute 'find_user' (verified). Run airflow config get-value core auth_manager first to see which manager actually handles auth.

Where does SimpleAuthManager store user passwords?​

In a passwords.json file (flat JSON like {"username": "password"}) pointed to by AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_PASSWORDS_FILE. The file is read only at container startup — after editing it you must run docker compose up -d --force-recreate airflow-api-server.

Why doesn't updating the ab_user table change the password?​

ab_user belongs to the FAB auth manager. On deployments migrated to SimpleAuthManager it is leftover data the auth flow never reads — we wrote a scrypt hash with rowcount=1 and /auth/token still returned 401.

While configuring a Claude Code Bash hook to stop the AI from writing to a production database, pm2 restart, grep, and Python debug commands got blocked one after another — every dangerous SQL was caught, but half of the legitimate ops commands were caught too. This is commonly discussed as hook overblocking or false positives. This post walks through the full narrowing from "keyword matching" to "execution semantics".

Encountered this while operating the production servers behind CCLee Server Sentinel — managed Linux server monitoring across performance, availability, security and backups, with alerts merged into conclusions plus remediation guidance.

TL;DR​

Using "command text contains a write keyword" as the hook criteria is guaranteed to overblock: sys.path.insert, pm2 --update-env, and grep 'UPDATE …' all contain whole-word keywords, and \b boundaries save none of them. Narrow the criteria to three conditions — ssh prefix to the prod host + psql in the command + word-boundary keyword — and 8 classes of false positives drop to zero. Meanwhile, define the legit write path as file piping (keywords never enter the command line), which by construction never triggers the guard. Finally, turn the criteria table into 22 regression cases: change criteria by changing tests first.

Symptom: pm2 restart blocked, ops commands caught in the net​

On day one, the v1 blocking hook caught pm2 restart, remote grep, and Python debug commands — none of the blocked commands actually wrote a database.

Some context: our workflow lets Claude Code connect to the production database for read-only troubleshooting — locate the trace in the logs table first, form a hypothesis, then read code. That path depends on one premise: INSERT / UPDATE / DROP writes must never be issued directly by the AI. The rule went into CLAUDE.md first, but rules are advisory; models forget under long-session pressure. So we added a mechanical layer: a PreToolUse hook that blocks (exit code 2) any ssh-to-production command whose text contains a write keyword.

The false positives piled up fast. All of these were blocked; none of them writes a database:

# Python path tweak inside a container (sys.path.insert contains "insert")
ssh prod "docker exec app python3 -c 'import sys; sys.path.insert(0, \"/app/lib\")'"

# pm2 rolling env reload (--update-env contains "update")
ssh prod "pm2 restart api --update-env"

# Service restart (pm2 delete contains "delete")
ssh prod "cd /app && pm2 delete api 2>/dev/null; pm2 start ecosystem.config.js && pm2 save"

# Searching code remotely (grep's argument contains "UPDATE")
ssh prod "grep -rn 'UPDATE api_keys' /app/server/"

# Counting errors in logs
ssh prod "tail -5 error.log | grep -c UPDATE"

# pandas inside a container (df.update contains "update")
ssh prod "docker exec airflow python3 -c 'df.update(other)'"

Every blocked command meant Claude had to stop and reroute, breaking the troubleshooting chain. The false-positive list grew to 8 classes (the regression suite keeps 6 representative samples) — v1 criteria declared failed.

Root cause: keyword matching hits text, not execution semantics​

Word boundaries cannot fix shell-command false positives — v1's regex already had \b, and the false positives happened anyway:

grep -qiE '\b(INSERT|UPDATE|DELETE|DROP|TRUNCATE|ALTER|GRANT|REVOKE)\b'

Word boundaries cannot fix shell-command false positives, and the reason is counter-intuitive: punctuation in code syntax is itself a word boundary. The . in sys.path.insert, the - in --update-env, the . and () in df.update() are all non-word characters to the regex engine, so insert and update count as "whole words" in every one of them. v1's boundaries prevented zero false positives.

One level deeper, the criteria was anchored on the wrong thing: a keyword appearing in the command text does not mean the command executes that keyword's semantics. grep 'UPDATE api_keys' means "search", not "update"; pm2 --update-env means "restart", not "modify data". No text-level trick (boundaries, casing, context windows) can tell them apart, because the discriminating information is not in the text — it is in the executor. What is actually dangerous is "psql receives a write statement", not "the word UPDATE appears somewhere".

So the fix is not finer matching but a different anchor: execution semantics. A command poses a database-write risk only if three things hold at once: it is an ssh to the production host, it actually invokes psql, and the text psql will execute contains a write keyword.

The Claude Code hook decision flow: three conditions, all required​

The flow below shows the gate: three conditions in series, every "no" branch allows, only three "yes" results block.

The entire v1-to-v2 diff is one line — a psql condition added before the keyword check:

# v1: keyword only (deprecated)
if echo "$CMD" | grep -qiE '\b(INSERT|UPDATE|DELETE|DROP|TRUNCATE|ALTER|GRANT|REVOKE)\b'; then

# v2: psql presence + keyword, checked together
if echo "$CMD" | grep -qi 'psql' && echo "$CMD" | grep -qiE '\b(INSERT|UPDATE|DELETE|DROP|TRUNCATE|ALTER|GRANT|REVOKE)\b'; then

The full hook (17 lines; replace prod with your production host's ssh alias):

#!/bin/bash
INPUT=$(cat)
TOOL=$(echo "$INPUT" | jq -r '.tool_name // empty')
[ "$TOOL" != "Bash" ] && exit 0

CMD=$(echo "$INPUT" | jq -r '.tool_input.command // empty')

if echo "$CMD" | grep -qi '^ssh prod'; then
    if echo "$CMD" | grep -qi 'psql' && echo "$CMD" | grep -qiE '\b(INSERT|UPDATE|DELETE|DROP|TRUNCATE|ALTER|GRANT|REVOKE)\b'; then
        echo "Blocked: psql inline write keyword detected: $CMD" >&2
        exit 2
    fi
    echo '{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"allow","permissionDecisionReason":"read-only ssh command"}}'
    exit 0
fi

exit 0

Register it in settings.json, matcher limited to the Bash tool, timeout 5 seconds so a hung script cannot stall the session:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "/path/to/bash-guard.sh", "timeout": 5 }
        ]
      }
    ]
  }
}

Precise blocking is half the job — writes still need a path. Schema changes and data fixes are real work, and they should not die because a hook exists. Our legit path is file piping: SQL goes to disk, stdin feeds psql, and write keywords never enter the command line:

cat x.sql | ssh prod "docker exec -i db psql -U app -d appdb"
# or
ssh prod "docker exec -i db psql -U app -d appdb" < x.sql

A pleasant side effect: the legit path never triggers the hook, no whitelist or exemptions needed. Neither side of the pipe (cat, or ssh … psql without -c) contains a keyword — criteria and legit path are structurally disjoint rather than exception-enumerated. The rule worth writing down is: what is forbidden is not "writing the database" but "inlining keywords into the command line" — team convention and hook criteria now say the same sentence.

The criteria table is the test: 22 regression cases lock the contract​

A hook encodes criteria; criteria need regression tests. We wrote the criteria table directly as a test table — one case per row: command, expected outcome (2 = block, 0 = allow). Running it only verifies; it does not negotiate:

run_case 2 'inline INSERT'   $'ssh prod "docker exec -i db psql -c \'INSERT INTO customers ...\'"'
run_case 0 'pm2 restart --update-env' $'ssh prod "pm2 restart api --update-env"'
run_case 0 'remote grep keyword' $'ssh prod "grep -rn \'UPDATE api_keys\' /app/server/"'
run_case 0 'file pipe cat|' $'cat /tmp/x.sql | ssh prod "docker exec -i db psql"'
run_case 0 'cd && prefix bypass (permission layer backstop)' $'cd /x && ssh prod "psql -c \'DROP TABLE t\'"'
run_case 2 'read-only query with literal keyword (residual)' $'ssh prod "psql -c \'SELECT body FROM logs WHERE body LIKE \\'%UPDATE%\\'\'"'

The 22 cases split into four groups: 7 true blocks, 6 v1 false-positive fixes, 6 read-only and legit-path, 3 known residuals. A comment at the top of the test file reads: "the expectation column is the confirmed contract — execution only verifies, never negotiates". Criteria changes must edit the expectation column first, then pass the suite.

The suite repaid itself quickly. The heredoc policy was once recorded as "blocked as well", then reverted the same day — and the revert rationale is now unrecoverable. Documentation-based memory forgets; the criteria now has an arbiter: a local heredoc to disk (cat > x.sql <<'EOF', starts with cat, never enters the ssh prefix gate) is allowed; a remote heredoc (ssh prod "… psql …" <<'EOF', full SQL in the command line) is blocked. Both cases sit in the test table — run it and the answer is there.

Residual risks: what this line of defense does not catch​

• cd x && ssh prod "…" prefix bypass: the criteria anchors on the ^ssh prefix, so a command starting with cd slips through. The backstop for this is not the hook — Claude Code still asks for permission confirmation on commands outside the allowlist, keeping a human in the loop.
• Read-only queries with literal keywords still blocked: SELECT body FROM logs WHERE body LIKE '%UPDATE%' gets blocked. The rate is negligible and the failure direction is safe (prefer blocking), so we accept it.
• psql substring false positives: a path containing "psql" plus a keyword trips the guard, e.g. grep UPDATE /tmp/psql-dump.log.
• pg_restore not in the enumeration: restores go through a tool, not SQL keywords, so ssh prod "pg_restore -d appdb snap.sql" passes. The enumeration covers SQL write keywords; tool-level dangerous commands need separate coverage.

Why a hook instead of just a CLAUDE.md rule? They are not substitutes: rules are the advisory layer — models forget; hooks are the mechanical layer — they block even when the model forgets; permission prompts are the human layer — they backstop what the hook misses. Each layer covers a different failure mode; no single layer is enough. If you remember one sentence about the criteria: block execution semantics, not text presence.

This is our second Claude Code engineering post; the first was VS Code panel 500s and GLM multi-session rejection — both looked like random failures, and in both the root cause lived in the mechanics, not the model.

FAQ​

How do I add a Claude Code hook to block Bash commands?​

Register a PreToolUse command hook with matcher Bash in settings.json, pointing to your script with timeout: 5. The script reads tool_name and tool_input.command from stdin; exit 2 blocks the call and feeds stderr back to the model, exit 0 allows it.

What are best practices for writing Claude Code hooks?​

Anchor the guard on execution semantics, not text presence: this guard fires only when three conditions hold together — an ssh prefix to the prod host, psql in the command, and word-boundary write keywords. Keep a regression table (22 cases here) so every criteria change is verified, and document the residual risks you accept.

How do Claude Code PreToolUse hooks work?​

Before a Bash call executes, the hook receives the full command as JSON and answers with an exit code: exit 2 blocks the call and sends stderr back to the model as the reason; exit 0 allows it. A bare exit 0 with no output passes silently.

Why is my Claude Code hook blocking harmless commands?​

Because keyword matching hits text, not semantics: sys.path.insert, --update-env and grep 'UPDATE' all contain whole-word keywords, and word boundaries cannot help since punctuation in code is itself a boundary. v1 blocked 8 classes of legitimate commands this way; adding the psql condition cleared all of them.

The alert email: production server root partition at 81% (30G/40G, past the 80% red line). First instinct: check docker system df — and there it is, images at RECLAIMABLE 100% (7), apparently a quick win. Except all 7 images are running; following the hint with docker image prune -a would be a production incident.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the disk in question belongs to the Docker server running its Airflow data pipeline.

TL;DR​

Past a disk red line, don't trust docker system df RECLAIMABLE — it's an estimate of "space with no container reference", and active images get flagged 100% anyway. The right move: sudo du -xh -d1 / layer by layer. This incident's three invisible hogs were none of them business data: task logs accumulating in container writable layers (no log volume in the Dockerfile), package-manager caches (npm + pnpm, ~4G combined), and a backup script that never pruned (full clones every run). Fixes: delete caches, reclaim writable layers with force-recreate, add TTL pruning to the backup script — 81% back down to 64%.

Symptoms​

$ df -h /
Filesystem      Size  Used Use% Mounted on
/dev/vda1        40G   30G  81% /

$ docker system df
TYPE        TOTAL   ACTIVE   SIZE     RECLAIMABLE
Images      7       7        4.2GB    100% (7)    ← all running
Containers  5       5        810MB    0%

Acting on docker system df (clean images) is a dead end — RECLAIMABLE says 100% but ACTIVE says 7/7. Where the space actually went, docker system df never shows:

sudo du -xh -d1 / | sort -rh | head
# selected output:
# 2.7G    /root/.npm              ← npm cache
# 1.1G    /root/.local/share/pnpm ← pnpm store
# 857M    /root/backups-git       ← backup clones, never pruned
# (hidden in overlay2: airflow scheduler writable layer 468M + dag-processor 257M)

Root Cause​

Three kinds of consumption, all invisible from the "business data" perspective.

Writable layers eating logs. Airflow task logs are written inside the containers with no external volume — scheduler writable layer 468M, dag-processor 257M, ~780M accumulated in two weeks, monotonically growing. Image layers never change; the writable layer does. docker system df buries it in the Containers SIZE total (810M), where it has no presence.

Package-manager caches only grow. On a server with frequent deploys/builds, ~/.npm (2.7G) and the pnpm store (1.1G) pile up indefinitely; nobody ever cleans them.

A backup script that never prunes. The backup script clones the repo worktree in full and pushes to GitHub every run — old clone directories are never removed, so 857M is mostly historical duplicates. Sneakier: orphan clones whose branch never pushed successfully can't just be deleted; verify first.

And docker system df RECLAIMABLE is a statistical estimate of "space not referenced by a running container" — active images can display 100% reclaimable (all 7 of ours did). It's a false-positive generator, not a cleanup guide.

Solution​

Step 1: Locate with du, layer by layer — look before deleting​

sudo du -xh -d1 / | sort -rh | head        # root partition, level by level
sudo du -xh -d1 /var/lib/docker | sort -rh | head   # drill into Docker's dir
docker ps -as --format "table {{.Names}}\t{{.Size}}"  # per-container writable layer

du -x stays on one filesystem, avoiding /proc, /sys noise and overlay confusion; docker ps -as SIZE exposes each container's writable layer — the key command for the "logs written inside the container" family.

Step 2: Delete caches outright​

npm cache clean --force        # or simply rm -rf ~/.npm/_cacache
pnpm store prune               # removes only unreferenced packages

~5.1G reclaimed, zero risk — caches re-download on demand.

Step 3: Reclaim writable layers with force-recreate​

docker compose up -d --force-recreate   # writable layer goes away with the old container

~780M reclaimed here. Two preconditions: pick a low-traffic window (services restart), and rescue anything valuable first — e.g. docker cp the task logs out, or they vanish with the container.

Step 4: TTL for backups, prevent recurrence​

# keep backup dirs for 7 days, prune older ones
find /root/backups-git -maxdepth 1 -type d -mtime +7 -exec rm -rf {} +

Add the pruning to the tail of the backup script (deployed in our github-backup-push.sh); before deleting, check for orphan clones whose branch never pushed (git ls-remote to compare) and confirm no unique commits — our 325M of orphans were removed only after verification.

After cleanup: 81% → 64%, with caches + pruning + writable layers together reclaiming ~6.7G.

FAQ​

docker system df shows 100% RECLAIMABLE — can I just delete?​

No. RECLAIMABLE estimates "space with no container reference", and running active images can still be flagged 100% reclaimable — all 7 of ours were. It answers "how much is theoretically unreferenced", not "what can be deleted". Measure first with du -xh -d1 and docker ps -as.

What's the difference between docker ps SIZE and docker system df SIZE?​

docker ps -as SIZE is the per-container writable layer; docker system df is the category-level aggregation over images/containers/volumes/cache. Logs accumulating inside a container only ever appear in the writable layer (visible via docker ps -as), never in any image size — which is exactly why "images look small but disk keeps growing".

How do I clean up a Docker server running out of disk?​

Three categories: package-manager caches deleted outright (npm cache clean --force, pnpm store prune), zero risk; container writable layers reclaimed via docker compose up -d --force-recreate (service restart; rescue logs first); backups, logs, and clone directories put on TTL pruning to prevent recurrence. Before any deletion, confirm the hogs with du -xh -d1 /.

While deploying a multi-package project (frontend and backend sharing one repository), npm audit reported 3 high vulnerabilities in the deploy log. We logged them against the backend — only to confirm later that all 3 highs lived in the frontend tree, and the backend had been a separate set of moderates all along.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the frontend lives at the repository root and the backend in server/, deployed separately from one repo.

TL;DR​

npm audit summary lines carry only counts, no path. When a deploy pipeline runs npm install in several directories in sequence, the outputs interleave and the summaries become unattributable. The fix: use each package tree's unique fingerprint — the N in audited N packages — to attribute findings first, then pin the vulnerable transitive dependencies with overrides in package.json. A second deploy brought both sides to zero.

Symptoms​

The deploy script runs npm install in the frontend (repo root) and then the backend (server/), and both outputs land in the same log stream:

# Deploy log (summary lines carry no path)
added 546 packages in 41s
found 3 high severity vulnerabilities

added 372 packages in 24s
found 4 moderate severity vulnerabilities

Our handover notes attributed the "3 high" to server/. Chasing the backend dependency chain led nowhere — the server/ audit, run locally or on the server, consistently showed 4 moderates and never a single high. "Visible in the log, unattributable in practice" is a chronic disease of mixed deploy pipelines; our earlier write-up on stale build artifacts after deployment is the same family of problem.

Root Cause​

The npm audit summary only prints "found X vulnerabilities" with no directory info, and the adjacent audited 546 packages rarely registers as an attribution clue. Two package trees with wildly different sizes (546 vs 372) turn out to be the only stable fingerprint.

The frontend root installs the full Vite + React + Ant Design Pro stack, so its tree is large; @ant-design/pro-components → @ant-design/pro-layout pulls in an old path-to-regexp, which is exactly where the 3 highs came from. The backend server/ is a lean Express + tsx tree whose only issue is the tsx → @esbuild-kit/core-utils → old esbuild chain of moderates.

Solution​

Step 1: Attribute findings by audited N​

Run npm install in each directory locally (or read added N packages straight from the deploy log) and record the tree sizes:

cd <repo-root> && npm install 2>&1 | tail -2   # added 546 packages ...
cd server      && npm install 2>&1 | tail -2   # added 372 packages ...

In the deploy log, found 3 high immediately follows added 546 → frontend; 4 moderate follows added 372 → backend. Get attribution right before touching any dependency.

Step 2: Expand the vulnerability chain​

npm audit                # see the Path field, full chain
npm ls path-to-regexp    # or reverse-lookup who depends on a package

The frontend output confirmed the chain: @ant-design/pro-components → @ant-design/pro-layout → path-to-regexp (old version, 3 high).

Step 3: Pin the transitive dependency with overrides​

Frontend root package.json:

{
  "overrides": {
    "path-to-regexp": "^8.4.2"
  }
}

Backend server/package.json (with tsx bumped to a newer release):

{
  "overrides": {
    "@esbuild-kit/core-utils": {
      "esbuild": "^0.25.12"
    }
  }
}

overrides supports nested syntax, scoping the pin to the child dependency under one specific parent — more surgical than overriding a package name globally.

Step 4: Reinstall and verify​

rm -rf node_modules package-lock.json && npm install && npm audit

After redeploying both sides, audit reports 0 vulnerabilities on each.

FAQ​

Why is npm audit fix not working?​

npm audit fix only upgrades versions inside the allowed semver range. When the vulnerability lives in a transitive dependency whose version is pinned by an upstream package's range, fix cannot touch it. Use overrides in package.json to force the version, then reinstall.

How do I find which dependency chain an npm audit finding comes from?​

Read the Path field in the full npm audit output — it lists the complete chain from a direct dependency down to the vulnerable package. You can also reverse-lookup with npm ls <package>. Summary lines carry counts only.

How do I map npm audit results to a specific subproject?​

Deploy log summaries carry no directory name, so match them by each directory's added N packages / audited N packages tree size. Run npm install once per directory locally, record each N, and the mapping stays stable.

Chasing a monitoring gap: server-monitor filters alert-grade logs by the level name warn, but 38 rows in the shared logs table carried level='warning'. The filter didn't match by one character — and in the monitor's eyes, those 38 rows did not exist.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; server-monitor is its alerting module, consuming one logs table written by four services.

TL;DR​

The cross-language logging contract defines lowercase warn/fatal; Python's stdlib record.levelname produces WARNING/CRITICAL — written raw or with a bare .lower() you get warning/critical, which contract-name filters never match. Two principles fix it: normalize at a single point in the write boundary (WARNING→warn, CRITICAL/FATAL→fatal) so no consumer ever has to juggle spellings; and write the contract doc as the intended implementation, not a snapshot of the current one — this drift survived so long precisely because the contract's Python column documented the buggy code.

Symptoms​

Four services write one logs table; the contract specifies level values: debug / info / warn / error / fatal. A reconciliation query:

SELECT service, level, count(*)
FROM logs
GROUP BY service, level ORDER BY 1, 2;

turns up spellings that don't exist in the contract:

 service    | level    | count
------------+----------+-------
 ai-dag     | warning  |    21   ← not in the contract
 rag-service| warning  |    17   ← not in the contract
 ...        | warn     |   ...   ← the actual contract name

The monitor filters level = 'warn'; these 38 alert-grade rows silently vanish.

Root Cause​

Layer one is literal mismatch: Python's stdlib levels are DEBUG / INFO / WARNING / ERROR / CRITICAL — there is no WARN (a deprecated alias) and no FATAL. Both services pushed record.levelname into the table: one raw (uppercase WARNING), one lowercased (warning). Neither matches the contract's warn.

Layer two is the one worth losing sleep over: the contract document itself specified the wrong implementation. In the cross-service contract's field table, the Python services' level column literally read "record.levelname" and "record.levelname.lower()". The doc was describing reality instead of prescribing it — so the buggy implementations carried the contract's endorsement, and nobody questioned them. This is the same harm shape as try/except swallowing exceptions into silent failures: nothing crashes, things just quietly go missing — by the time anyone looks, dozens of alert rows were never seen.

Solution​

Step 1: Single-point mapping at the write boundary​

Each service defines one normalization function; every write path (formatter and DB sink) goes through it:

_LEVEL_NAME_MAP = {"WARNING": "warn", "CRITICAL": "fatal", "FATAL": "fatal"}

def normalize_level(levelname: str) -> str:
    """WARNING→warn, CRITICAL/FATAL→fatal, everything else lowercased."""
    return _LEVEL_NAME_MAP.get(levelname.upper(), levelname.lower())

payload = {"level": normalize_level(record.levelname)}   # always emits a contract name

The keyword is "single point": the JSON formatter and the DB handler share one function, so the mapping changes in exactly one place and no second implementation can appear.

Step 2: Rewrite the contract doc as the intended implementation​

The field table's Python columns now read normalize_level(record.levelname), and a new "level name mapping" section documents the rules, the anti-patterns (no raw writes, no bare lowercasing), and each service's function entry point. A contract is a spec — not a snapshot of whatever happens to be deployed.

Step 3: Add a reconciliation query so drift is discoverable​

SELECT level, count(*) FROM logs
WHERE service IN ('ai-dag', 'rag-service')
GROUP BY level ORDER BY 2 DESC;

Any spelling besides warn is drift. This query belongs in routine inspection, turning "contract vs implementation" from a verbal promise into an assertable check.

Step 4: Clean up存量 (optional)​

New writes no longer produce off-contract names; handle the existing 38 rows as needed:

UPDATE logs SET level = 'warn' WHERE level = 'warning';

Small volumes can be left to age out; large ones, or anything feeding historical statistics, deserves the UPDATE.

FAQ​

Why can't Python's WARNING be written straight into the logs table?​

The stdlib literals are WARNING/CRITICAL, while cross-language contracts define warn/fatal. Raw or lowercased values (warning/critical) are unknown levels in contract-land — every consumer filtering by contract names silently drops them. The Python side must map at the write boundary.

Are WARNING and WARN the same level?​

Same semantics, different literals. Python logging has no WARN level (deprecated alias; the emitted name is always WARNING) and no FATAL (it's CRITICAL). That's why lowercasing doesn't help — you need an explicit mapping: WARNING→warn, CRITICAL→fatal.

How do I detect that a logging contract and its implementation have drifted?​

Periodically reconcile by contract level names (GROUP BY level); any off-contract spelling is drift. More importantly, the contract doc should specify the intended implementation and name the mapping function — when the doc copies the current implementation, the bug gets an endorsement, which is exactly why this drift survived so long.

After switching an ads data source from daily tables to weekly tables, I found an empty table still sitting in the schema — one that was only ever CREATEd in the earliest migration, held zero rows, and was never referenced by runtime code. It even carried stale field names and unnormalized columns.

Encountered this while building AI Ops — AI-powered analytics that surfaces market trends, user behavior, and sales data to drive precise operational strategy. After this data-source switch, the write path had long since moved to the new weekly tables and a cleanup migration had already dropped the old daily table. The one thing missed was a monthly table that existed only in the baseline CREATE — it had no new writes, no corresponding DROP, and just sat dormant in the schema with its deprecated field definitions.

TL;DR​

The signature of an orphan table: only CREATEd in an early/baseline migration, zero references in current code, often with stale or unnormalized column names. Batch migrations don't touch these automatically. You have to actively list every table in the schema with information_schema.tables, compare against code references, identify the orphans, and write a DROP migration to clean them up — not just psql your way to a one-off delete.

The symptom​

A typical orphan table looks like this:

• Zero rows — the business stopped writing to it long ago;
• Zero runtime references — no SELECT/INSERT anywhere in code, only the CREATE in a migration file;
• Stale fields — column names from a previous naming convention (e.g. ad_plan_id/product_id), out of step with current standards;
• Unnormalized columns — possibly even Chinese column names that were never cleaned up.

It doesn't crash and doesn't affect production, so from a "nothing's broken online" perspective it's invisible. But the harm is implicit: it misleads newcomers into thinking it's still in use, pollutes the schema namespace, adds noise to cross-table audits, and could be read as dirty data by some mistaken SELECT *.

Root cause​

Database migrations follow a pervasive pattern: migrations are "additive".

A data-source switch typically evolves like this:

1. An early baseline migration CREATEs a batch of tables (daily, monthly);
2. Once the business runs, the write path starts depending on them;
3. Requirements change, new tables (weekly) are introduced, and writes migrate over;
4. The old daily table's writes stop, and a migration DROPs it;
5. But the monthly table (or any table that was only ever CREATEd in the baseline and never directly used by the write path) gets no corresponding DROP.

The problem is step 5: migration attention focuses on "tables in use right now" — which ones are being written, which ones queries hit. A table that "once existed but never entered the main path" is in neither the write path nor the query path, so it never triggers a DROP and becomes an orphan. This is the same family as Airflow DAG metadata lingering after deletion: "removed the entry, forgot to clean the structure" — a high-frequency failure mode in migration work.

The fix​

Core flow: list all tables → compare references → confirm empty → write a DROP migration → verify.

Step 1: list all base tables in the schema with information_schema​

-- List all base tables in a schema (exclude views)
SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'your_schema'
  AND table_type = 'BASE TABLE'
ORDER BY table_name;

information_schema.tables is a SQL-standard catalog view, portable across PostgreSQL/MySQL/SQL Server with stable fields — ideal for baking into an audit script.

Step 2: grep the codebase to confirm runtime references​

For each candidate table, search for references in the codebase, excluding migration files themselves:

# Search runtime code references, excluding the migrations directory
grep -rn "ad_product_monthly_stats" src/ --include="*.py" \
  | grep -v "migrations/"
# 0 lines of output → no runtime reference, it's a candidate

Zero references is the key evidence for an orphan. Make sure to exclude the migration directory — the CREATE in the baseline doesn't count as a "reference".

Step 3: confirm it's empty​

SELECT count(*) FROM your_schema.ad_product_monthly_stats;
-- 0 → confirmed no data, safe to clean up

Be extra careful with tables that have data: first confirm they're truly abandoned (not just recently unwritten), and back up logically if in doubt.

Step 4: write a DROP migration (not a manual delete)​

-- db-migrations/{project}/027_drop_ad_product_monthly_stats.sql
DROP TABLE IF EXISTS your_schema.ad_product_monthly_stats;

Always go through a migration file: it's version-controlled, replayed consistently across environments (dev/staging/prod), and leaves an audit trail. A one-off psql delete only works on the current machine — on another box, the table grows back.

Step 5: verify the drop​

SELECT to_regclass('your_schema.ad_product_monthly_stats');
-- Returns NULL → the table no longer exists

to_regclass() is the standard way to check whether a relation exists; NULL confirms the drop succeeded.

Batch audit: sweep same-prefix siblings at once​

After dropping one table, list all same-prefix siblings and walk through each — avoid "dropped one, missed its siblings":

-- List all tables under a prefix, run steps 2-5 on each
SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'your_schema'
  AND table_name LIKE 'ad_%'
ORDER BY table_name;

FAQ​

How do I list all tables in a PostgreSQL database?​

Query information_schema.tables, filtering by table_schema and table_type = 'BASE TABLE' to list all base tables in a schema. It's more scriptable than psql's \dt, and because it's the SQL standard, the same query is portable across databases.

How do I find unused or orphan tables in PostgreSQL?​

List all tables with information_schema.tables, then compare against references in your codebase or query logs. Tables with zero runtime references and no writes are orphan candidates; confirm the row count with SELECT count(*), and once you've verified no data and no foreign-key dependencies, write a DROP migration to clean them up.

What's the difference between information_schema and pg_catalog?​

information_schema is the SQL-standard catalog view — portable across PostgreSQL/MySQL/SQL Server with stable fields, ideal for portable audit scripts. pg_catalog is the PostgreSQL-specific catalog, richer and more detailed (e.g. precise row-count estimates, storage details) but subject to change between versions. For portable schema audits, prefer information_schema.

After deleting a DAG's .py file in Airflow to retire it, the dag_id still hangs around in the Web UI list and the database; even stranger — if you clear the metadata first and delete the file second, the just-cleared rows "come back to life."

Encountered this while building AI Ops — an LLM-powered analytics pipeline where retiring an old report DAG required cleaning its metadata too, otherwise the UI list and scheduled scans stayed polluted by residual rows.

TL;DR​

Airflow's dag-processor periodically scans the DAG folder and re-registers DAGs, and airflow dags reserialize doesn't purge "file-already-deleted" orphan rows — so just deleting the .py file won't make the dag_id vanish from the UI or DB. Conversely, clearing metadata before deleting the file lets the processor re-register the cleared rows on its next scan ("revival"). Correct order: ①delete the file first so the processor stops registering → ②SQL DELETE in foreign-key order → ③run airflow dags reserialize to verify.

Symptoms​

Retiring the shop_report_aggregation DAG — after deleting its .py file:

$ ls /opt/airflow/project/airflow_dags/shop_report_aggregation.py
ls: cannot access '.../shop_report_aggregation.py': No such file or directory

$ # but it's still in the database
$ docker exec cclhub-db psql -U airflow -d airflow -c \
    "SELECT dag_id, is_paused, is_active FROM dag WHERE dag_id='shop_report_aggregation';"
       dag_id            | is_paused | is_active
--------------------------+-----------+-----------
 shop_report_aggregation | f         | t          ← still there

It's not just the dag table — the matching rows in serialized_dag, dag_code, and dag_version are all still there, so the Web UI keeps showing this "deleted" DAG.

Worse is the reverse order — clear metadata first, delete file second:

T0  DELETE FROM dag WHERE dag_id='shop_report_aggregation';   ← cleared
T1  (.py file not deleted yet)
T2  dag-processor scan fires; file exists, dag table has no row → re-registers
T3  SELECT ... FROM dag WHERE dag_id='shop_report_aggregation';   ← it's back (revival)

Root Cause​

Two mechanisms stack up:

1. dag-processor scans and re-registers periodically. Airflow's dag-processor (part of the Scheduler) scans dags_folder on processor_poll_interval (default ~5 min), parses each .py file, and upserts into the metadata tables (dag, serialized_dag, dag_version). As long as the file exists, the next scan rewrites those rows. That's the direct source of "revival" — you clear the row, the file is still there, and the processor re-registers it as a new DAG.

2. reserialize ignores "file-gone" orphan rows. airflow dags reserialize re-serializes existing DAG files and refreshes serialized_dag; it does not delete orphan dag rows whose files have vanished. And airflow dags cleanup only purges expired dag_run history by default — it also leaves the dag / serialized_dag / dag_code / dag_version metadata tables alone. So after you delete the file, the metadata rows become orphans nobody cleans.

┌─ dag-processor ──────────────────────────────┐
│  scans dags_folder                            │
│  ├─ file present → upsert dag / serialized... │  ← source of revival
│  └─ file absent  → skip, no row deletion      │  ← orphan residue
└───────────────────────────────────────────────┘

Conclusion: to actually remove the metadata, you must give the processor no file to register (delete the file first), then manually clean the residual rows.

Solution​

Step 1: Delete the file first​

Make the .py file disappear from the DAG folder so dag-processor stops registering it.

# In production this is usually synced to the volume-mounted DAG folder
# /opt/airflow/project/airflow_dags/ via git pull
git pull   # removes shop_report_aggregation.py from the repo and the folder

# or delete directly (after confirming nothing depends on it)
rm /opt/airflow/project/airflow_dags/shop_report_aggregation.py

Step 2: Clean metadata in foreign-key order​

DELETE in foreign-key dependency order to avoid constraint violations. Deleting dag_run CASCADEs to task_instance:

BEGIN;

-- 1. run history (CASCADEs to task_instance)
DELETE FROM dag_run      WHERE dag_id = 'shop_report_aggregation';

-- 2. serialized DAG
DELETE FROM serialized_dag WHERE dag_id = 'shop_report_aggregation';

-- 3. version
DELETE FROM dag_version  WHERE dag_id = 'shop_report_aggregation';

-- 4. dag main table
DELETE FROM dag          WHERE dag_id = 'shop_report_aggregation';

-- 5. dag_code is keyed by source hash; multiple DAGs may share the same code;
--    only delete hashes no longer referenced by any serialized_dag
DELETE FROM dag_code
WHERE dag_hash NOT IN (SELECT dag_hash FROM serialized_dag);

COMMIT;

Step 3: Verify​

airflow dags reserialize

# confirm the dag row is not rebuilt
docker exec cclhub-db psql -U airflow -d airflow -c \
  "SELECT count(*) FROM dag WHERE dag_id='shop_report_aggregation';"
#  count
# -------
#      0   ✅

After reserialize, dag / serialized_dag / dag_code / dag_version are all 0 for that dag_id, and the next processor scan doesn't rebuild them — the cleanup is stable.

As a side note, on the same pipeline, pandas NaN crashing XCom serialization is another pitfall worth bookmarking.

Notes​

FAQ​

Why does a DAG still show in Airflow after deleting its .py file?​

Deleting the file doesn't clean the database. Rows in dag / serialized_dag / dag_code / dag_version still exist, and the Web UI reads those tables to render the list, so the deleted DAG keeps showing. Airflow has no built-in command to purge these orphan rows automatically; you must SQL DELETE them manually in foreign-key order.

How do I completely delete an Airflow DAG and all its metadata?​

Three steps: 1) delete the .py file so dag-processor stops registering it; 2) SQL DELETE in foreign-key order (dag_run → serialized_dag → dag_version → dag → orphan dag_code); 3) run airflow dags reserialize, then query the dag table to confirm the dag_id row count stays at 0 and isn't rebuilt.

What's the correct order to clean Airflow DAG metadata, and why not clear metadata before deleting the file?​

Delete the file first, then clean metadata. If you reverse it, the .py file still exists, so dag-processor re-registers the cleared dag row on its next scan — the metadata "comes back to life." Only by making the file vanish first (so the processor has nothing to register) and then cleaning the residual rows can you fully retire the DAG.

Encountered this while building AI Ops — LLM-powered analytics that surfaces market trends, user behavior, and sales data for precise operational strategy.

TL;DR​

dotenv treats any # in unquoted values as an inline comment. KEY=value#hash is actually loaded as value, with #hash dropped — no warning, no error. The fix has two steps: wrap any .env value containing # in double quotes, then run pm2 restart <app> --update-env to force-refresh the process environment — without the refresh the edit changes nothing, because PM2 re-injects its cached snapshot into the new process.

Symptom​

Backend calls to an upstream service keep returning 401 Invalid credentials:

POST /api/v1/dag/trigger → 500
Stack: Airflow JWT auth failed (401): {"detail":"Invalid credentials"}
  at getJwtToken (airflow-client.ts)

My first assumption was a wrong password or a disabled account. Investigation shows the password written in .env is 24 chars and contains # and &:

AIRFLOW_PASSWORD=Aq7#mZx&V3nKp9RtWu2yBc4d

But the value loaded into process.env.AIRFLOW_PASSWORD is only 3 chars long — the 21 characters after # are gone. Calling the upstream auth endpoint with the full password via curl returns 201; calling it with the truncated value parsed from .env returns 401. The credential is fine; the value loaded from .env is truncated. If you searched for ".env not loading", "environment variable value wrong" or "password is correct but auth fails", it's all the same root cause.

Root cause: dotenv treats any # in unquoted values as an inline comment​

When dotenv parses a .env file, an unquoted value ends at the first # — everything from # onward is discarded as an inline comment, with no warning at all. This behavior is documented, but the silence is what makes it nasty — no error, no warning; newer versions print a one-line injected-env summary at startup (verified on 18.0.4), but it only shows how many keys were loaded.

I ran the same parse matrix against dotenv 16.6.1 (the version pinned in the project) and 18.0.4 (current latest at the time); the results were identical:

Three things stand out:

• No space needed. In a shell, # only starts a comment when preceded by whitespace. dotenv doesn't care — val#hash with the # glued to the value is truncated all the same. Strong-random strings like JWT_SECRET, API_KEY, and DATABASE_URL frequently contain # in arbitrary positions — high-risk territory.
• Quotes are the literal switch. Inside single or double quotes, # is kept as a plain character; a # after the closing quote still starts a comment.
• & and spaces are safe in dotenv. Neither triggers truncation in the tests, and core dotenv does not expand $VAR (that's the dotenv-expand plugin's job).

The fix: quote the value, force-refresh the process environment​

First, wrap any value containing # in double quotes:

# truncated: the process actually gets Aq7
AIRFLOW_PASSWORD=Aq7#mZx&V3nKp9RtWu2yBc4d

# correct: preserved verbatim
AIRFLOW_PASSWORD="Aq7#mZx&V3nKp9RtWu2yBc4d"

Second, restart with --update-env:

pm2 restart analytics-api --update-env

That flag is not optional, because two "don't overwrite" defaults stack up:

1. PM2 snapshots the environment when a process is first started and injects that stale snapshot back into every restart;
2. dotenv does not overwrite keys that already exist in process.env (verified: set process.env.X='oldvalue', run dotenv.config(), and X is still oldvalue).

Editing the file without refreshing the process environment changes nothing — the process keeps reading PM2's cached snapshot. The same applies to docker compose and systemd: the service must actually rebuild its environment (docker compose up -d --force-recreate, systemctl restart).

After restarting, verify what the process actually loaded instead of guessing:

# print the length and compare it against the value in .env
node -e "console.log(process.env.AIRFLOW_PASSWORD.length)"

To go one step further, validate critical variable lengths at startup so a silent failure becomes a startup failure:

// Validate critical env vars at startup to catch truncation early
const required = ['AIRFLOW_PASSWORD', 'JWT_SECRET', 'DATABASE_URL'] as const;
for (const key of required) {
  const v = process.env[key];
  if (!v || v.length < 16) {
    throw new Error(`${key} not loaded correctly (length ${v?.length ?? 0}); check .env quoting`);
  }
}

Boundary cases: docker compose plays by different rules​

Truncation at # is not a universal env-parser convention — docker compose follows the opposite rule, and the same file can load differently across tools. The compose-spec states plainly: "Inline comments for unquoted values must be preceded with a space", and its official example shows VAR=VAL# not a comment loading as VAL# not a comment, kept verbatim.

For dotenv alone, only # is dangerous; but one .env file often serves several tools — local shell, docker compose, PM2, CI — each with its own parser. Rather than memorizing every parser's quirks, our team settled on one hard rule: any value containing #, & or spaces gets double quotes. Two extra characters buy cross-tool predictability.

FAQ​

Why does a password with # in .env get shorter?​

dotenv treats any # in unquoted values as an inline comment and drops everything after it — KEY=value#hash loads as value, with no warning (verified on dotenv 16.6.1 and 18.0.4). Wrap the whole value in double quotes to preserve it.

How do I debug dotenv not working?​

Three steps: first confirm dotenv.config() runs before all imports (ES Module imports are hoisted statically — see debugging silent JWT signature failures); then check .env values for unquoted #; finally print process.env.XXX and diff its length against the .env source file — a mismatch means truncation, not a missing load.

Does docker compose treat # the same way in env files?​

No — the rules differ. compose-spec requires inline comments in unquoted values to be preceded by a space, so VAL=B#C is kept as-is; dotenv truncates it to B regardless of position. Double quotes are the only form consistent across tools.