2 posts tagged with "Ops"

During a disk-cleanup pass, docker ps -as showed the etcd container's writable layer at 385MB — while its mounted data volume held just 8K. We didn't dare touch it during cleanup: one recreate, and every piece of Milvus metadata would evaporate with the writable layer.

Encountered this while building AI Customer Service — a 24/7 AI support agent answering product questions; its knowledge-base retrieval runs on Milvus, and all of Milvus's metadata lives in this etcd.

TL;DR​

compose's volumes: only guarantees "the volume is mounted at this path" — where the app writes is decided by its own parameters. etcd without a data-dir defaults to default.etcd under its working directory: it never used the /etcd mount, and all 385MB sat in the writable layer, one docker compose up -d rebuild away from oblivion. The fix follows etcd's native migration path: online snapshot → restore → swap data during a brief stop → add ETCD_DATA_DIR → recreate with the volume. Zero data loss, under two minutes of downtime.

Symptoms​

The etcd service in compose looks "correct" — the volume is declared:

  etcd:
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
      # ... nothing about data-dir
    volumes:
      - etcd_data:/etcd

But two numbers disagree:

$ docker ps -as --format "{{.Names}}\t{{.Size}}" | grep etcd
rag-service-etcd-1   385MB (virtual 199MB)     ← writable layer: 385MB

$ docker exec rag-service-etcd-1 du -sh /etcd
8K    /etcd                                     ← the mounted volume: empty

$ docker exec rag-service-etcd-1 ls -la / | grep etcd
drwx------  3 root root 4096 default.etcd         ← the data is here: container root

Root Cause​

Mounted ≠ used. volumes: etcd_data:/etcd only mounts the volume at the path /etcd; where etcd writes is decided by its --data-dir parameter. etcd's default data-dir is default.etcd under the working directory — this compose set neither the ETCD_DATA_DIR env var nor a --data-dir flag, so etcd created default.etcd at the container root and wrote everything into the writable layer. The /etcd mount point had been empty since day one.

The nastiest property of a "decorative volume" is that it's completely symptom-free: the service runs, reads and writes work, dashboards stay green. It only bites at the moment you run docker compose up -d --force-recreate (config change, writable-layer recycling, host migration) — the layer is discarded wholesale and the data vanishes, precisely when you're doing urgent ops work and can least afford a second incident.

Solution​

Use etcd's native snapshot migration: the online snapshot guarantees consistency, and downtime only happens at the final data swap.

Step 1: Consistent online snapshot​

docker exec rag-service-etcd-1 sh -c \
  'ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 snapshot save /tmp/etcd-snap.db'
docker cp rag-service-etcd-1:/tmp/etcd-snap.db /root/etcd-snap.db

snapshot save is safe against a running etcd (it goes through the Raft backend, not file copying) — no stop, no write lock.

Step 2: Restore into the target data-dir structure​

docker run --rm -v /root:/host quay.io/coreos/etcd:v3.5.5 \
  etcdctl snapshot restore /host/etcd-snap.db --data-dir /host/etcd-restored

Restore produces a full member/ data directory (a raw snapshot file cannot be used as a data-dir directly).

Step 3: Brief stop, move data into the volume​

docker stop rag-service-etcd-1
rm -rf /var/lib/docker/volumes/etcd_data/_data/*      # volume is empty; clear mount residue
cp -a /root/etcd-restored/. /var/lib/docker/volumes/etcd_data/_data/

Step 4: Add the missing config, recreate with the volume​

  etcd:
    environment:
      - ETCD_DATA_DIR=/etcd        # ← the missing line
      # ...
    volumes:
      - etcd_data:/etcd

docker compose up -d etcd    # config change triggers recreate; new container uses the volume

Step 5: Verify​

docker exec rag-service-etcd-1 etcdctl --endpoints=http://127.0.0.1:2379 endpoint health
du -sh /var/lib/docker/volumes/etcd_data/_data    # data should be in the volume
docker ps -as | grep etcd                          # writable layer should drop to KB scale

After this fix: writable layer 385MB → 8KB, 123MB of data living on the volume, Milvus reconnected automatically with metadata reads and writes healthy.

FAQ​

Why is my Docker volume mount empty?​

Two usual causes: an empty volume shadows whatever the image had at that path (documented Docker behavior); or the application's data-directory setting never pointed at the mount, so data went to the container's writable layer — the etcd case in this post. du on the volume vs the writable layer tells them apart instantly.

How do I safely migrate etcd to a new data-dir?​

etcdctl snapshot save for a consistent online snapshot (no downtime), etcdctl snapshot restore --data-dir to build a directory with proper member structure, stop the container, place the data, start with the new data-dir, then verify with endpoint health plus upstream reconnection. Downtime is only the swap itself.

I declared a volume in compose — why isn't it used?​

volumes: mounts the volume at a container path; the application decides where to write from its own config — etcd's data-dir, postgres's PGDATA, redis's dir. When the mount point and the app config disagree, the volume is an empty directory and all data sits in the writable layer, lost on recreate.

The alert email: production server root partition at 81% (30G/40G, past the 80% red line). First instinct: check docker system df — and there it is, images at RECLAIMABLE 100% (7), apparently a quick win. Except all 7 images are running; following the hint with docker image prune -a would be a production incident.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the disk in question belongs to the Docker server running its Airflow data pipeline.

TL;DR​

Past a disk red line, don't trust docker system df RECLAIMABLE — it's an estimate of "space with no container reference", and active images get flagged 100% anyway. The right move: sudo du -xh -d1 / layer by layer. This incident's three invisible hogs were none of them business data: task logs accumulating in container writable layers (no log volume in the Dockerfile), package-manager caches (npm + pnpm, ~4G combined), and a backup script that never pruned (full clones every run). Fixes: delete caches, reclaim writable layers with force-recreate, add TTL pruning to the backup script — 81% back down to 64%.

Symptoms​

$ df -h /
Filesystem      Size  Used Use% Mounted on
/dev/vda1        40G   30G  81% /

$ docker system df
TYPE        TOTAL   ACTIVE   SIZE     RECLAIMABLE
Images      7       7        4.2GB    100% (7)    ← all running
Containers  5       5        810MB    0%

Acting on docker system df (clean images) is a dead end — RECLAIMABLE says 100% but ACTIVE says 7/7. Where the space actually went, docker system df never shows:

sudo du -xh -d1 / | sort -rh | head
# selected output:
# 2.7G    /root/.npm              ← npm cache
# 1.1G    /root/.local/share/pnpm ← pnpm store
# 857M    /root/backups-git       ← backup clones, never pruned
# (hidden in overlay2: airflow scheduler writable layer 468M + dag-processor 257M)

Root Cause​

Three kinds of consumption, all invisible from the "business data" perspective.

Writable layers eating logs. Airflow task logs are written inside the containers with no external volume — scheduler writable layer 468M, dag-processor 257M, ~780M accumulated in two weeks, monotonically growing. Image layers never change; the writable layer does. docker system df buries it in the Containers SIZE total (810M), where it has no presence.

Package-manager caches only grow. On a server with frequent deploys/builds, ~/.npm (2.7G) and the pnpm store (1.1G) pile up indefinitely; nobody ever cleans them.

A backup script that never prunes. The backup script clones the repo worktree in full and pushes to GitHub every run — old clone directories are never removed, so 857M is mostly historical duplicates. Sneakier: orphan clones whose branch never pushed successfully can't just be deleted; verify first.

And docker system df RECLAIMABLE is a statistical estimate of "space not referenced by a running container" — active images can display 100% reclaimable (all 7 of ours did). It's a false-positive generator, not a cleanup guide.

Solution​

Step 1: Locate with du, layer by layer — look before deleting​

sudo du -xh -d1 / | sort -rh | head        # root partition, level by level
sudo du -xh -d1 /var/lib/docker | sort -rh | head   # drill into Docker's dir
docker ps -as --format "table {{.Names}}\t{{.Size}}"  # per-container writable layer

du -x stays on one filesystem, avoiding /proc, /sys noise and overlay confusion; docker ps -as SIZE exposes each container's writable layer — the key command for the "logs written inside the container" family.

Step 2: Delete caches outright​

npm cache clean --force        # or simply rm -rf ~/.npm/_cacache
pnpm store prune               # removes only unreferenced packages

~5.1G reclaimed, zero risk — caches re-download on demand.

Step 3: Reclaim writable layers with force-recreate​

docker compose up -d --force-recreate   # writable layer goes away with the old container

~780M reclaimed here. Two preconditions: pick a low-traffic window (services restart), and rescue anything valuable first — e.g. docker cp the task logs out, or they vanish with the container.

Step 4: TTL for backups, prevent recurrence​

# keep backup dirs for 7 days, prune older ones
find /root/backups-git -maxdepth 1 -type d -mtime +7 -exec rm -rf {} +

Add the pruning to the tail of the backup script (deployed in our github-backup-push.sh); before deleting, check for orphan clones whose branch never pushed (git ls-remote to compare) and confirm no unique commits — our 325M of orphans were removed only after verification.

After cleanup: 81% → 64%, with caches + pruning + writable layers together reclaiming ~6.7G.

FAQ​

docker system df shows 100% RECLAIMABLE — can I just delete?​

No. RECLAIMABLE estimates "space with no container reference", and running active images can still be flagged 100% reclaimable — all 7 of ours were. It answers "how much is theoretically unreferenced", not "what can be deleted". Measure first with du -xh -d1 and docker ps -as.

What's the difference between docker ps SIZE and docker system df SIZE?​

docker ps -as SIZE is the per-container writable layer; docker system df is the category-level aggregation over images/containers/volumes/cache. Logs accumulating inside a container only ever appear in the writable layer (visible via docker ps -as), never in any image size — which is exactly why "images look small but disk keeps growing".

How do I clean up a Docker server running out of disk?​

Three categories: package-manager caches deleted outright (npm cache clean --force, pnpm store prune), zero risk; container writable layers reclaimed via docker compose up -d --force-recreate (service restart; rescue logs first); backups, logs, and clone directories put on TTL pruning to prevent recurrence. Before any deletion, confirm the hogs with du -xh -d1 /.