5 posts tagged with "Milvus"

During a disk-cleanup pass, docker ps -as showed the etcd container's writable layer at 385MB — while its mounted data volume held just 8K. We didn't dare touch it during cleanup: one recreate, and every piece of Milvus metadata would evaporate with the writable layer.

Encountered this while building AI Customer Service — a 24/7 AI support agent answering product questions; its knowledge-base retrieval runs on Milvus, and all of Milvus's metadata lives in this etcd.

TL;DR​

compose's volumes: only guarantees "the volume is mounted at this path" — where the app writes is decided by its own parameters. etcd without a data-dir defaults to default.etcd under its working directory: it never used the /etcd mount, and all 385MB sat in the writable layer, one docker compose up -d rebuild away from oblivion. The fix follows etcd's native migration path: online snapshot → restore → swap data during a brief stop → add ETCD_DATA_DIR → recreate with the volume. Zero data loss, under two minutes of downtime.

Symptoms​

The etcd service in compose looks "correct" — the volume is declared:

  etcd:
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
      # ... nothing about data-dir
    volumes:
      - etcd_data:/etcd

But two numbers disagree:

$ docker ps -as --format "{{.Names}}\t{{.Size}}" | grep etcd
rag-service-etcd-1   385MB (virtual 199MB)     ← writable layer: 385MB

$ docker exec rag-service-etcd-1 du -sh /etcd
8K    /etcd                                     ← the mounted volume: empty

$ docker exec rag-service-etcd-1 ls -la / | grep etcd
drwx------  3 root root 4096 default.etcd         ← the data is here: container root

Root Cause​

Mounted ≠ used. volumes: etcd_data:/etcd only mounts the volume at the path /etcd; where etcd writes is decided by its --data-dir parameter. etcd's default data-dir is default.etcd under the working directory — this compose set neither the ETCD_DATA_DIR env var nor a --data-dir flag, so etcd created default.etcd at the container root and wrote everything into the writable layer. The /etcd mount point had been empty since day one.

The nastiest property of a "decorative volume" is that it's completely symptom-free: the service runs, reads and writes work, dashboards stay green. It only bites at the moment you run docker compose up -d --force-recreate (config change, writable-layer recycling, host migration) — the layer is discarded wholesale and the data vanishes, precisely when you're doing urgent ops work and can least afford a second incident.

Solution​

Use etcd's native snapshot migration: the online snapshot guarantees consistency, and downtime only happens at the final data swap.

Step 1: Consistent online snapshot​

docker exec rag-service-etcd-1 sh -c \
  'ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 snapshot save /tmp/etcd-snap.db'
docker cp rag-service-etcd-1:/tmp/etcd-snap.db /root/etcd-snap.db

snapshot save is safe against a running etcd (it goes through the Raft backend, not file copying) — no stop, no write lock.

Step 2: Restore into the target data-dir structure​

docker run --rm -v /root:/host quay.io/coreos/etcd:v3.5.5 \
  etcdctl snapshot restore /host/etcd-snap.db --data-dir /host/etcd-restored

Restore produces a full member/ data directory (a raw snapshot file cannot be used as a data-dir directly).

Step 3: Brief stop, move data into the volume​

docker stop rag-service-etcd-1
rm -rf /var/lib/docker/volumes/etcd_data/_data/*      # volume is empty; clear mount residue
cp -a /root/etcd-restored/. /var/lib/docker/volumes/etcd_data/_data/

Step 4: Add the missing config, recreate with the volume​

  etcd:
    environment:
      - ETCD_DATA_DIR=/etcd        # ← the missing line
      # ...
    volumes:
      - etcd_data:/etcd

docker compose up -d etcd    # config change triggers recreate; new container uses the volume

Step 5: Verify​

docker exec rag-service-etcd-1 etcdctl --endpoints=http://127.0.0.1:2379 endpoint health
du -sh /var/lib/docker/volumes/etcd_data/_data    # data should be in the volume
docker ps -as | grep etcd                          # writable layer should drop to KB scale

After this fix: writable layer 385MB → 8KB, 123MB of data living on the volume, Milvus reconnected automatically with metadata reads and writes healthy.

FAQ​

Why is my Docker volume mount empty?​

Two usual causes: an empty volume shadows whatever the image had at that path (documented Docker behavior); or the application's data-directory setting never pointed at the mount, so data went to the container's writable layer — the etcd case in this post. du on the volume vs the writable layer tells them apart instantly.

How do I safely migrate etcd to a new data-dir?​

etcdctl snapshot save for a consistent online snapshot (no downtime), etcdctl snapshot restore --data-dir to build a directory with proper member structure, stop the container, place the data, start with the new data-dir, then verify with endpoint health plus upstream reconnection. Downtime is only the swap itself.

I declared a volume in compose — why isn't it used?​

volumes: mounts the volume at a container path; the application decides where to write from its own config — etcd's data-dir, postgres's PGDATA, redis's dir. When the mount point and the app config disagree, the volume is an empty directory and all data sits in the writable layer, lost on recreate.

While prefixing vector collections per tenant with {tenant_id}_{collection}, the very first request bounced straight back from Milvus — invalid collection name: the first character ... must be an underscore or letter — and the endpoint returned 500.

Encountered this while building AI Customer Service — 24/7 AI support that answers product usage questions with instant guidance and best practices.

TL;DR​

Milvus strictly validates collection names: the first character must be a letter or underscore, only [a-zA-Z0-9_] are allowed (no hyphens), and length ≤ 255 — otherwise it raises invalid collection name (error code 1100). A UUID typically starts with a digit and always contains hyphens -, tripping both rules, so you cannot concat a tenant_id UUID into a collection name for isolation. Use the original name plus a tenant field filter instead.

The symptom​

A query endpoint with collection=system_product_help returned 500, with a single line in the rag-service log:

pymilvus.exceptions.MilvusException: code=1100,
Invalid collection name: 00000000-0000-0000-0000-000000000001_system_product_help.
the first character of a collection name must be an underscore or letter

The strange part: another endpoint with the same parameter (/query-logs) returned 200 — because it only reads PostgreSQL and never touches Milvus. Only paths that actually call Milvus has_collection trigger the validation.

Root cause​

The code built the collection name with f"{tenant_id}_{collection}", yielding e.g. 00000000-0000-0000-0000-000000000001_system_product_help. This name breaks two rules at once:

00000000-0000-0000-0000-000000000001_system_product_help
^              ^ ^
│              │ └─ underscore is fine here, but...
│              └─── hyphen `-` is illegal
└────────────────── first char is digit `0` (must be letter/underscore)

Milvus's collection name rules (source nameutil.go, regex ^[a-zA-Z_][a-zA-Z0-9_]*$, length ≤ 255):

A UUID almost always violates this: the standard 8-4-4-4-12 form carries 4 hyphens, and the first segment usually starts with a digit. Prefixing a collection name with such a token gets every has_collection / describe_collection / create call rejected server-side with code 1100.

Worse: because the concatenated name was never valid, the supposed "per-tenant prefix isolation" never actually worked — the collections that really exist in the database all use the un-prefixed original names. The concat logic was systematically disconnected from the real data; an assumption baked into code that no one ever verified.

The fix​

Don't put tenant_id in the collection name. Always use the original name; let a regular field handle tenant isolation:

from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

# ❌ Wrong: UUID prefix — starts with a digit + contains hyphens → code 1100
tenant_id = "00000000-0000-0000-0000-000000000001"
bad_name = f"{tenant_id}_system_product_help"   # illegal

# ✅ Right: collection keeps its original name; tenant_id is a schema field
client.create_collection(
    collection_name="system_product_help",        # legal, stable
    schema=client.create_schema(auto_id=True, enable_dynamic_field=False),
)
# Filter by tenant_id at write and query time, instead of renaming the collection
client.insert(
    collection_name="system_product_help",
    data=[{"tenant_id": tenant_id, "text": "...", "vector": [...]}],
)

If you genuinely need "a readable prefix" for multi-tenant or environment isolation, convert any arbitrary string into a safe slug before concatenating:

import re

def safe_slug(raw: str) -> str:
    # Replace anything outside [a-zA-Z0-9_] with underscore; prefix if first char is illegal
    s = re.sub(r"[^a-zA-Z0-9_]", "_", raw)
    if not re.match(r"^[a-zA-Z_]", s):
        s = "_" + s
    return s[:255]   # keep within the length cap

name = f"{safe_slug(tenant_id)}_system_product_help"   # legal

When debugging a 500 like this, first scan the service logs (PM2 or equivalent) for MilvusException — the error code and the "first character must be ..." hint pinpoint an illegal name almost immediately, so you don't need to dig into business logic.

As an aside, services that depend on Milvus have their own gotcha: containers without a restart policy take the whole RAG pipeline down after a crash — see Docker Compose service won't come back? Check the restart policy. On the query side, watch out for RRF scores being incompatible with the similarity threshold in hybrid search.

Caveats​

FAQ​

What are the Milvus collection name naming rules?​

The first character must be a letter or underscore; the remaining characters allow only letters, digits, and underscores ([a-zA-Z0-9_]). Hyphens and spaces are forbidden, and the maximum length is 255 characters. Milvus enforces this server-side with a regex; violations raise invalid collection name (error code 1100), failing both creation and lookup. snake_case is the safe choice.

What is the maximum length of a Milvus collection name?​

255 characters. Anything longer is rejected with invalid collection name (error code 1100). Real names rarely approach this limit — what usually pushes you over is concatenating long UUIDs or multi-segment paths into the name, which is itself a sign you shouldn't be putting that dynamic string in the collection name at all.

Why can't a UUID be used as a Milvus collection name prefix?​

The standard UUID form usually starts with a digit (violating "first char must be a letter/underscore") and always contains four hyphens - (not in the allowed charset) — both break the rules. Using tenant_id as a collection-name prefix for isolation is a common misuse: not only is the name illegal, it also makes the collection count balloon with tenants. The right approach is to put tenant_id in a regular field or partition key and keep the collection name stable.

While maintaining a 2-core 7GB cloud server for a client, I discovered disk usage at 93% (only 3GB free) and Airflow scheduler consistently consuming 160% CPU. Here's the complete diagnosis and fix.

Debugging hybrid retrieval scoring in a RAG knowledge-base project — the full troubleshooting trail below.

TL;DR​

Milvus hybrid retrieval with weighted fusion scores as 0.7 * dense_score + 0.3 * sparse_score tops out around 0.7 in theory — and lower in practice. Filtering with min_similarity=0.7 removes nearly everything. Fix: drop the threshold to 0.3, or adapt it to the fusion strategy dynamically.

Debugging hybrid retrieval scoring in a RAG knowledge-base project — the full troubleshooting trail below.

TL;DR​

Milvus hybrid search (Dense + Sparse) has four common pitfalls: empty sparse vectors erroring out, unloaded collections, malformed sparse formats, and thresholds set too high. Minimal fix code for each below.