Skip to main content

2 posts tagged with "Data Pipeline"

View all tags

Every LLM Batch Validation Falls Back? The All-or-Nothing Trap in Capacity Gates

Β· 5 min read
CCLEE
CCLEE

The first production run of an LLM batch validation job finished green β€” status success, no errors. But the output told another story: all 2,449 items to validate were tagged with the fallback marker, and the actual LLM call count was zero. A feature shipped as "semantic validation on by default" had never once run.

Encountered this while building AI Analytics β€” an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; this job runs in the title-optimization stage of its data pipeline.

TL;DR​

The capacity gate is all-or-nothing: if the workload exceeds the cap, the entire batch degrades to fallback with zero LLM calls β€” and the job still reports success. The trial cap was 400 while production actually had 2,449 keywordΓ—product pairs. Two lessons: calibrate capacity limits against measured production scale, and degrade at unit granularity (per group / queue / truncation) with abnormal fallback rates made observable.

Symptoms​

Stage 2 of the pipeline is LLM semantic validation, guarded by a capacity gate at the entrance:

def semantic_validate(pairs, cap=400):
    if len(pairs) > cap:
        # over the cap: degrade the whole batch, not a single LLM call
        return [mark_overflow(p) for p in pairs]
    return [llm_validate(p) for p in pairs]

First production run:

keywordΓ—product pairs: 2449/2449 all marked validation_mode='overflow'
actual LLM calls: 0
job status: success (no errors at all)

Judging by "did it finish", everything looks fine; only the distribution of the output column reveals the feature was disabled wholesale.

Root Cause​

Two problems stack up. The number: the trial cap was 400, but production scale β€” 60 market keywords Γ— same-category products plus 50 own-store keywords Γ— products across 92 products β€” pushed the pair count to 2,449. Off by an order of magnitude. The structure: the gate is all-or-nothing β€” over the cap means the whole batch degrades. What was meant as a capacity constraint effectively became "over the limit = feature off", and the degradation landed silently in a data column: no exception, no log line.

This is the same family as DeepSeek thinking consuming the output budget and silently returning empty: the fallback logic digests the failure, and the surface always says success.

Solution​

Step 1: Calibrate the cap against measured production scale​

Count the real workload before launch; don't estimate:

# dry-run: measure the scale, zero LLM calls
python -c "from pipeline import build_pairs; print(len(build_pairs(shop='prod')))"

Measured 2,449 β†’ set the cap to 3,000 (roughly 1.2-2x headroom), and confirm an oversized shop still has a degradation path instead of hitting a wall.

Step 2: Switch the granularity from "total" to "grouped"​

Call per product group so call counts grow linearly with product count instead of exploding with the keywordΓ—product product:

def semantic_validate(pairs, cap):
    groups = group_by_product(pairs)          # 92 products β†’ ~92 calls/round
    results = []
    for g in groups:
        if within_budget(g, cap):             # per-group check, no wholesale give-up
            results.extend(llm_validate(g))
        else:
            log.warning("capacity gate: group degraded",
                        extra={"size": len(g), "cap": cap})
            results.extend([mark_overflow(p) for p in g])
    return results

After calibration the third run showed 2,449/2,449 going through LLM validation; larger shops now degrade per group instead of losing everything.

Step 3: Make degradation observable​

Instrument the fallback path and alert on abnormal ratios (e.g. fallback rate > 50%). Degradation is a safety net, not a cover β€” it should be seen, not silently absorb the over-limit condition for you.

Notes

  • Capacity-style parameters (caps, concurrency, batch size) must be calibrated against measured production scale before launch; small test-environment samples never reproduce production magnitudes.
  • All-or-nothing gates only fit "hard cost ceiling" scenarios and must come with explicit alerting; otherwise they are a silent kill switch for the feature.
  • Land degradation in a dedicated queryable column/metric (here: a validation_mode column), and during acceptance check the distribution before the correctness.
  • Another silent-failure family in LLM outputs comes from structural validation β€” see Zod validates LLM output but fails silently? Don't use .strict().

FAQ​

How should a fallback mechanism in an LLM pipeline be designed?​

Keep the granularity small β€” degrade per item or per group instead of abandoning the batch; leave traces (marker columns, logs, metrics) and configure alerts. An all-or-nothing gate, once triggered, equals switching the feature off; it only fits hard-cost-ceiling scenarios with explicit alarms.

How do I set a capacity limit for LLM batch jobs?​

Don't guess. Run a dry-run on production-scale or proportionally sampled data to measure the actual workload, set the limit at 1.5-2x the measured value, and revisit it as the business grows. An estimate off by an order of magnitude is the standard setup for this incident.

How do I detect that an LLM job was silently degraded?​

The job status usually still says success. Inspect the output: count the share of fallback markers and check whether actual LLM calls match expectations. Alert on abnormal fallback rates (especially 100%) to turn silent failures into explicit signals.

CCLEE

Independent developer, 24 years in e-commerce, focused on grounding AI in real business scenarios.

Work with me

Data Pipeline Snapshot Slots Misaligned? Empty Segments Collapse Positions

Β· 5 min read
CCLEE
CCLEE

Auditing a production task's decision snapshot, we found stage 5's feature data sitting in array slot 3 instead of the documented slot 4. Downstream consumers and the detail drawer read by "segment number minus one" β€” and were silently getting the previous stage's output.

Encountered this while building AI Analytics β€” an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the snapshot is what the pipeline leaves behind for frontend rendering and post-hoc audit.

TL;DR​

The pipeline appends each stage's output into the snapshot array in execution order, and an if rows.empty: skip guard drops 0-row segments entirely, shifting every later segment forward. The static mapping "segment number βˆ’ 1 = slot" breaks at will, and which segments are empty depends on runtime state β€” slots differ per run. Two fixes: resolve slots at read time by in-row keys (recommended), or keep placeholders for empty segments so slots stay constant.

Symptoms​

The assembly logic looks like this:

snapshot = {"features": [], "rule_output": []}
for seg in segments:                    # stages 1..5 run in order
    df = execute_sql(seg.sql)
    if not df.empty:                    # 0-row segments dropped here
        snapshot["features"].append(df.to_dict("records"))

The design assumption was "stage 5 β†’ slot 4". Production snapshot audit found stage 5's features in slot 3:

all segments populated:   β‘ β†’0  β‘‘β†’1  β‘’β†’2  β‘£β†’3  β‘€β†’4   βœ“ matches assumption
stage 4 empty, dropped:   β‘ β†’0  β‘‘β†’1  β‘’β†’2       β‘€β†’3   βœ— shifted
stages 3+4 empty:         β‘ β†’0  β‘‘β†’1               β‘€β†’2   βœ— shifted again

The same code version produces completely different slot layouts depending on shop and permissions β€” an empty whitelist table drops stage 4; a disabled feature flag drops stage 3. Slots drift with runtime state.

Root Cause​

Positional addressing collided with sparse assembly. The snapshot array is a runtime product of concatenating "segments that produced output" β€” a sparsely-filled collection compressed. Consumers hard-coding "segment βˆ’ 1" implicitly assume every segment always yields at least one row. That assumption shatters in three routine situations: empty whitelist tables, disabled feature flags, naturally empty business data. 0-row segments are the norm, not the exception.

Deeper down, the emptiness guard itself (if not df.empty) is not wrong β€” wrong is the contract's implicit premise. The design doc says "slot = segment βˆ’ 1" but nobody declared it as an explicit contract. Every position-addressing consumer inherits an unacknowledged, unmaintained assumption.

Solution​

Let every row carry a segment identifier key; consumers resolve positions at read time with no static mapping:

def locate_segment(features: list, seg_key: str) -> dict:
    for row in features:
        if seg_key in row:              # row carries its own identity
            return row
    raise KeyError(f"segment '{seg_key}' missing in snapshot")

Slot drift becomes irrelevant β€” you look for "the segment whose keys look like this", not "element N". The one requirement: all consumers go through this single resolution entry point (write it into the processor docstring and the consumer contract: no hard-coded slots, ever).

Option B: keep placeholders for empty segments on the write side​

If downstream can't change yet, keep "slot = segment number" constant at assembly time:

snapshot["features"].append(
    df.to_dict("records") if not df.empty else {"__empty__": True}
)

The cost: placeholder objects appear in the snapshot and every consumer must handle them. Fine as a transition; converge on Option A long-term.

Step 3: contract tests over multiple runtime states​

Build snapshot fixtures for "all populated / one empty / several empty" and assert consumers parse all three identically. Testing only the all-full scenario tests nothing.

Notes

  • Any positional mapping in a spec must explicitly declare the addressing scheme (by key / by ID) and note that hard-coded indices are forbidden; implicit assumptions get broken by some runtime state eventually.
  • Empty-skip guards are the most common source of compression β€” a sibling case of silent data loss is Airflow PostgresHook truncating multi-statement SQL to the first result: again "no error, quietly less data".
  • Before rolling out consumer changes, regression-test with all three runtime-state fixtures; validating only the all-populated state misses every misalignment path.

FAQ​

How do you handle schema drift in data engineering?​

Replace positional contracts with key-based ones: address snapshots, messages, and interfaces by field name or segment identifier, never array index. When upstream changes without notice (empty segments skipped, fields added or removed), key-addressed consumers at worst raise "not found" β€” they never silently read the wrong data.

What is the difference between schema drift and schema evolution?​

Evolution is explicitly managed versioning: add a field, cut a release, migrate consumers. Drift is passive: upstream changes and downstream quietly misaligns. The slot shifting in this post is textbook drift β€” nobody touched the contract; the data shape changed.

How do I detect this kind of slot misalignment in a pipeline?​

Two layers: contract tests covering multiple runtime states (all populated / one empty / several empty) asserting identical consumer parsing; and periodic production snapshot audits checking each slot's content against its declared segment. "Content and position disagree" is drift.

CCLEE

Independent developer, 24 years in e-commerce, focused on grounding AI in real business scenarios.

Work with me