2 posts tagged with "Zod"

While sending a batch backfill request to our bookkeeping assistant in production, the model returned a tool call with valid JSON — and nothing happened. The user just saw a generic "didn't catch that" line.

I hit this while building Life, an AI bookkeeping assistant — you talk naturally, and the AI extracts amounts, categories, and accounts. The debugging session turned up two structural problems that both fail silently, so I'm writing them up as two scenarios.

TL;DR​

• Scenario 1: the enum in our tool definition and the enum in our server-side validator were two hand-written copies that had drifted. The model emitted a value that was legal per the definition and the validator rejected it; because the count check lived at the array level, one invalid candidate voided the whole card. Fix: single-source the enum, then validate per item.
• Scenario 2: to reduce dead-end candidates, we taught the model "every candidate must carry complete arguments." The teaching made outputs longer, and the small model's bad-JSON rate rose from ~6% to 12-28%. Fix: revert the teaching, relax the validation, and let a downstream deterministic guard catch missing arguments.
• Two rules: paired enums must reference one shared constant; teach routing, not argument extraction.

Two sides: the model sees the definition, your code runs the validator​

In function calling, the tool definition and the parameter validation are two separate pieces of code, and the model only ever sees the first one — every silent failure in this post lives in the mismatch between them.

The model never executes anything. Each call, your server sends tool definitions (JSON Schema format) to the model; the model returns a tool name plus an arguments JSON; your code validates it before executing. This means any "set of legal values" exists in at least two places: in the definition sent to the model, and in your validation code.

"Silent failure" here means: no API error, a tool call comes back, the arguments look like valid JSON — but the feature doesn't run. In other words, the tool call never takes effect, or the feature fails intermittently. The user sees a generic fallback message; the details sit in server logs. Without log instrumentation, this kind of failure can live in production for a long time.

Scenario 1: the model emitted a schema-legal enum value — why did the whole card get rejected?​

When the enum in the tool definition and the server-side validation enum aren't the same constant, a "legal" value emitted from the definition side is guaranteed to be rejected by the validation side — and an array-level count check amplifies one invalid item into a fully voided card.

The symptom: a failure chain visible only in logs​

Our routing model has a family of disambiguation tools: when the user's input is ambiguous, the model doesn't execute directly — it calls a meta-tool that outputs 2-4 candidate intents, which we render as tappable cards. Each candidate carries a tool field whose value must be a real action tool name.

One day, production logs showed a routing failure: the user said "please backfill the missing dates in batch", the model did call the disambiguation tool, and arguments was valid JSON — but the whole candidate card was judged invalid, and the user got the fallback line. The confusing part was the contrast: by round six of rephrasing, the same request routed fine. That ruled out "the model is hallucinating" and pointed at a deterministic failure triggered by one specific value.

Root cause: two hand-written enums, plus a count check at the array level​

Digging in found two compounding causes.

Cause one: enum drift. In the tool definition, the candidates' tool enum used the full set of tool names. The server-side validation enum, however, was the full set minus one batch-split tool reserved for an image-only pipeline — that tool must never be nominated by the text path, because nominating it would bypass the image pipeline's dedicated contract. The two sets were written in two places, maintained separately:

// Definition side: JSON Schema sent to the model (before the fix)
tool: { type: 'string', enum: [...ALL_TOOLS] }  // full set

// Validation side: server-side zod (before the fix, a separate hand-written copy)
const ALLOWED = ALL_TOOLS.filter((t) => t !== 'batch_split')  // subset
tool: z.enum(ALLOWED)

The spec the model saw said batch_split was legal, so it dutifully used it; the validator disagreed. The model followed your schema and got rejected by your own validation — this isn't a model error, it's the two sides telling different stories.

Cause two: array-level validation voids the whole card. The candidate count constraint sat on the array:

options: z.array(OptionSchema).min(2).max(4)

Zod validates every element of the array, and any single failure fails the entire array. So one invalid candidate (that batch_split) dragged down the whole card — three perfectly valid candidates buried with it. The upper layer only had a fallback line; the truth stayed in the logs.

The fix: single-source the enum, then tolerate per item​

Step one, make both sides reference the same constant (a one-line fix):

const DISAMBIGUABLE_TOOLS = ALL_TOOLS.filter((t) => t !== 'batch_split')

// Definition side
tool: { type: 'string', enum: [...DISAMBIGUABLE_TOOLS] }
// Validation side
tool: z.enum(DISAMBIGUABLE_TOOLS)

Step two, turn "one bad element voids the array" into "a bad element only drops itself": run a per-item check inside a preprocess, drop invalid items with a log line, and apply the count floor after filtering:

import { z } from 'zod'

const OptionSchema = z.object({
  tool: z.enum(DISAMBIGUABLE_TOOLS),
  label: z.string().min(1).max(30),
  args: z.record(z.unknown()).optional(),
})

// Per-item tolerance: invalid candidates are dropped, not propagated; every drop is logged
function filterOptions(raw: unknown): unknown {
  if (!Array.isArray(raw)) return raw
  return raw.filter((item) => OptionSchema.safeParse(item).success)
}

const ArgsSchema = z.object({
  // min becomes a "post-filter" floor: only an emptied array rejects the whole card
  options: z.preprocess(filterOptions, z.array(OptionSchema).min(1).max(4)),
})

One boundary to be explicit about: per-item tolerance silently discards invalid candidates, so every drop must emit a warn log — otherwise bad candidates vanish without a trace and your observability goes blind. Also note min now means "post-filter floor": the card is only rejected when every candidate was dropped, and at that point there's genuinely nothing to render, so falling back is correct.

This "one extra field/value sinks the whole output" problem is a sibling of Zod .strict() silently failing on LLM output: strict validation is right for fully-controlled client input; for model output, tolerance has to go down to the item level.

Scenario 2: after adding "arguments required" teaching, bad-JSON rate went from 6% to 12-28%​

For payload-length-sensitive meta-tools, teaching that forces complete arguments makes outputs longer and directly inflates the small model's bad-JSON rate — an A/B test on the same cases measured ~6% rising to 12-28%.

A reasonable-looking teaching​

After scenario one was fixed, we found a follow-up annoyance: some candidates reached the card, and only after tapping did the user discover required arguments were missing, costing another round trip. So we naturally added a teaching — "every candidate's args must carry all required arguments for that tool; never nominate a candidate you can't pre-fill completely" — injected in three places at once: a rule in the system prompt, the argument description in the tool definition, and the note on a few-shot example.

// Few-shot example (the teaching we later reverted)
{
  input: 'delete it',
  tool: 'suggest_options',
  note: 'ambiguous reference → disambiguate; every candidate\'s args must carry all required
         arguments for that tool (e.g. delete_record needs domain); never nominate a candidate
         you cannot pre-fill completely',
}

The symptom: the eval gate went red three runs in a row​

We run an eval gate: a fixed set of routing cases, re-run in batch after every change; failure rate over threshold means red. After the teaching shipped, the gate failed three consecutive runs, and the failure shape was highly consistent — every failure was a JSON format error on the disambiguation tool call (truncation, extra closing brackets), not a validation rejection; frequency climbed from 2 to 5 per run.

Attribution: an old-build A/B pinned the variable down​

The first suspicion was "the teaching text itself misleads the model." But to actually conclude that, we ran a controlled experiment: old build vs new build, the same 4 cases, 16 runs each, counting the malformed-JSON rate.

That single 6% case deserves a note: the old build also flaked occasionally on the full gate — that's the small model's baseline noise, not something the change introduced. The real signal is the jump from 6% to 12-28%: the teaching made every candidate carry full arguments, the disambiguation tool's outputs got significantly longer, and the small model's JSON format error rate amplifies with longer outputs. You cannot see this relationship from a single red/green run.

The fix: revert the teaching, relax validation, guard downstream​

The ruling had three parts:

1. Strip the args-required teaching from all three places; the few-shot examples teach routing direction only (which inputs go to disambiguation, which candidates to converge on), not argument extraction.
2. Relax validation to "only validate candidates that carry args": if a candidate includes args, pre-validate them with that tool's own schema — invalid values still get dropped; missing args are tolerated on the card.
3. Let a downstream deterministic exit catch missing arguments: when the user taps a candidate with missing args, the server returns 422 with guiding text, the original card stays, and the user supplies the missing piece in one message.

// Same-ruler validation only for candidates that carry args: invalid values still dropped, missing tolerated
if (option.args !== undefined && !parseToolCall(option.tool, JSON.stringify(option.args))) {
  dropped.push(option.tool)  // drop + warn log
  continue
}

Why is it safe to let missing args onto the card? Because the downstream guard is deterministic — it doesn't rely on the model's good behavior, it's a fixed branch in server code. Once code catches the problem the teaching tried to prevent, the teaching is all cost (longer outputs, higher bad-JSON rate). This is the same family as thinking-mode consuming the output budget and returning empty content: small models' structured output is sensitive to payload length. For another silent failure of that kind, see DeepSeek/Qwen structured calls returning empty.

Two rules to keep​

LLM tool-calling reliability comes down to two rules: paired enums must reference a single shared constant, and prompt teaching should cover routing direction only — never argument extraction.

Rule one: paired enums must reference a single shared constant. Anywhere a set of legal values appears both in "the definition sent to the model" and in "server-side validation", writing the two copies by hand is forbidden — expand both from one constant. The same holds for any paired constraint (field lists, count limits), not just enums.

Rule two: teach routing, not extraction; decide teaching with old-build A/Bs. Every line of teaching adds output length, and small models' format-error rate climbs with length. Routing direction (which input goes to which tool) is cheap guidance worth teaching; argument completeness, catchable by deterministic validation, belongs in code — not in the prompt. And when judging whether a teaching should stay, don't conclude from a single red/green run — run old and new builds against the same cases a dozen-plus times each and look for the jump in bad-JSON rate.

FAQ​

How does LLM tool calling work?​

The model never runs anything: it returns a tool name plus a JSON payload that must match the tool's JSON Schema, and your server does the validation and execution. The definition you send and the validator you run are two separate pieces of code — both failure modes in this post (enum drift voiding a whole candidate card, bad-JSON rate jumping from ~6% to 12-28%) live in that mismatch.

Why is my function calling JSON rejected even though it looks valid?​

Because the enum in your tool definition and the enum in your server-side validator are separate hand-written copies. When they drift, the model emits a definition-legal value and your array-level check (a min(2) on the candidates array, in our case) rejects the entire call — one invalid element fails the whole array under zod-style validation. Fix it with a single shared constant plus per-item filtering.

How do I make LLM tool calling more reliable?​

Three rules from our production A/B tests: single-source every paired enum; validate arrays per item so one bad candidate doesn't void the whole card; keep prompt teaching at the routing level instead of forcing full argument payloads. Our measured bad-JSON rate jumped from ~6% to 12-28% when we forced complete arguments, and fell back to the baseline after reverting.

When you validate LLM tool_call / function call output with Zod, the model occasionally emits an extra field — say you only defined amount / category, and it also fills in note — and the whole validation fails, the action is silently dropped, and the user just gets "not recognized." In reality an entire tool_call was whole-rejected.

Encountered this while building Life — a natural-language bookkeeping and wellness assistant where you just talk to record entries and the AI extracts amount, category, and account. When a user said "delete that coffee from yesterday," the model slipped an extra note: "coffee" into the delete locator, trying to locate by remark.

TL;DR​

Zod's .strict() means "this object must not contain any unknown keys — one extra throws." That constraint fits validating clients you fully control, but LLM function call output is model-generated and inherently uncontrollable — it fills in fields it "thinks should be there," especially when multiple tools share similar schemas. One irrelevant field kills the whole tool_call, validation returns null, and the action is silently lost. Fix: drop .strict() and use Zod's default strip (silently removes unknown keys) for tolerance, paired with safeParse as a fallback.

Symptoms​

The locator schema for delete/update defines a few known fields, tightened with .strict():

import { z } from "zod";

// ❌ dangerous: with .strict()
const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
}).strict();   // ← any unknown key throws

// parse the LLM's tool_call arguments
function parseToolCall(raw: unknown) {
  const parsed = LocatorSchema.safeParse(raw);
  if (!parsed.success) {
    return null;   // ← the whole tool_call is dropped
  }
  return parsed.data;
}

The user says "delete that coffee from yesterday," and the model produces a reasonable but extra-fielded output:

{
  "date": "yesterday",
  "noteContains": "coffee",
  "note": "coffee"
}

The model filled both noteContains (in schema) and note (out of schema, which it thought should exist). .strict() rejects the unknown key note outright, parseToolCall returns null, and the delete action is silently dropped — the user gets "not recognized" when it was actually a whole-reject.

Root Cause​

.strict() changes Zod's policy on unknown keys, and LLM output naturally carries unknown keys.

Zod z.object() has three policies for unknown keys:

.strict() is designed for "contract strictness" — the server defines which fields exist, the client should supply only those, and anything extra is a breach. That logic holds for traditional APIs because the client is developer-written and can be held to the contract.

But LLM function calling flips the premise:

1. The output comes from model generation, not a developer-written client. The model guesses what to fill based on the schema's description and examples; schemas reused across domains (e.g. a locator shared by budget / mood / todo) confuse it further, so it fills in fields it "thinks should be there."
2. Wrong fields are the norm, not an exception. The model occasionally emitting an extra note or omitting an optional field is expected behavior in LLM apps and shouldn't be punished by failing the whole call.
3. The failure is silently swallowed. After safeParse fails and returns null, the upstream can only vaguely say "not recognized," while the real cause (an unknown key) sits unseen in parsed.error.

LLM output { date, noteContains, note }
                        │
                        ▼
         .strict() hits unknown key "note"
                        │
                        ▼
           safeParse → { success: false }
                        │
                        ▼
          parseToolCall returns null (action dropped)
                        │
                        ▼
       user gets "not recognized" (actually a whole-reject)

Solution​

1. Drop .strict(), use default strip for tolerance​

// ✅ recommended: no .strict(), Zod defaults to stripping unknown keys
const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
});
// the extra "note" is silently removed; known fields parse normally

After dropping .strict(), "delete that coffee from yesterday" parses cleanly into { date, noteContains }; the extra note is stripped and the delete action runs correctly.

2. If unknown keys are useful, keep them with .passthrough()​

When the extra field actually carries semantics you want to use (e.g. the model filled note to express "locate by remark"), don't drop it — keep it and decide how to consume it:

const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
}).passthrough();   // keep unknown keys; parsed.data.note is still readable

Even better, promote it to a known field — if the model keeps filling some unknown key, the schema is missing a capability slot, so add it (here noteContains was added after absorbing the "locate by remark" need).

3. Make failures observable — don't silently return null​

Regardless of policy, when safeParse fails, log the specific error instead of swallowing it into null:

function parseToolCall(raw: unknown) {
  const parsed = LocatorSchema.safeParse(raw);
  if (!parsed.success) {
    // log Zod's concrete error (which key, what problem) for triage
    logger.warn(
      { raw, issues: parsed.error.issues },
      "locator parse failed"
    );
    return null;
  }
  return parsed.data;
}

Now when something goes wrong, the log has the full issues (including the unknown-key path), instead of an unactionable "not recognized."

After the fix, "delete that coffee from yesterday" → delete_record { locator: { noteContains: "coffee" } } parses correctly with no silent drop.

Notes​

FAQ​

Why does Zod .strict() make LLM output validation fail?​

.strict() requires an object to have no unknown keys — one extra throws. LLM function call output is model-generated, guessing from the schema description, and routinely includes fields it thinks should be there (especially with cross-domain schemas). The moment an unknown key appears, .strict() fails the entire validation and drops the whole tool_call.

Should I use strict when validating LLM function calling output with Zod?​

Not recommended. .strict() suits validating clients you fully control (developer-written code can be held to a contract), but LLM output is uncontrollable and over/under-filling is the norm. Drop .strict() and use Zod's default strip (silently removes unknown keys) for better tolerance; if unknown keys carry semantics you want, use .passthrough() to keep them, or promote them to known fields.

Does Zod strip or throw on unknown fields by default?​

Default is strip — it silently removes unknown keys without error. .strict() makes it throw on unknown keys; .passthrough() keeps them as-is. For uncontrollable output like LLM generations, prefer default strip or passthrough over .strict(), which kills the whole payload over a single irrelevant field.