20 posts tagged with "Bug Fix"

Clicking the "Extract this page" button in the browser extension throws Cannot access 'Hc' before initialization — while typecheck, build, and the regular test suite all pass, and the error never reproduces in local development.

I hit this while building an e-commerce automated data collection tool for a client — bulk-scraping product images, SKUs, prices, and reviews, cleaned and exported as structured data for inventory management and competitor analysis. "Extract this page" is the collection entry of that toolchain, and the content script module was what broke.

TL;DR​

A third-party classic script (vendored) contains an implicit global assignment (CandidateElement = function(...), assigned without declaration). Legal in a classic script — it silently creates a global — the line becomes a ReferenceError once the file is inlined into an ESM module graph that runs in strict mode. Module initialization aborts on the spot, and the namespace the downstream code receives through dynamic import sits in the temporal dead zone (TDZ). Fix: add one line var CandidateElement; at the top of the vendor file.

The symptom: green build, instant crash on click​

The error fires when the user clicks, not at page load:

TypeError: Cannot access 'Hc' before initialization

Three counterintuitive facts:

1. typecheck passes — nothing wrong at the type level;
2. build passes — the bundler does static analysis only, it never executes module code;
3. regular tests pass — no test imports the module graph completely.

The failing identifier is Hc, a 2-letter minified name, not any business symbol. Hold that thought; it matters for identification later.

Root cause: how an implicit global breaks the ESM module graph​

The offending line comes from a third-party library, at reader-finder.js:878:

// classic script semantics: assign without declaring = create a global, legal
CandidateElement = function(e, t) { ... }

The failure chain has 4 steps, each enabling the next:

Step 1: legal under classic script semantics. The file originally loaded via a <script> tag; in sloppy mode, assigning without declaring silently creates a global variable — the original author relied on exactly that.

Step 2: a minefield once inlined into ESM. The file got inlined into the extension's ESM module graph, and ESM code always runs in strict mode — the implicit global assignment now throws a ReferenceError, and module evaluation of that vendor module aborts immediately.

Step 3: the break spreads along the module graph. The content script content.js evaluates its inline module graph and stalls at the vendor module: earlier modules got their message listeners registered, but later module facades never executed — the graph is left half-initialized.

Step 4: the dynamic import lands in the TDZ. When the user clicks the button, code loads the namespace facade through dynamic import. Because of the step-3 break, that namespace is in the temporal dead zone, and touching it throws Cannot access 'Hc' before initialization — Hc being the renamed internal binding of the module that never finished initializing.

This explains every observation: static checks stay green because they never execute module code; the error appears on click because the dynamic import lives inside the click handler; and the identifier is minified because the broken binding was renamed by the bundler.

For the other high-frequency dynamic import pitfall (module not found), see: Node.js ESM dynamic import says module not found? Check the file extension.

The fix: one var declaration at the vendor file head​

No third-party logic changes — just turn the implicit global into an explicit declaration by adding at the top of the vendor file:

var ReaderArticleFinder;
var CandidateElement;

The assignment changes from "create a global" to "assign to a declared variable", which is legal in strict mode. Module initialization completes, and the facade downstream code imports dynamically works as expected.

ReaderArticleFinder in the same file was already handled this way — the same library planted the same trap twice; the first got fixed, the second (CandidateElement) slipped through.

Verification: reproduce it locally with Vitest​

Before fixing, make it reproduce on demand — otherwise every check means a production deploy. Regular tests never reach this path, but importing the module directly in Vitest (jsdom environment) does:

import { describe, it, expect } from 'vitest';

describe('vendor reader-finder strict-mode', () => {
  it('initializes the full module graph without a ReferenceError', async () => {
    const mod = await import('./lib/vendor/reader-finder');
    expect(mod).toBeDefined();
  });
});

This test reproduced the error before the fix and produced a stack with real file line numbers (reader-finder.js:878) — far more actionable than a minified Hc from production. It turned green after the fix.

The complete verification runs the extraction chain end to end: the module graph initializes fully and extraction actually works — both must pass to call it closed.

Regression guard​

The reproduction case became a permanent smoke test (extractor.test.ts), plus a rule for the repo: any new classic script vendored into the project must pass this test, or an equivalent strict-mode check, before landing.

Quick identification for this kind of TDZ error​

Next time Cannot access 'xxx' before initialization shows up, two traits tell you whether it is the same species:

Left column on both rows: suspect an implicit global in a vendored classic script — search for assign-without-declare and reproduce with a direct Vitest import. The other classic ESM migration error (CJS require ESM) is covered here: Node.js require nanoid throws ERR_REQUIRE_ESM? Alternatives after v5 went ESM-only.

FAQ​

Why does 'Cannot access before initialization' only appear at runtime when typecheck and build are green?​

Static checks and bundlers never execute module code, while the ReferenceError from an implicit global assignment fires only when the module actually initializes. If the offending module sits on a dynamic import chain, the error defers to the moment of interaction — in this case, clicking the extension button, with 0 errors at build time. Importing the module directly in Vitest (jsdom) reproduces it locally with real line numbers.

What does 'Cannot access xxx before initialization' have to do with the temporal dead zone (TDZ)?​

The namespace object returned by a dynamic import stays in the temporal dead zone once its dependency module's initialization broke, so touching any export throws. The giveaway is the identifier name: a 2-letter minified name like Hc instead of a business symbol means the break happened during module graph evaluation, not in your code.

How do I fix implicit globals in a vendored classic script?​

Declare each implicit global at the top of the vendor file with var (2 in this case: ReaderArticleFinder and CandidateElement) so the assignment targets a declared binding. One line removes the whole module-graph initialization break; run a strict-mode smoke test before any new vendor file lands.

The first production run of an LLM batch validation job finished green — status success, no errors. But the output told another story: all 2,449 items to validate were tagged with the fallback marker, and the actual LLM call count was zero. A feature shipped as "semantic validation on by default" had never once run.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; this job runs in the title-optimization stage of its data pipeline.

TL;DR​

The capacity gate is all-or-nothing: if the workload exceeds the cap, the entire batch degrades to fallback with zero LLM calls — and the job still reports success. The trial cap was 400 while production actually had 2,449 keyword×product pairs. Two lessons: calibrate capacity limits against measured production scale, and degrade at unit granularity (per group / queue / truncation) with abnormal fallback rates made observable.

Symptoms​

Stage 2 of the pipeline is LLM semantic validation, guarded by a capacity gate at the entrance:

def semantic_validate(pairs, cap=400):
    if len(pairs) > cap:
        # over the cap: degrade the whole batch, not a single LLM call
        return [mark_overflow(p) for p in pairs]
    return [llm_validate(p) for p in pairs]

First production run:

keyword×product pairs: 2449/2449 all marked validation_mode='overflow'
actual LLM calls: 0
job status: success (no errors at all)

Judging by "did it finish", everything looks fine; only the distribution of the output column reveals the feature was disabled wholesale.

Root Cause​

Two problems stack up. The number: the trial cap was 400, but production scale — 60 market keywords × same-category products plus 50 own-store keywords × products across 92 products — pushed the pair count to 2,449. Off by an order of magnitude. The structure: the gate is all-or-nothing — over the cap means the whole batch degrades. What was meant as a capacity constraint effectively became "over the limit = feature off", and the degradation landed silently in a data column: no exception, no log line.

This is the same family as DeepSeek thinking consuming the output budget and silently returning empty: the fallback logic digests the failure, and the surface always says success.

Solution​

Step 1: Calibrate the cap against measured production scale​

Count the real workload before launch; don't estimate:

# dry-run: measure the scale, zero LLM calls
python -c "from pipeline import build_pairs; print(len(build_pairs(shop='prod')))"

Measured 2,449 → set the cap to 3,000 (roughly 1.2-2x headroom), and confirm an oversized shop still has a degradation path instead of hitting a wall.

Step 2: Switch the granularity from "total" to "grouped"​

Call per product group so call counts grow linearly with product count instead of exploding with the keyword×product product:

def semantic_validate(pairs, cap):
    groups = group_by_product(pairs)          # 92 products → ~92 calls/round
    results = []
    for g in groups:
        if within_budget(g, cap):             # per-group check, no wholesale give-up
            results.extend(llm_validate(g))
        else:
            log.warning("capacity gate: group degraded",
                        extra={"size": len(g), "cap": cap})
            results.extend([mark_overflow(p) for p in g])
    return results

After calibration the third run showed 2,449/2,449 going through LLM validation; larger shops now degrade per group instead of losing everything.

Step 3: Make degradation observable​

Instrument the fallback path and alert on abnormal ratios (e.g. fallback rate > 50%). Degradation is a safety net, not a cover — it should be seen, not silently absorb the over-limit condition for you.

FAQ​

How should a fallback mechanism in an LLM pipeline be designed?​

Keep the granularity small — degrade per item or per group instead of abandoning the batch; leave traces (marker columns, logs, metrics) and configure alerts. An all-or-nothing gate, once triggered, equals switching the feature off; it only fits hard-cost-ceiling scenarios with explicit alarms.

How do I set a capacity limit for LLM batch jobs?​

Don't guess. Run a dry-run on production-scale or proportionally sampled data to measure the actual workload, set the limit at 1.5-2x the measured value, and revisit it as the business grows. An estimate off by an order of magnitude is the standard setup for this incident.

How do I detect that an LLM job was silently degraded?​

The job status usually still says success. Inspect the output: count the share of fallback markers and check whether actual LLM calls match expectations. Alert on abnormal fallback rates (especially 100%) to turn silent failures into explicit signals.

While deploying a multi-package project (frontend and backend sharing one repository), npm audit reported 3 high vulnerabilities in the deploy log. We logged them against the backend — only to confirm later that all 3 highs lived in the frontend tree, and the backend had been a separate set of moderates all along.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the frontend lives at the repository root and the backend in server/, deployed separately from one repo.

TL;DR​

npm audit summary lines carry only counts, no path. When a deploy pipeline runs npm install in several directories in sequence, the outputs interleave and the summaries become unattributable. The fix: use each package tree's unique fingerprint — the N in audited N packages — to attribute findings first, then pin the vulnerable transitive dependencies with overrides in package.json. A second deploy brought both sides to zero.

Symptoms​

The deploy script runs npm install in the frontend (repo root) and then the backend (server/), and both outputs land in the same log stream:

# Deploy log (summary lines carry no path)
added 546 packages in 41s
found 3 high severity vulnerabilities

added 372 packages in 24s
found 4 moderate severity vulnerabilities

Our handover notes attributed the "3 high" to server/. Chasing the backend dependency chain led nowhere — the server/ audit, run locally or on the server, consistently showed 4 moderates and never a single high. "Visible in the log, unattributable in practice" is a chronic disease of mixed deploy pipelines; our earlier write-up on stale build artifacts after deployment is the same family of problem.

Root Cause​

The npm audit summary only prints "found X vulnerabilities" with no directory info, and the adjacent audited 546 packages rarely registers as an attribution clue. Two package trees with wildly different sizes (546 vs 372) turn out to be the only stable fingerprint.

The frontend root installs the full Vite + React + Ant Design Pro stack, so its tree is large; @ant-design/pro-components → @ant-design/pro-layout pulls in an old path-to-regexp, which is exactly where the 3 highs came from. The backend server/ is a lean Express + tsx tree whose only issue is the tsx → @esbuild-kit/core-utils → old esbuild chain of moderates.

Solution​

Step 1: Attribute findings by audited N​

Run npm install in each directory locally (or read added N packages straight from the deploy log) and record the tree sizes:

cd <repo-root> && npm install 2>&1 | tail -2   # added 546 packages ...
cd server      && npm install 2>&1 | tail -2   # added 372 packages ...

In the deploy log, found 3 high immediately follows added 546 → frontend; 4 moderate follows added 372 → backend. Get attribution right before touching any dependency.

Step 2: Expand the vulnerability chain​

npm audit                # see the Path field, full chain
npm ls path-to-regexp    # or reverse-lookup who depends on a package

The frontend output confirmed the chain: @ant-design/pro-components → @ant-design/pro-layout → path-to-regexp (old version, 3 high).

Step 3: Pin the transitive dependency with overrides​

Frontend root package.json:

{
  "overrides": {
    "path-to-regexp": "^8.4.2"
  }
}

Backend server/package.json (with tsx bumped to a newer release):

{
  "overrides": {
    "@esbuild-kit/core-utils": {
      "esbuild": "^0.25.12"
    }
  }
}

overrides supports nested syntax, scoping the pin to the child dependency under one specific parent — more surgical than overriding a package name globally.

Step 4: Reinstall and verify​

rm -rf node_modules package-lock.json && npm install && npm audit

After redeploying both sides, audit reports 0 vulnerabilities on each.

FAQ​

Why is npm audit fix not working?​

npm audit fix only upgrades versions inside the allowed semver range. When the vulnerability lives in a transitive dependency whose version is pinned by an upstream package's range, fix cannot touch it. Use overrides in package.json to force the version, then reinstall.

How do I find which dependency chain an npm audit finding comes from?​

Read the Path field in the full npm audit output — it lists the complete chain from a direct dependency down to the vulnerable package. You can also reverse-lookup with npm ls <package>. Summary lines carry counts only.

How do I map npm audit results to a specific subproject?​

Deploy log summaries carry no directory name, so match them by each directory's added N packages / audited N packages tree size. Run npm install once per directory locally, record each N, and the mapping stays stable.

Auditing a production task's decision snapshot, we found stage 5's feature data sitting in array slot 3 instead of the documented slot 4. Downstream consumers and the detail drawer read by "segment number minus one" — and were silently getting the previous stage's output.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the snapshot is what the pipeline leaves behind for frontend rendering and post-hoc audit.

TL;DR​

The pipeline appends each stage's output into the snapshot array in execution order, and an if rows.empty: skip guard drops 0-row segments entirely, shifting every later segment forward. The static mapping "segment number − 1 = slot" breaks at will, and which segments are empty depends on runtime state — slots differ per run. Two fixes: resolve slots at read time by in-row keys (recommended), or keep placeholders for empty segments so slots stay constant.

Symptoms​

The assembly logic looks like this:

snapshot = {"features": [], "rule_output": []}
for seg in segments:                    # stages 1..5 run in order
    df = execute_sql(seg.sql)
    if not df.empty:                    # 0-row segments dropped here
        snapshot["features"].append(df.to_dict("records"))

The design assumption was "stage 5 → slot 4". Production snapshot audit found stage 5's features in slot 3:

all segments populated:   ①→0  ②→1  ③→2  ④→3  ⑤→4   ✓ matches assumption
stage 4 empty, dropped:   ①→0  ②→1  ③→2       ⑤→3   ✗ shifted
stages 3+4 empty:         ①→0  ②→1               ⑤→2   ✗ shifted again

The same code version produces completely different slot layouts depending on shop and permissions — an empty whitelist table drops stage 4; a disabled feature flag drops stage 3. Slots drift with runtime state.

Root Cause​

Positional addressing collided with sparse assembly. The snapshot array is a runtime product of concatenating "segments that produced output" — a sparsely-filled collection compressed. Consumers hard-coding "segment − 1" implicitly assume every segment always yields at least one row. That assumption shatters in three routine situations: empty whitelist tables, disabled feature flags, naturally empty business data. 0-row segments are the norm, not the exception.

Deeper down, the emptiness guard itself (if not df.empty) is not wrong — wrong is the contract's implicit premise. The design doc says "slot = segment − 1" but nobody declared it as an explicit contract. Every position-addressing consumer inherits an unacknowledged, unmaintained assumption.

Solution​

Option A (recommended): resolve segments by in-row keys at read time​

Let every row carry a segment identifier key; consumers resolve positions at read time with no static mapping:

def locate_segment(features: list, seg_key: str) -> dict:
    for row in features:
        if seg_key in row:              # row carries its own identity
            return row
    raise KeyError(f"segment '{seg_key}' missing in snapshot")

Slot drift becomes irrelevant — you look for "the segment whose keys look like this", not "element N". The one requirement: all consumers go through this single resolution entry point (write it into the processor docstring and the consumer contract: no hard-coded slots, ever).

Option B: keep placeholders for empty segments on the write side​

If downstream can't change yet, keep "slot = segment number" constant at assembly time:

snapshot["features"].append(
    df.to_dict("records") if not df.empty else {"__empty__": True}
)

The cost: placeholder objects appear in the snapshot and every consumer must handle them. Fine as a transition; converge on Option A long-term.

Step 3: contract tests over multiple runtime states​

Build snapshot fixtures for "all populated / one empty / several empty" and assert consumers parse all three identically. Testing only the all-full scenario tests nothing.

FAQ​

How do you handle schema drift in data engineering?​

Replace positional contracts with key-based ones: address snapshots, messages, and interfaces by field name or segment identifier, never array index. When upstream changes without notice (empty segments skipped, fields added or removed), key-addressed consumers at worst raise "not found" — they never silently read the wrong data.

What is the difference between schema drift and schema evolution?​

Evolution is explicitly managed versioning: add a field, cut a release, migrate consumers. Drift is passive: upstream changes and downstream quietly misaligns. The slot shifting in this post is textbook drift — nobody touched the contract; the data shape changed.

How do I detect this kind of slot misalignment in a pipeline?​

Two layers: contract tests covering multiple runtime states (all populated / one empty / several empty) asserting identical consumer parsing; and periodic production snapshot audits checking each slot's content against its declared segment. "Content and position disagree" is drift.

While triaging a dashboard lead in an ad-data pipeline: an offer was flagged "suspected paused" in the diagnostic detail, complete with a reinvestment suggestion — yet the "suggested actions" list had no row for it. The detail page pointed at a table that didn't contain it.

Encountered this while building AI Analytics — an LLM-powered analytics platform that surfaces market trends, user behavior, and sales data; the diagnostic detail and the suggestion list come from two adjacent queries in the pipeline.

TL;DR​

Two queries defined the same business predicate ("suspected paused") at different aggregation granularity: the detail query judged at campaign×offer level — any single campaign with no spend in the last week flags it; the candidate list judged at whole-offer level — all campaigns must be spend-free to qualify. One offer still had spend (3.09) in one campaign, so the detail flagged it while the summary skipped it, leaving a dangling reference in the UI. Three fixes: unify the definition at the coarser granularity, make the predicate CTEs verbatim-identical, and verify row-set containment before cross-referencing.

Symptoms​

Two queries, each with its own definition of "suspected paused":

-- Query 4, diagnostic detail: pair granularity (campaign × offer)
WITH consumption_paused_q4 AS (
  SELECT offer_id, campaign_id
  FROM spend_weekly
  GROUP BY offer_id, campaign_id        -- each campaign judged alone
  HAVING SUM(spend) FILTER (WHERE is_last_week) = 0
)

-- Query 5, suggestion candidates: offer granularity (summed across campaigns)
WITH suspected_paused_q5 AS (
  SELECT offer_id
  FROM spend_weekly
  GROUP BY offer_id                     -- the whole offer judged together
  HAVING SUM(spend) FILTER (WHERE is_last_week) = 0
)

The problematic offer hung under multiple campaigns, one of which still had spend (3.09) in its last week:

Query 4 (pair level):   campaign A last-week spend = 0    → flagged ✓, suggestion attached
Query 5 (offer level):  summed last-week spend = 3.09     → excluded ✗
UI: detail says "see the suggestion table"; the suggestion table has no such row

No errors, plausible totals — the dangling reference only surfaces when someone follows a specific detail row.

Root Cause​

The "same source" contract had a coverage gap. The pipeline spec requires the feature-column CTEs of adjacent queries to be verbatim identical — that rule was followed to the letter. But it only covers feature columns; the predicate set (the business judgment before WHERE) was out of scope. "Suspected paused" was implemented twice, at different granularities: pair-level judgment is sensitive to "one campaign has no spend", offer-level judgment to "all campaigns have no spend". For any offer spanning both situations, the two queries must disagree — the overlap zone is mathematically guaranteed.

Cross-query referencing amplified the fork into a dangling reference: the UI treats query 4's rows as details and query 5's table as the entry point, with nobody ever checking "rows(q4) ⊆ rows(q5)". Aggregation-level inconsistency is the classic data-warehouse consistency trap — our earlier post on SUM over ratio columns inflating aggregated metrics is the same disease in another organ: aggregation happening at the wrong level.

Solution​

Step 1: Decide the canonical definition before touching SQL​

The business question has one answer: "should we keep funding this offer" is an offer-level decision, so the predicate must tighten to whole-offer granularity — every attached campaign spend-free in the last week AND no operation annotation rows. Bump the rule version so the change is traceable.

Step 2: Share the predicate CTE verbatim​

Extract the predicate CTE into one text block referenced by both queries; upgrade the contract at the same time: verbatim sharing covers the predicate set (including granularity), not just feature columns. One definition, one place to change.

Step 3: Before cross-referencing, check row-set containment​

Make "marker set ⊆ target row set" a standing check (promote it to a test):

-- Dangling detection: flagged by q4 but absent from q5
SELECT q4.offer_id
FROM consumption_paused_q4 q4
LEFT JOIN suspected_paused_q5 q5 USING (offer_id)
WHERE q5.offer_id IS NULL;

Only when this returns 0 rows does the UI earn the right to render both outputs on one page.

Step 4: Replay against history before shipping​

Replay the new definition over historical snapshots: verify every verdict flip (flagged ↔ normal) is correct, with zero collateral flips and zero dangling references, then re-run production validation.

FAQ​

Why do two SQL queries return different results on the same data?​

Three usual differences: aggregation granularity (different GROUP BY dimensions — campaign×entity vs whole entity in this case), filter logic (different WHERE/HAVING conditions), and point in time. Granularity is the sneakiest: both queries are individually correct, yet their verdicts can contradict.

When do I use HAVING vs WHERE for aggregate conditions in SQL?​

WHERE filters rows before grouping; HAVING filters groups after. But before picking the keyword, pick the granularity — "what counts as one group" determines sensitivity: finer granularity flags more easily (any group qualifies), coarser granularity is conservative (all groups must qualify). Different granularity, opposite verdicts.

How do I keep data definitions consistent across reports?​

One definition per business predicate, predicate CTEs shared verbatim across queries, containment checks (A ⊆ B) before cross-query references, and versioned definition changes replayed over history. Consistency doesn't come from convention — it comes from contracts plus checks.

After deleting a DAG's .py file in Airflow to retire it, the dag_id still hangs around in the Web UI list and the database; even stranger — if you clear the metadata first and delete the file second, the just-cleared rows "come back to life."

Encountered this while building AI Ops — an LLM-powered analytics pipeline where retiring an old report DAG required cleaning its metadata too, otherwise the UI list and scheduled scans stayed polluted by residual rows.

TL;DR​

Airflow's dag-processor periodically scans the DAG folder and re-registers DAGs, and airflow dags reserialize doesn't purge "file-already-deleted" orphan rows — so just deleting the .py file won't make the dag_id vanish from the UI or DB. Conversely, clearing metadata before deleting the file lets the processor re-register the cleared rows on its next scan ("revival"). Correct order: ①delete the file first so the processor stops registering → ②SQL DELETE in foreign-key order → ③run airflow dags reserialize to verify.

Symptoms​

Retiring the shop_report_aggregation DAG — after deleting its .py file:

$ ls /opt/airflow/project/airflow_dags/shop_report_aggregation.py
ls: cannot access '.../shop_report_aggregation.py': No such file or directory

$ # but it's still in the database
$ docker exec cclhub-db psql -U airflow -d airflow -c \
    "SELECT dag_id, is_paused, is_active FROM dag WHERE dag_id='shop_report_aggregation';"
       dag_id            | is_paused | is_active
--------------------------+-----------+-----------
 shop_report_aggregation | f         | t          ← still there

It's not just the dag table — the matching rows in serialized_dag, dag_code, and dag_version are all still there, so the Web UI keeps showing this "deleted" DAG.

Worse is the reverse order — clear metadata first, delete file second:

T0  DELETE FROM dag WHERE dag_id='shop_report_aggregation';   ← cleared
T1  (.py file not deleted yet)
T2  dag-processor scan fires; file exists, dag table has no row → re-registers
T3  SELECT ... FROM dag WHERE dag_id='shop_report_aggregation';   ← it's back (revival)

Root Cause​

Two mechanisms stack up:

1. dag-processor scans and re-registers periodically. Airflow's dag-processor (part of the Scheduler) scans dags_folder on processor_poll_interval (default ~5 min), parses each .py file, and upserts into the metadata tables (dag, serialized_dag, dag_version). As long as the file exists, the next scan rewrites those rows. That's the direct source of "revival" — you clear the row, the file is still there, and the processor re-registers it as a new DAG.

2. reserialize ignores "file-gone" orphan rows. airflow dags reserialize re-serializes existing DAG files and refreshes serialized_dag; it does not delete orphan dag rows whose files have vanished. And airflow dags cleanup only purges expired dag_run history by default — it also leaves the dag / serialized_dag / dag_code / dag_version metadata tables alone. So after you delete the file, the metadata rows become orphans nobody cleans.

┌─ dag-processor ──────────────────────────────┐
│  scans dags_folder                            │
│  ├─ file present → upsert dag / serialized... │  ← source of revival
│  └─ file absent  → skip, no row deletion      │  ← orphan residue
└───────────────────────────────────────────────┘

Conclusion: to actually remove the metadata, you must give the processor no file to register (delete the file first), then manually clean the residual rows.

Solution​

Step 1: Delete the file first​

Make the .py file disappear from the DAG folder so dag-processor stops registering it.

# In production this is usually synced to the volume-mounted DAG folder
# /opt/airflow/project/airflow_dags/ via git pull
git pull   # removes shop_report_aggregation.py from the repo and the folder

# or delete directly (after confirming nothing depends on it)
rm /opt/airflow/project/airflow_dags/shop_report_aggregation.py

Step 2: Clean metadata in foreign-key order​

DELETE in foreign-key dependency order to avoid constraint violations. Deleting dag_run CASCADEs to task_instance:

BEGIN;

-- 1. run history (CASCADEs to task_instance)
DELETE FROM dag_run      WHERE dag_id = 'shop_report_aggregation';

-- 2. serialized DAG
DELETE FROM serialized_dag WHERE dag_id = 'shop_report_aggregation';

-- 3. version
DELETE FROM dag_version  WHERE dag_id = 'shop_report_aggregation';

-- 4. dag main table
DELETE FROM dag          WHERE dag_id = 'shop_report_aggregation';

-- 5. dag_code is keyed by source hash; multiple DAGs may share the same code;
--    only delete hashes no longer referenced by any serialized_dag
DELETE FROM dag_code
WHERE dag_hash NOT IN (SELECT dag_hash FROM serialized_dag);

COMMIT;

Step 3: Verify​

airflow dags reserialize

# confirm the dag row is not rebuilt
docker exec cclhub-db psql -U airflow -d airflow -c \
  "SELECT count(*) FROM dag WHERE dag_id='shop_report_aggregation';"
#  count
# -------
#      0   ✅

After reserialize, dag / serialized_dag / dag_code / dag_version are all 0 for that dag_id, and the next processor scan doesn't rebuild them — the cleanup is stable.

As a side note, on the same pipeline, pandas NaN crashing XCom serialization is another pitfall worth bookmarking.

Notes​

FAQ​

Why does a DAG still show in Airflow after deleting its .py file?​

Deleting the file doesn't clean the database. Rows in dag / serialized_dag / dag_code / dag_version still exist, and the Web UI reads those tables to render the list, so the deleted DAG keeps showing. Airflow has no built-in command to purge these orphan rows automatically; you must SQL DELETE them manually in foreign-key order.

How do I completely delete an Airflow DAG and all its metadata?​

Three steps: 1) delete the .py file so dag-processor stops registering it; 2) SQL DELETE in foreign-key order (dag_run → serialized_dag → dag_version → dag → orphan dag_code); 3) run airflow dags reserialize, then query the dag table to confirm the dag_id row count stays at 0 and isn't rebuilt.

What's the correct order to clean Airflow DAG metadata, and why not clear metadata before deleting the file?​

Delete the file first, then clean metadata. If you reverse it, the .py file still exists, so dag-processor re-registers the cleared dag row on its next scan — the metadata "comes back to life." Only by making the file vanish first (so the processor has nothing to register) and then cleaning the residual rows can you fully retire the DAG.

When an Airflow task calls ti.xcom_push() to pass pandas-processed results downstream, the task crashes outright — ValueError: Out of range float values are not JSON compliant: nan — and the app's custom logs table shows no error at all.

Encountered this while building AI Ops — an LLM-powered analytics pipeline where Airflow DAGs pull SQL data, process it with pandas, and pass results between tasks via XCom.

TL;DR​

XCom serializes with JSON under the hood, and Airflow calls json.dumps(..., allow_nan=False) to follow the JSON spec strictly — which has no NaN / Infinity whatsoever. The moment a float NaN (converted from SQL NULL by pandas) enters the data passed to xcom_push, serialization throws ValueError. Fix: recursively walk the data before push and convert NaN / ±Inf to None (JSON null).

Symptoms​

A quarterly report DAG failed, but the symptom was baffling — in the app's custom logs table, steps 1–4 for that trace were all fine, "analysis done" even logged twice (a retry), then it cut off: step 5 missing, with no error row at all:

trace=91c126c3
├─ step 1  SQL fetch            ✅
├─ step 2  pandas process       ✅
├─ step 3  rule judge           ✅
├─ step 4  LLM analysis done    ✅  ← retried after this
└─ step 5  XCom push result     ❌  ← missing, no error record

The real traceback was only in the Airflow task log:

# inside container /opt/airflow/logs/dag_id=ai_analysis_v2/run_id=.../task_id=analyze_results/attempt=N.log
ValueError: Out of range float values are not JSON compliant: nan
  File ".../ai_analysis_tasks.py", line 142, in analyze_results
    ti.xcom_push(key='sql_metadata', value=result)

The crash landed exactly on ti.xcom_push — the instant the task pushed results into XCom.

Root Cause​

Three layers stack up, all required:

1. The JSON spec has no NaN / Infinity. RFC 8259 only allows finite numeric literals. Python's json.dumps will happily emit bare NaN and Infinity by default, but those are Python-specific extensions, not valid JSON — any strict parser (Airflow included) rejects them.

2. Airflow XCom serializes with allow_nan=False. XCom's default JSON serializer explicitly disables NaN tolerance, so encountering NaN throws ValueError: Out of range float values are not JSON compliant instead of silently emitting invalid JSON.

3. pandas reads SQL NULL as NaN. pandas.read_sql returns float('nan') for SQL NULL columns. Once such a column flows through computation and to_dict('records') into the result object, NaN hitches a ride into xcom_push:

import pandas as pd

# A SQL NULL cell → pandas reads it as NaN
df = pd.DataFrame({"ad_roi": [1.2, None, 0.8]})
records = df.to_dict("records")
# [{'ad_roi': 1.2}, {'ad_roi': nan}, {'ad_roi': 0.8}]   ← nan slipped in

# downstream task crashes on push
ti.xcom_push(key="result", value=records)
# ValueError: Out of range float values are not JSON compliant: nan

This stayed latent for a long time because the data usually had values in those columns; it only surfaced when a client had zero ad spend for an entire quarter and ad_roi came back NULL across the board — the first time NaN entered the XCom path at scale.

Why no error in the logs table? Because the crash happens during XCom serialization, outside the task function's try/except — the exception bubbles straight up to the Airflow scheduler and only lands in Airflow's own task log. The app's custom logs table catch never gets a chance to record it. That's what makes this failure so confusing: it looks "silent."

Solution​

Scrub all NaN / ±Inf from the data before it enters XCom.

1. Write a pure recursive cleaner​

import math

def json_safe_value(obj):
    """
    Recursively convert NaN / +Inf / -Inf to None so the data is
    strictly JSON-serializable. Handles dict / list / tuple / scalar;
    unknown types pass through unchanged.
    """
    if isinstance(obj, float):
        if math.isnan(obj) or math.isinf(obj):
            return None
        return obj
    if isinstance(obj, dict):
        return {k: json_safe_value(v) for k, v in obj.items()}
    if isinstance(obj, (list, tuple)):
        return [json_safe_value(v) for v in obj]
    return obj

Why not df.fillna(None)? Because fillna(None) on numeric columns is unstable across pandas versions and dtypes — sometimes it coerces the dtype instead of nulling values. It also only handles DataFrames, not floats already nested inside dicts/lists after to_dict. Recursive cleaning at the "data is now native Python structures" layer is the most robust fallback.

2. Centralize the guard before push​

The worry-free approach is to hang the cleanup on the single chokepoint all xcom_push calls go through, rather than remembering to call it at every push site:

def push_safe(ti, key, value):
    """Clean NaN/Inf before XCom push to prevent serialization crashes."""
    ti.xcom_push(key=key, value=json_safe_value(value))

# inside the task
push_safe(ti, "sql_metadata", result)
push_safe(ti, "processor_output", processor_result)

3. Fix the "silent failure" observability gap​

Fixing serialization alone isn't enough — the gap where exceptions outside try/except never reach the app's logs table must be closed too. Attach a failure decorator that logs the top-level exception to your table before re-raising:

import functools
import logging

logger = logging.getLogger(__name__)

def log_task_failure(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        try:
            return fn(*args, **kwargs)
        except Exception:
            logger.error("task %s failed", fn.__name__, exc_info=True)
            # write the traceback into the app's custom logs table here
            raise
    return wrapper

@log_task_failure
def analyze_results(**context):
    ...

Now if another exception slips outside a catch, the app's logs table still gets an error row — no more "silent failure."

After the fix, rerunning the same conf: DAG all green, DB write success, and the formerly-NaN ad_roi lands as null in the database; downstream is happy.

On the same Airflow analytics pipeline, this isn't the only way data silently misbehaves — PostgresHook silently dropping multi-statement SQL results is another classic.

Notes​

FAQ​

How to fix Airflow "Out of range float values are not JSON compliant"?​

XCom serializes with json.dumps(allow_nan=False) and ran into NaN / Infinity, which the JSON spec does not allow. The usual root cause is pandas reading a SQL NULL into float('nan') that then flows into xcom_push. Fix it by recursively converting NaN / ±Inf to None (JSON null) before push, centralized in a pure json_safe_value helper.

Why does my Airflow task fail but my custom logs table has no error?​

If the exception happens during XCom serialization, outside the task function's try/except, it only bubbles up to the Airflow scheduler and lands in the Airflow task log (inside the container at /opt/airflow/logs/). The app's custom logs table catch never sees it, so it looks like a "silent failure." To triage, read the Airflow task log traceback directly instead of only checking app logs.

Can Airflow XCom store pandas NaN directly?​

No. XCom defaults to JSON serialization, and the JSON spec only has finite numbers — no NaN / Infinity. The right fix is to convert NaN to None (JSON null) before push. Switching to binary object serialization sidesteps the type limit but produces unreadable, non-portable values with deserialization security risk; not recommended for production.

When you validate LLM tool_call / function call output with Zod, the model occasionally emits an extra field — say you only defined amount / category, and it also fills in note — and the whole validation fails, the action is silently dropped, and the user just gets "not recognized." In reality an entire tool_call was whole-rejected.

Encountered this while building Life — a natural-language bookkeeping and wellness assistant where you just talk to record entries and the AI extracts amount, category, and account. When a user said "delete that coffee from yesterday," the model slipped an extra note: "coffee" into the delete locator, trying to locate by remark.

TL;DR​

Zod's .strict() means "this object must not contain any unknown keys — one extra throws." That constraint fits validating clients you fully control, but LLM function call output is model-generated and inherently uncontrollable — it fills in fields it "thinks should be there," especially when multiple tools share similar schemas. One irrelevant field kills the whole tool_call, validation returns null, and the action is silently lost. Fix: drop .strict() and use Zod's default strip (silently removes unknown keys) for tolerance, paired with safeParse as a fallback.

Symptoms​

The locator schema for delete/update defines a few known fields, tightened with .strict():

import { z } from "zod";

// ❌ dangerous: with .strict()
const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
}).strict();   // ← any unknown key throws

// parse the LLM's tool_call arguments
function parseToolCall(raw: unknown) {
  const parsed = LocatorSchema.safeParse(raw);
  if (!parsed.success) {
    return null;   // ← the whole tool_call is dropped
  }
  return parsed.data;
}

The user says "delete that coffee from yesterday," and the model produces a reasonable but extra-fielded output:

{
  "date": "yesterday",
  "noteContains": "coffee",
  "note": "coffee"
}

The model filled both noteContains (in schema) and note (out of schema, which it thought should exist). .strict() rejects the unknown key note outright, parseToolCall returns null, and the delete action is silently dropped — the user gets "not recognized" when it was actually a whole-reject.

Root Cause​

.strict() changes Zod's policy on unknown keys, and LLM output naturally carries unknown keys.

Zod z.object() has three policies for unknown keys:

.strict() is designed for "contract strictness" — the server defines which fields exist, the client should supply only those, and anything extra is a breach. That logic holds for traditional APIs because the client is developer-written and can be held to the contract.

But LLM function calling flips the premise:

1. The output comes from model generation, not a developer-written client. The model guesses what to fill based on the schema's description and examples; schemas reused across domains (e.g. a locator shared by budget / mood / todo) confuse it further, so it fills in fields it "thinks should be there."
2. Wrong fields are the norm, not an exception. The model occasionally emitting an extra note or omitting an optional field is expected behavior in LLM apps and shouldn't be punished by failing the whole call.
3. The failure is silently swallowed. After safeParse fails and returns null, the upstream can only vaguely say "not recognized," while the real cause (an unknown key) sits unseen in parsed.error.

LLM output { date, noteContains, note }
                        │
                        ▼
         .strict() hits unknown key "note"
                        │
                        ▼
           safeParse → { success: false }
                        │
                        ▼
          parseToolCall returns null (action dropped)
                        │
                        ▼
       user gets "not recognized" (actually a whole-reject)

Solution​

1. Drop .strict(), use default strip for tolerance​

// ✅ recommended: no .strict(), Zod defaults to stripping unknown keys
const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
});
// the extra "note" is silently removed; known fields parse normally

After dropping .strict(), "delete that coffee from yesterday" parses cleanly into { date, noteContains }; the extra note is stripped and the delete action runs correctly.

2. If unknown keys are useful, keep them with .passthrough()​

When the extra field actually carries semantics you want to use (e.g. the model filled note to express "locate by remark"), don't drop it — keep it and decide how to consume it:

const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
}).passthrough();   // keep unknown keys; parsed.data.note is still readable

Even better, promote it to a known field — if the model keeps filling some unknown key, the schema is missing a capability slot, so add it (here noteContains was added after absorbing the "locate by remark" need).

3. Make failures observable — don't silently return null​

Regardless of policy, when safeParse fails, log the specific error instead of swallowing it into null:

function parseToolCall(raw: unknown) {
  const parsed = LocatorSchema.safeParse(raw);
  if (!parsed.success) {
    // log Zod's concrete error (which key, what problem) for triage
    logger.warn(
      { raw, issues: parsed.error.issues },
      "locator parse failed"
    );
    return null;
  }
  return parsed.data;
}

Now when something goes wrong, the log has the full issues (including the unknown-key path), instead of an unactionable "not recognized."

After the fix, "delete that coffee from yesterday" → delete_record { locator: { noteContains: "coffee" } } parses correctly with no silent drop.

Notes​

FAQ​

Why does Zod .strict() make LLM output validation fail?​

.strict() requires an object to have no unknown keys — one extra throws. LLM function call output is model-generated, guessing from the schema description, and routinely includes fields it thinks should be there (especially with cross-domain schemas). The moment an unknown key appears, .strict() fails the entire validation and drops the whole tool_call.

Should I use strict when validating LLM function calling output with Zod?​

Not recommended. .strict() suits validating clients you fully control (developer-written code can be held to a contract), but LLM output is uncontrollable and over/under-filling is the norm. Drop .strict() and use Zod's default strip (silently removes unknown keys) for better tolerance; if unknown keys carry semantics you want, use .passthrough() to keep them, or promote them to known fields.

Does Zod strip or throw on unknown fields by default?​

Default is strip — it silently removes unknown keys without error. .strict() makes it throw on unknown keys; .passthrough() keeps them as-is. For uncontrollable output like LLM generations, prefer default strip or passthrough over .strict(), which kills the whole payload over a single irrelevant field.

When an Airflow DAG reads a .sql template file as a single string and passes it to PostgresHook.get_pandas_df(), prior SELECT results are silently dropped — the DAG reports "SQL query returned no results", but copying the same SQL into psql returns data normally.

Encountered this while building AI Analytics — an LLM-powered analytics pipeline where an Airflow DAG reads multi-query report templates from .sql files and executes them.

TL;DR​

PostgresHook.get_pandas_df(sql) internally calls pandas.read_sql(sql, conn) → psycopg2 cursor.execute(sql). When sql is a single string with multiple ;-separated SELECTs, the DBAPI only exposes the cursor of the last result set — prior query results are silently dropped with no error. Fix: split by top-level semicolons into a list[str] and call get_pandas_df per statement, or pass the list directly so DbApiHook runs them sequentially.

Symptom​

The DAG task executing shop_monthly_overview.sql reports "SQL query returned no results":

sql_count = 1   ← template clearly contains 4 queries
result = "❌ SQL query returned no results"

But the same SQL pasted into psql against the same database with the same parameters returns data for all 4 SELECTs.

Reproduction​

Verify get_pandas_df behavior with multi-statement SQL directly inside the Airflow container:

from airflow.providers.postgres.hooks.postgres import PostgresHook

hook = PostgresHook(postgres_conn_id="postgres_default")

# Three SELECTs concatenated into one string
sql = "SELECT 1 AS a; SELECT 2 AS b; SELECT 99 AS c WHERE 1=0;"

df = hook.get_pandas_df(sql)
print(df.columns.tolist())  # ['c']  ← only got the last statement's columns
print(df)                    # Empty  ← the last statement itself returns 0 rows

Expected three result sets, got only the last one (SELECT 99 ... WHERE 1=0, 0 rows). The first two completely disappear with no error or warning.

Root Cause​

The call chain is PostgresHook.get_pandas_df → DbApiHook.get_pandas_df → pandas.io.sql.read_sql → psycopg2 cursor.execute(sql).

The DBAPI protocol (PEP 249) allows execute to accept a string with multiple ;-separated statements. PostgreSQL executes all of them, but the cursor only exposes the last result set — this is inherent PostgreSQL wire protocol behavior, not an Airflow or pandas bug.

┌────────────────────────────────────────────────────┐
│ SELECT 1;  ← executed, result set 1 dropped at once │
│ SELECT 2;  ← executed, result set 2 dropped at once │
│ SELECT 99 WHERE 1=0;  ← executed, result set 3 exposed │
└────────────────────────────────────────────────────┘
                       ↓
            pandas.read_sql only fetches result set 3

The root cause in our code is task_execute_sql reading the entire .sql file as a single string and passing it to get_pandas_df:

# ❌ Problematic code
sql_text = open(sql_path).read()  # full text with 4 SELECTs
df = pg_hook.get_pandas_df(sql_text)  # only gets the last result

Why does psql return data? Because the psql frontend actively iterates through all result sets and prints them one by one, while a DBAPI cursor does not.

Solution​

Option A (Recommended): Split by top-level semicolon, execute one by one​

Suitable for .sql template files — they contain comments, quotes, and multiple queries that need robust splitting.

def split_sql_statements(sql: str) -> list:
    """
    Split SQL by top-level semicolons, correctly handling:
    - Semicolons inside single-quoted strings ('a;b' is not split)
    - SQL-standard '' escape ('it''s' is not split)
    - Semicolons inside -- line comments (-- note; not split is not split)
    """
    statements = []
    buf = []
    i, n = 0, len(sql)
    in_quote = False

    while i < n:
        ch = sql[i]

        # Inside a single-quoted string
        if in_quote:
            buf.append(ch)
            if ch == "'":
                # '' = literal single quote, does not end the string
                if i + 1 < n and sql[i + 1] == "'":
                    buf.append(sql[i + 1])
                    i += 2
                    continue
                in_quote = False
            i += 1
            continue

        # Top level
        if ch == "'":
            in_quote = True
            buf.append(ch)
        elif ch == '-' and i + 1 < n and sql[i + 1] == '-':
            # Line comment, swallow to end of line (; inside is not a split point)
            while i < n and sql[i] != '\n':
                buf.append(sql[i])
                i += 1
            continue
        elif ch == ';':
            stmt = ''.join(buf).strip()
            if stmt:
                statements.append(stmt)
            buf = []
            i += 1
            continue
        else:
            buf.append(ch)
        i += 1

    # Trailing block without a final semicolon
    stmt = ''.join(buf).strip()
    if stmt:
        statements.append(stmt)

    return statements


# Caller
sql_text = open(sql_path).read()
statements = split_sql_statements(sql_text)

# Execute one by one, collect all results
all_results = []
for idx, stmt in enumerate(statements, start=1):
    df = pg_hook.get_pandas_df(stmt)
    if not df.empty:
        all_results.append({
            "sql_index": idx,
            "sql": stmt,
            "data": df.to_dict("records"),
            "columns": df.columns.tolist(),
            "row_count": len(df),
        })

Option B: Pass a list directly to DbApiHook​

Airflow's DbApiHook.run and get_records accept list[str] and execute sequentially — but get_pandas_df return behavior in list mode is inconsistent across providers. For production, Option A gives you full control.

Why not sqlparse.split?​

Community answers often recommend sqlparse.split(sqlparse.format(sql, strip_comments=True)), but strip_comments=True discards comments. If your downstream processor depends on metadata in comments (e.g. -- dimension: shop), you lose context. A hand-rolled splitter preserves the original comment text and gives you control.

Caveats​

FAQ​

How do I execute multiple SQL statements in Airflow PostgresHook?​

Pass list[str] instead of a single string. DbApiHook.get_pandas_df and run accept sql as a list and execute sequentially; a single string with semicolon-separated statements causes psycopg2 to return only the last result set. For production, split yourself and call per-statement so you control result aggregation and sql_index.

Why does get_pandas_df only return the last result for multi-statement SQL?​

pandas.io.sql.read_sql calls psycopg2 cursor.execute with the full string; the DBAPI protocol only exposes the cursor of the last result set for multi-statement execution, and prior SELECT results are dropped by the server immediately, with no error or warning. psql returns data because its frontend actively iterates all result sets, while a DBAPI cursor does not.

How do I split SQL by semicolon safely with comments and quotes?​

Scan character by character and split only at top-level semicolons outside single-quoted strings and -- line comments. Single-quote literals use the SQL-standard '' escape; do not use str.split(';'), it will mis-cut semicolons inside comments and strings. If you use sqlparse.split, note that strip_comments=True discards the original comment text.

While building a WooCommerce product import tool for a client, POST /wp-json/wp/v2/media would succeed for the first few images, then suddenly return 405 Not Allowed for all subsequent requests.

TL;DR​

Hostinger CDN (hcdn) blocks POST /wp-json/wp/v2/media requests by default. The response headers server: hcdn and x-hcdn-request-id are the smoking gun. Disable CDN or contact Hostinger support to whitelist /wp-json/* POST requests.

The Problem​

Uploading images to WordPress Media Library via REST API:

curl -X POST 'https://example.com/wp-json/wp/v2/media' \
  -u 'user:app_password' \
  -H 'Content-Disposition: attachment; filename="product-01.jpg"' \
  -H 'Content-Type: image/jpeg' \
  --data-binary @image.jpg

The first 2-4 images return 201 Created, then all subsequent requests fail with:

<html>
<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>

This "partial success" pattern is misleading — it looks like rate limiting, but the real cause is entirely different.

Root Cause​

Using curl -v to inspect the full response headers revealed:

< HTTP/2 405
< server: hcdn
< x-hcdn-request-id: cfc5ad1198938cd9f1e02ce71ed0ae61-kul-edge1

Key findings:

• server: hcdn — This is Hostinger's custom CDN (hcdn), not the origin nginx server
• x-hcdn-request-id — CDN edge node ID (kul-edge1 = Kuala Lumpur), confirming the request was blocked at the CDN layer before reaching WordPress

Hostinger CDN's default security rules block POST requests to /wp-json/wp/v2/media. The initial successes were likely due to CDN rule cold-start or cache misses.

Solution​

Option 1: Disable CDN (Quick Fix)​

Go to Hostinger hPanel → Website → CDN → Disable.

This takes effect immediately but removes CDN acceleration. Suitable for staging environments or emergency fixes.

Option 2: Contact Hostinger Support to Whitelist API Paths (Recommended)​

Submit a support ticket requesting to whitelist POST requests to /wp-json/*. Hostinger's Manage panel currently doesn't offer custom CDN rule configuration — you must go through support.

Option 3: Add Retry Logic in Code (Defensive Measure)​

Even with correct CDN configuration, retry logic handles occasional CDN throttling:

import time
import random

def upload_image(url, image_bytes, filename, auth, max_retries=3):
    for attempt in range(max_retries):
        resp = httpx.post(
            url,
            content=image_bytes,
            headers={
                "Content-Disposition": f'attachment; filename="{filename}"',
                "Content-Type": "image/jpeg",
            },
            auth=auth,
            timeout=30,
        )
        if resp.status_code != 405:
            return resp
        delay = 3 * (attempt + 1) + random.uniform(0, 2)
        time.sleep(delay)
    resp.raise_for_status()

Troubleshooting Journey​

This issue led down several dead ends. Here's the full排查 path for reference:

The turning point was using curl -v and spotting server: hcdn — only then did we realize the requests never reached WordPress.

FAQ​

Why does WordPress REST API image upload return 405 Not Allowed?​

Check the server field in response headers. If it shows hcdn (Hostinger CDN) or another CDN identifier, the request is being blocked at the CDN layer before reaching WordPress. Disable the CDN or contact your hosting provider to whitelist the endpoint.

How to tell if 405 comes from CDN or WordPress?​

Use curl -v and inspect response headers: a server value of hcdn, cloudflare, or other CDN identifiers indicates CDN-level blocking; a server value of nginx/apache with X-WP-* or X-RateLimit-* headers means the request reached WordPress.

Encountered this issue while building a WooCommerce product import tool for a client. If you're also developing with Hostinger + WordPress and running into REST API issues, reach out.