15 posts tagged with "Bug Fix"

After deleting a DAG's .py file in Airflow to retire it, the dag_id still hangs around in the Web UI list and the database; even stranger — if you clear the metadata first and delete the file second, the just-cleared rows "come back to life."

Encountered this while building AI Ops — an LLM-powered analytics pipeline where retiring an old report DAG required cleaning its metadata too, otherwise the UI list and scheduled scans stayed polluted by residual rows.

TL;DR​

Airflow's dag-processor periodically scans the DAG folder and re-registers DAGs, and airflow dags reserialize doesn't purge "file-already-deleted" orphan rows — so just deleting the .py file won't make the dag_id vanish from the UI or DB. Conversely, clearing metadata before deleting the file lets the processor re-register the cleared rows on its next scan ("revival"). Correct order: ①delete the file first so the processor stops registering → ②SQL DELETE in foreign-key order → ③run airflow dags reserialize to verify.

Symptoms​

Retiring the shop_report_aggregation DAG — after deleting its .py file:

$ ls /opt/airflow/project/airflow_dags/shop_report_aggregation.py
ls: cannot access '.../shop_report_aggregation.py': No such file or directory

$ # but it's still in the database
$ docker exec cclhub-db psql -U airflow -d airflow -c \
    "SELECT dag_id, is_paused, is_active FROM dag WHERE dag_id='shop_report_aggregation';"
       dag_id            | is_paused | is_active
--------------------------+-----------+-----------
 shop_report_aggregation | f         | t          ← still there

It's not just the dag table — the matching rows in serialized_dag, dag_code, and dag_version are all still there, so the Web UI keeps showing this "deleted" DAG.

Worse is the reverse order — clear metadata first, delete file second:

T0  DELETE FROM dag WHERE dag_id='shop_report_aggregation';   ← cleared
T1  (.py file not deleted yet)
T2  dag-processor scan fires; file exists, dag table has no row → re-registers
T3  SELECT ... FROM dag WHERE dag_id='shop_report_aggregation';   ← it's back (revival)

Root Cause​

Two mechanisms stack up:

1. dag-processor scans and re-registers periodically. Airflow's dag-processor (part of the Scheduler) scans dags_folder on processor_poll_interval (default ~5 min), parses each .py file, and upserts into the metadata tables (dag, serialized_dag, dag_version). As long as the file exists, the next scan rewrites those rows. That's the direct source of "revival" — you clear the row, the file is still there, and the processor re-registers it as a new DAG.

2. reserialize ignores "file-gone" orphan rows. airflow dags reserialize re-serializes existing DAG files and refreshes serialized_dag; it does not delete orphan dag rows whose files have vanished. And airflow dags cleanup only purges expired dag_run history by default — it also leaves the dag / serialized_dag / dag_code / dag_version metadata tables alone. So after you delete the file, the metadata rows become orphans nobody cleans.

┌─ dag-processor ──────────────────────────────┐
│  scans dags_folder                            │
│  ├─ file present → upsert dag / serialized... │  ← source of revival
│  └─ file absent  → skip, no row deletion      │  ← orphan residue
└───────────────────────────────────────────────┘

Conclusion: to actually remove the metadata, you must give the processor no file to register (delete the file first), then manually clean the residual rows.

Solution​

Step 1: Delete the file first​

Make the .py file disappear from the DAG folder so dag-processor stops registering it.

# In production this is usually synced to the volume-mounted DAG folder
# /opt/airflow/project/airflow_dags/ via git pull
git pull   # removes shop_report_aggregation.py from the repo and the folder

# or delete directly (after confirming nothing depends on it)
rm /opt/airflow/project/airflow_dags/shop_report_aggregation.py

Step 2: Clean metadata in foreign-key order​

DELETE in foreign-key dependency order to avoid constraint violations. Deleting dag_run CASCADEs to task_instance:

BEGIN;

-- 1. run history (CASCADEs to task_instance)
DELETE FROM dag_run      WHERE dag_id = 'shop_report_aggregation';

-- 2. serialized DAG
DELETE FROM serialized_dag WHERE dag_id = 'shop_report_aggregation';

-- 3. version
DELETE FROM dag_version  WHERE dag_id = 'shop_report_aggregation';

-- 4. dag main table
DELETE FROM dag          WHERE dag_id = 'shop_report_aggregation';

-- 5. dag_code is keyed by source hash; multiple DAGs may share the same code;
--    only delete hashes no longer referenced by any serialized_dag
DELETE FROM dag_code
WHERE dag_hash NOT IN (SELECT dag_hash FROM serialized_dag);

COMMIT;

Step 3: Verify​

airflow dags reserialize

# confirm the dag row is not rebuilt
docker exec cclhub-db psql -U airflow -d airflow -c \
  "SELECT count(*) FROM dag WHERE dag_id='shop_report_aggregation';"
#  count
# -------
#      0   ✅

After reserialize, dag / serialized_dag / dag_code / dag_version are all 0 for that dag_id, and the next processor scan doesn't rebuild them — the cleanup is stable.

As a side note, on the same pipeline, pandas NaN crashing XCom serialization is another pitfall worth bookmarking.

Notes​

FAQ​

Why does a DAG still show in Airflow after deleting its .py file?​

Deleting the file doesn't clean the database. Rows in dag / serialized_dag / dag_code / dag_version still exist, and the Web UI reads those tables to render the list, so the deleted DAG keeps showing. Airflow has no built-in command to purge these orphan rows automatically; you must SQL DELETE them manually in foreign-key order.

How do I completely delete an Airflow DAG and all its metadata?​

Three steps: 1) delete the .py file so dag-processor stops registering it; 2) SQL DELETE in foreign-key order (dag_run → serialized_dag → dag_version → dag → orphan dag_code); 3) run airflow dags reserialize, then query the dag table to confirm the dag_id row count stays at 0 and isn't rebuilt.

What's the correct order to clean Airflow DAG metadata, and why not clear metadata before deleting the file?​

Delete the file first, then clean metadata. If you reverse it, the .py file still exists, so dag-processor re-registers the cleared dag row on its next scan — the metadata "comes back to life." Only by making the file vanish first (so the processor has nothing to register) and then cleaning the residual rows can you fully retire the DAG.

When an Airflow task calls ti.xcom_push() to pass pandas-processed results downstream, the task crashes outright — ValueError: Out of range float values are not JSON compliant: nan — and the app's custom logs table shows no error at all.

Encountered this while building AI Ops — an LLM-powered analytics pipeline where Airflow DAGs pull SQL data, process it with pandas, and pass results between tasks via XCom.

TL;DR​

XCom serializes with JSON under the hood, and Airflow calls json.dumps(..., allow_nan=False) to follow the JSON spec strictly — which has no NaN / Infinity whatsoever. The moment a float NaN (converted from SQL NULL by pandas) enters the data passed to xcom_push, serialization throws ValueError. Fix: recursively walk the data before push and convert NaN / ±Inf to None (JSON null).

Symptoms​

A quarterly report DAG failed, but the symptom was baffling — in the app's custom logs table, steps 1–4 for that trace were all fine, "analysis done" even logged twice (a retry), then it cut off: step 5 missing, with no error row at all:

trace=91c126c3
├─ step 1  SQL fetch            ✅
├─ step 2  pandas process       ✅
├─ step 3  rule judge           ✅
├─ step 4  LLM analysis done    ✅  ← retried after this
└─ step 5  XCom push result     ❌  ← missing, no error record

The real traceback was only in the Airflow task log:

# inside container /opt/airflow/logs/dag_id=ai_analysis_v2/run_id=.../task_id=analyze_results/attempt=N.log
ValueError: Out of range float values are not JSON compliant: nan
  File ".../ai_analysis_tasks.py", line 142, in analyze_results
    ti.xcom_push(key='sql_metadata', value=result)

The crash landed exactly on ti.xcom_push — the instant the task pushed results into XCom.

Root Cause​

Three layers stack up, all required:

1. The JSON spec has no NaN / Infinity. RFC 8259 only allows finite numeric literals. Python's json.dumps will happily emit bare NaN and Infinity by default, but those are Python-specific extensions, not valid JSON — any strict parser (Airflow included) rejects them.

2. Airflow XCom serializes with allow_nan=False. XCom's default JSON serializer explicitly disables NaN tolerance, so encountering NaN throws ValueError: Out of range float values are not JSON compliant instead of silently emitting invalid JSON.

3. pandas reads SQL NULL as NaN. pandas.read_sql returns float('nan') for SQL NULL columns. Once such a column flows through computation and to_dict('records') into the result object, NaN hitches a ride into xcom_push:

import pandas as pd

# A SQL NULL cell → pandas reads it as NaN
df = pd.DataFrame({"ad_roi": [1.2, None, 0.8]})
records = df.to_dict("records")
# [{'ad_roi': 1.2}, {'ad_roi': nan}, {'ad_roi': 0.8}]   ← nan slipped in

# downstream task crashes on push
ti.xcom_push(key="result", value=records)
# ValueError: Out of range float values are not JSON compliant: nan

This stayed latent for a long time because the data usually had values in those columns; it only surfaced when a client had zero ad spend for an entire quarter and ad_roi came back NULL across the board — the first time NaN entered the XCom path at scale.

Why no error in the logs table? Because the crash happens during XCom serialization, outside the task function's try/except — the exception bubbles straight up to the Airflow scheduler and only lands in Airflow's own task log. The app's custom logs table catch never gets a chance to record it. That's what makes this failure so confusing: it looks "silent."

Solution​

Scrub all NaN / ±Inf from the data before it enters XCom.

1. Write a pure recursive cleaner​

import math

def json_safe_value(obj):
    """
    Recursively convert NaN / +Inf / -Inf to None so the data is
    strictly JSON-serializable. Handles dict / list / tuple / scalar;
    unknown types pass through unchanged.
    """
    if isinstance(obj, float):
        if math.isnan(obj) or math.isinf(obj):
            return None
        return obj
    if isinstance(obj, dict):
        return {k: json_safe_value(v) for k, v in obj.items()}
    if isinstance(obj, (list, tuple)):
        return [json_safe_value(v) for v in obj]
    return obj

Why not df.fillna(None)? Because fillna(None) on numeric columns is unstable across pandas versions and dtypes — sometimes it coerces the dtype instead of nulling values. It also only handles DataFrames, not floats already nested inside dicts/lists after to_dict. Recursive cleaning at the "data is now native Python structures" layer is the most robust fallback.

2. Centralize the guard before push​

The worry-free approach is to hang the cleanup on the single chokepoint all xcom_push calls go through, rather than remembering to call it at every push site:

def push_safe(ti, key, value):
    """Clean NaN/Inf before XCom push to prevent serialization crashes."""
    ti.xcom_push(key=key, value=json_safe_value(value))

# inside the task
push_safe(ti, "sql_metadata", result)
push_safe(ti, "processor_output", processor_result)

3. Fix the "silent failure" observability gap​

Fixing serialization alone isn't enough — the gap where exceptions outside try/except never reach the app's logs table must be closed too. Attach a failure decorator that logs the top-level exception to your table before re-raising:

import functools
import logging

logger = logging.getLogger(__name__)

def log_task_failure(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        try:
            return fn(*args, **kwargs)
        except Exception:
            logger.error("task %s failed", fn.__name__, exc_info=True)
            # write the traceback into the app's custom logs table here
            raise
    return wrapper

@log_task_failure
def analyze_results(**context):
    ...

Now if another exception slips outside a catch, the app's logs table still gets an error row — no more "silent failure."

After the fix, rerunning the same conf: DAG all green, DB write success, and the formerly-NaN ad_roi lands as null in the database; downstream is happy.

On the same Airflow analytics pipeline, this isn't the only way data silently misbehaves — PostgresHook silently dropping multi-statement SQL results is another classic.

Notes​

FAQ​

How to fix Airflow "Out of range float values are not JSON compliant"?​

XCom serializes with json.dumps(allow_nan=False) and ran into NaN / Infinity, which the JSON spec does not allow. The usual root cause is pandas reading a SQL NULL into float('nan') that then flows into xcom_push. Fix it by recursively converting NaN / ±Inf to None (JSON null) before push, centralized in a pure json_safe_value helper.

Why does my Airflow task fail but my custom logs table has no error?​

If the exception happens during XCom serialization, outside the task function's try/except, it only bubbles up to the Airflow scheduler and lands in the Airflow task log (inside the container at /opt/airflow/logs/). The app's custom logs table catch never sees it, so it looks like a "silent failure." To triage, read the Airflow task log traceback directly instead of only checking app logs.

Can Airflow XCom store pandas NaN directly?​

No. XCom defaults to JSON serialization, and the JSON spec only has finite numbers — no NaN / Infinity. The right fix is to convert NaN to None (JSON null) before push. Switching to binary object serialization sidesteps the type limit but produces unreadable, non-portable values with deserialization security risk; not recommended for production.

When you validate LLM tool_call / function call output with Zod, the model occasionally emits an extra field — say you only defined amount / category, and it also fills in note — and the whole validation fails, the action is silently dropped, and the user just gets "not recognized." In reality an entire tool_call was whole-rejected.

Encountered this while building Life — a natural-language bookkeeping and wellness assistant where you just talk to record entries and the AI extracts amount, category, and account. When a user said "delete that coffee from yesterday," the model slipped an extra note: "coffee" into the delete locator, trying to locate by remark.

TL;DR​

Zod's .strict() means "this object must not contain any unknown keys — one extra throws." That constraint fits validating clients you fully control, but LLM function call output is model-generated and inherently uncontrollable — it fills in fields it "thinks should be there," especially when multiple tools share similar schemas. One irrelevant field kills the whole tool_call, validation returns null, and the action is silently lost. Fix: drop .strict() and use Zod's default strip (silently removes unknown keys) for tolerance, paired with safeParse as a fallback.

Symptoms​

The locator schema for delete/update defines a few known fields, tightened with .strict():

import { z } from "zod";

// ❌ dangerous: with .strict()
const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
}).strict();   // ← any unknown key throws

// parse the LLM's tool_call arguments
function parseToolCall(raw: unknown) {
  const parsed = LocatorSchema.safeParse(raw);
  if (!parsed.success) {
    return null;   // ← the whole tool_call is dropped
  }
  return parsed.data;
}

The user says "delete that coffee from yesterday," and the model produces a reasonable but extra-fielded output:

{
  "date": "yesterday",
  "noteContains": "coffee",
  "note": "coffee"
}

The model filled both noteContains (in schema) and note (out of schema, which it thought should exist). .strict() rejects the unknown key note outright, parseToolCall returns null, and the delete action is silently dropped — the user gets "not recognized" when it was actually a whole-reject.

Root Cause​

.strict() changes Zod's policy on unknown keys, and LLM output naturally carries unknown keys.

Zod z.object() has three policies for unknown keys:

.strict() is designed for "contract strictness" — the server defines which fields exist, the client should supply only those, and anything extra is a breach. That logic holds for traditional APIs because the client is developer-written and can be held to the contract.

But LLM function calling flips the premise:

1. The output comes from model generation, not a developer-written client. The model guesses what to fill based on the schema's description and examples; schemas reused across domains (e.g. a locator shared by budget / mood / todo) confuse it further, so it fills in fields it "thinks should be there."
2. Wrong fields are the norm, not an exception. The model occasionally emitting an extra note or omitting an optional field is expected behavior in LLM apps and shouldn't be punished by failing the whole call.
3. The failure is silently swallowed. After safeParse fails and returns null, the upstream can only vaguely say "not recognized," while the real cause (an unknown key) sits unseen in parsed.error.

LLM output { date, noteContains, note }
                        │
                        ▼
         .strict() hits unknown key "note"
                        │
                        ▼
           safeParse → { success: false }
                        │
                        ▼
          parseToolCall returns null (action dropped)
                        │
                        ▼
       user gets "not recognized" (actually a whole-reject)

Solution​

1. Drop .strict(), use default strip for tolerance​

// ✅ recommended: no .strict(), Zod defaults to stripping unknown keys
const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
});
// the extra "note" is silently removed; known fields parse normally

After dropping .strict(), "delete that coffee from yesterday" parses cleanly into { date, noteContains }; the extra note is stripped and the delete action runs correctly.

2. If unknown keys are useful, keep them with .passthrough()​

When the extra field actually carries semantics you want to use (e.g. the model filled note to express "locate by remark"), don't drop it — keep it and decide how to consume it:

const LocatorSchema = z.object({
  date: z.string().optional(),
  category: z.string().optional(),
  noteContains: z.string().optional(),
}).passthrough();   // keep unknown keys; parsed.data.note is still readable

Even better, promote it to a known field — if the model keeps filling some unknown key, the schema is missing a capability slot, so add it (here noteContains was added after absorbing the "locate by remark" need).

3. Make failures observable — don't silently return null​

Regardless of policy, when safeParse fails, log the specific error instead of swallowing it into null:

function parseToolCall(raw: unknown) {
  const parsed = LocatorSchema.safeParse(raw);
  if (!parsed.success) {
    // log Zod's concrete error (which key, what problem) for triage
    logger.warn(
      { raw, issues: parsed.error.issues },
      "locator parse failed"
    );
    return null;
  }
  return parsed.data;
}

Now when something goes wrong, the log has the full issues (including the unknown-key path), instead of an unactionable "not recognized."

After the fix, "delete that coffee from yesterday" → delete_record { locator: { noteContains: "coffee" } } parses correctly with no silent drop.

Notes​

FAQ​

Why does Zod .strict() make LLM output validation fail?​

.strict() requires an object to have no unknown keys — one extra throws. LLM function call output is model-generated, guessing from the schema description, and routinely includes fields it thinks should be there (especially with cross-domain schemas). The moment an unknown key appears, .strict() fails the entire validation and drops the whole tool_call.

Should I use strict when validating LLM function calling output with Zod?​

Not recommended. .strict() suits validating clients you fully control (developer-written code can be held to a contract), but LLM output is uncontrollable and over/under-filling is the norm. Drop .strict() and use Zod's default strip (silently removes unknown keys) for better tolerance; if unknown keys carry semantics you want, use .passthrough() to keep them, or promote them to known fields.

Does Zod strip or throw on unknown fields by default?​

Default is strip — it silently removes unknown keys without error. .strict() makes it throw on unknown keys; .passthrough() keeps them as-is. For uncontrollable output like LLM generations, prefer default strip or passthrough over .strict(), which kills the whole payload over a single irrelevant field.

When an Airflow DAG reads a .sql template file as a single string and passes it to PostgresHook.get_pandas_df(), prior SELECT results are silently dropped — the DAG reports "SQL query returned no results", but copying the same SQL into psql returns data normally.

Encountered this while building AI Analytics — an LLM-powered analytics pipeline where an Airflow DAG reads multi-query report templates from .sql files and executes them.

TL;DR​

PostgresHook.get_pandas_df(sql) internally calls pandas.read_sql(sql, conn) → psycopg2 cursor.execute(sql). When sql is a single string with multiple ;-separated SELECTs, the DBAPI only exposes the cursor of the last result set — prior query results are silently dropped with no error. Fix: split by top-level semicolons into a list[str] and call get_pandas_df per statement, or pass the list directly so DbApiHook runs them sequentially.

Symptom​

The DAG task executing shop_monthly_overview.sql reports "SQL query returned no results":

sql_count = 1   ← template clearly contains 4 queries
result = "❌ SQL query returned no results"

But the same SQL pasted into psql against the same database with the same parameters returns data for all 4 SELECTs.

Reproduction​

Verify get_pandas_df behavior with multi-statement SQL directly inside the Airflow container:

from airflow.providers.postgres.hooks.postgres import PostgresHook

hook = PostgresHook(postgres_conn_id="postgres_default")

# Three SELECTs concatenated into one string
sql = "SELECT 1 AS a; SELECT 2 AS b; SELECT 99 AS c WHERE 1=0;"

df = hook.get_pandas_df(sql)
print(df.columns.tolist())  # ['c']  ← only got the last statement's columns
print(df)                    # Empty  ← the last statement itself returns 0 rows

Expected three result sets, got only the last one (SELECT 99 ... WHERE 1=0, 0 rows). The first two completely disappear with no error or warning.

Root Cause​

The call chain is PostgresHook.get_pandas_df → DbApiHook.get_pandas_df → pandas.io.sql.read_sql → psycopg2 cursor.execute(sql).

The DBAPI protocol (PEP 249) allows execute to accept a string with multiple ;-separated statements. PostgreSQL executes all of them, but the cursor only exposes the last result set — this is inherent PostgreSQL wire protocol behavior, not an Airflow or pandas bug.

┌────────────────────────────────────────────────────┐
│ SELECT 1;  ← executed, result set 1 dropped at once │
│ SELECT 2;  ← executed, result set 2 dropped at once │
│ SELECT 99 WHERE 1=0;  ← executed, result set 3 exposed │
└────────────────────────────────────────────────────┘
                       ↓
            pandas.read_sql only fetches result set 3

The root cause in our code is task_execute_sql reading the entire .sql file as a single string and passing it to get_pandas_df:

# ❌ Problematic code
sql_text = open(sql_path).read()  # full text with 4 SELECTs
df = pg_hook.get_pandas_df(sql_text)  # only gets the last result

Why does psql return data? Because the psql frontend actively iterates through all result sets and prints them one by one, while a DBAPI cursor does not.

Solution​

Option A (Recommended): Split by top-level semicolon, execute one by one​

Suitable for .sql template files — they contain comments, quotes, and multiple queries that need robust splitting.

def split_sql_statements(sql: str) -> list:
    """
    Split SQL by top-level semicolons, correctly handling:
    - Semicolons inside single-quoted strings ('a;b' is not split)
    - SQL-standard '' escape ('it''s' is not split)
    - Semicolons inside -- line comments (-- note; not split is not split)
    """
    statements = []
    buf = []
    i, n = 0, len(sql)
    in_quote = False

    while i < n:
        ch = sql[i]

        # Inside a single-quoted string
        if in_quote:
            buf.append(ch)
            if ch == "'":
                # '' = literal single quote, does not end the string
                if i + 1 < n and sql[i + 1] == "'":
                    buf.append(sql[i + 1])
                    i += 2
                    continue
                in_quote = False
            i += 1
            continue

        # Top level
        if ch == "'":
            in_quote = True
            buf.append(ch)
        elif ch == '-' and i + 1 < n and sql[i + 1] == '-':
            # Line comment, swallow to end of line (; inside is not a split point)
            while i < n and sql[i] != '\n':
                buf.append(sql[i])
                i += 1
            continue
        elif ch == ';':
            stmt = ''.join(buf).strip()
            if stmt:
                statements.append(stmt)
            buf = []
            i += 1
            continue
        else:
            buf.append(ch)
        i += 1

    # Trailing block without a final semicolon
    stmt = ''.join(buf).strip()
    if stmt:
        statements.append(stmt)

    return statements


# Caller
sql_text = open(sql_path).read()
statements = split_sql_statements(sql_text)

# Execute one by one, collect all results
all_results = []
for idx, stmt in enumerate(statements, start=1):
    df = pg_hook.get_pandas_df(stmt)
    if not df.empty:
        all_results.append({
            "sql_index": idx,
            "sql": stmt,
            "data": df.to_dict("records"),
            "columns": df.columns.tolist(),
            "row_count": len(df),
        })

Option B: Pass a list directly to DbApiHook​

Airflow's DbApiHook.run and get_records accept list[str] and execute sequentially — but get_pandas_df return behavior in list mode is inconsistent across providers. For production, Option A gives you full control.

Why not sqlparse.split?​

Community answers often recommend sqlparse.split(sqlparse.format(sql, strip_comments=True)), but strip_comments=True discards comments. If your downstream processor depends on metadata in comments (e.g. -- dimension: shop), you lose context. A hand-rolled splitter preserves the original comment text and gives you control.

Caveats​

FAQ​

How do I execute multiple SQL statements in Airflow PostgresHook?​

Pass list[str] instead of a single string. DbApiHook.get_pandas_df and run accept sql as a list and execute sequentially; a single string with semicolon-separated statements causes psycopg2 to return only the last result set. For production, split yourself and call per-statement so you control result aggregation and sql_index.

Why does get_pandas_df only return the last result for multi-statement SQL?​

pandas.io.sql.read_sql calls psycopg2 cursor.execute with the full string; the DBAPI protocol only exposes the cursor of the last result set for multi-statement execution, and prior SELECT results are dropped by the server immediately, with no error or warning. psql returns data because its frontend actively iterates all result sets, while a DBAPI cursor does not.

How do I split SQL by semicolon safely with comments and quotes?​

Scan character by character and split only at top-level semicolons outside single-quoted strings and -- line comments. Single-quote literals use the SQL-standard '' escape; do not use str.split(';'), it will mis-cut semicolons inside comments and strings. If you use sqlparse.split, note that strip_comments=True discards the original comment text.

While building a WooCommerce product import tool for a client, POST /wp-json/wp/v2/media would succeed for the first few images, then suddenly return 405 Not Allowed for all subsequent requests.

TL;DR​

Hostinger CDN (hcdn) blocks POST /wp-json/wp/v2/media requests by default. The response headers server: hcdn and x-hcdn-request-id are the smoking gun. Disable CDN or contact Hostinger support to whitelist /wp-json/* POST requests.

The Problem​

Uploading images to WordPress Media Library via REST API:

curl -X POST 'https://example.com/wp-json/wp/v2/media' \
  -u 'user:app_password' \
  -H 'Content-Disposition: attachment; filename="product-01.jpg"' \
  -H 'Content-Type: image/jpeg' \
  --data-binary @image.jpg

The first 2-4 images return 201 Created, then all subsequent requests fail with:

<html>
<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>

This "partial success" pattern is misleading — it looks like rate limiting, but the real cause is entirely different.

Root Cause​

Using curl -v to inspect the full response headers revealed:

< HTTP/2 405
< server: hcdn
< x-hcdn-request-id: cfc5ad1198938cd9f1e02ce71ed0ae61-kul-edge1

Key findings:

• server: hcdn — This is Hostinger's custom CDN (hcdn), not the origin nginx server
• x-hcdn-request-id — CDN edge node ID (kul-edge1 = Kuala Lumpur), confirming the request was blocked at the CDN layer before reaching WordPress

Hostinger CDN's default security rules block POST requests to /wp-json/wp/v2/media. The initial successes were likely due to CDN rule cold-start or cache misses.

Solution​

Option 1: Disable CDN (Quick Fix)​

Go to Hostinger hPanel → Website → CDN → Disable.

This takes effect immediately but removes CDN acceleration. Suitable for staging environments or emergency fixes.

Option 2: Contact Hostinger Support to Whitelist API Paths (Recommended)​

Submit a support ticket requesting to whitelist POST requests to /wp-json/*. Hostinger's Manage panel currently doesn't offer custom CDN rule configuration — you must go through support.

Option 3: Add Retry Logic in Code (Defensive Measure)​

Even with correct CDN configuration, retry logic handles occasional CDN throttling:

import time
import random

def upload_image(url, image_bytes, filename, auth, max_retries=3):
    for attempt in range(max_retries):
        resp = httpx.post(
            url,
            content=image_bytes,
            headers={
                "Content-Disposition": f'attachment; filename="{filename}"',
                "Content-Type": "image/jpeg",
            },
            auth=auth,
            timeout=30,
        )
        if resp.status_code != 405:
            return resp
        delay = 3 * (attempt + 1) + random.uniform(0, 2)
        time.sleep(delay)
    resp.raise_for_status()

Troubleshooting Journey​

This issue led down several dead ends. Here's the full排查 path for reference:

The turning point was using curl -v and spotting server: hcdn — only then did we realize the requests never reached WordPress.

FAQ​

Why does WordPress REST API image upload return 405 Not Allowed?​

Check the server field in response headers. If it shows hcdn (Hostinger CDN) or another CDN identifier, the request is being blocked at the CDN layer before reaching WordPress. Disable the CDN or contact your hosting provider to whitelist the endpoint.

How to tell if 405 comes from CDN or WordPress?​

Use curl -v and inspect response headers: a server value of hcdn, cloudflare, or other CDN identifiers indicates CDN-level blocking; a server value of nginx/apache with X-WP-* or X-RateLimit-* headers means the request reached WordPress.

Encountered this issue while building a WooCommerce product import tool for a client. If you're also developing with Hostinger + WordPress and running into REST API issues, reach out.

Encountered these five issues repeatedly while developing WordPress Block Themes for clients. Each one took significant debugging time. This guide covers the root causes and provides ready-to-use solutions.

TL;DR​

Five issues ranked by frequency: file changes not applying (database cache overrides files), block nesting errors (unclosed comments), child theme content not rendering (missing post-content block), SVG icons disappearing (WP_Filesystem polluted by plugins), and WP-CLI mail failures (SMTP plugins don't hook in CLI). Each scenario includes copy-paste diagnostic commands.

TL;DR​

Embedding a self-hosted timeline into a SITE123 event page hits 5 platform walls: Custom Code can't target a page or position, scripts run before DOM is ready, selectors hit hidden elements, float layout pushes adjacent blocks out of place, and platform cache injects the script twice. Fix: JS DOM manipulation + horizontal fishbone layout + DOMContentLoaded wrapper.

Encountered this issue while building an AI Agent chat interface. Here's the root cause and solution.

TL;DR​

Date.now() millisecond timestamps can duplicate within the same millisecond. When used as React list keys, this causes DOM errors. Fix by adding a random suffix or using crypto.randomUUID().

TL;DR​

Frontend code pushed to Git but new feature missing on production? The root cause is usually outdated build artifacts on the server. Compare local and server dist/ directory timestamps to confirm, then run npm run build on the server.

Encountered this issue while upgrading Node.js version for an enterprise SaaS system. Documenting the root cause and solution.

TL;DR​

Node.js v24 changed Web Crypto API implementation. The jose library now requires keys to be KeyObject or CryptoKey type. Wrap your key with crypto.createSecretKey() to fix.