Push code, the GitHub Actions deploy step turns red and exits — yet the service on the server is already the latest version. Another day, the reverse: everything is green locally while pnpm install --frozen-lockfile fails on CI every single time. These two opposite signal failures both live in the deploy pipeline, not in the code.

I hit this while building an e-commerce automated data collection tool for a client — bulk-scraping product images, SKUs, prices, and reviews, cleaned and exported as structured data for inventory management and competitor analysis. The tool's server deploys through PM2 + GitHub Actions, and both false failures happened on that pipeline.

TL;DR​

• CI red but the deploy actually landed: two deploy channels ran pm2 restart concurrently; one looked up the process mid-restart and misreported failure. Signature: pm2.log shows a successful restart and a process already online error within the same second. Fix: collapse deploys to a single channel.
• Local green but CI always fails: pnpm add ran in a subdirectory, updating only the child package.json while the root pnpm-lock.yaml went stale. Fix: run pnpm install at the workspace root and commit the regenerated lockfile.

Scenario 1: green locally, CI always red — pnpm lockfile drift​

This failure has nothing to do with code quality — pnpm-lock.yaml simply drifted out of sync with package.json, and CI is the only environment that checks the two strictly.

The symptom​

CI fails on every run at the same step:

ERR_PNPM_OUTDATED_LOCKFILE

Locally, install, build, and tests all pass. That "works on my machine but CI fails" combination tempts you to suspect CI caching or the Node version first — both wrong here.

Root cause: pnpm add ran in the wrong directory​

This project is a pnpm workspace monorepo with a single lockfile at the root. The dependency was added like this:

# executed inside client/
pnpm --filter @ccl-ext/client add <pkg>

client/package.json got updated, but the root pnpm-lock.yaml was never regenerated and committed. CI then received a lockfile inconsistent with package.json, and the --frozen-lockfile check refused to install.

Why local never catches it: node_modules already has the packages physically installed, so a local install reuses what is there and never exercises the frozen check. CI starts from a clean environment and compares the lockfile strictly, every time.

There is also a telltale side effect: running pnpm in the wrong cwd leaves a stray client/pnpm-lock.yaml behind. Spotting that file is near-proof that a command ran in the wrong place again.

The fix​

Two steps:

1. From the workspace root, run pnpm install to regenerate the root lockfile, and commit it with the code;
2. From now on, run pnpm add / pnpm remove at the workspace root only.

The specifiers diff in the CI failure details names exactly which package's dependency range changed — use it to confirm the fix targets the right spot.

A similar "wrong attribution" issue in multi-package workspaces is covered here: npm audit blames the wrong directory? Multi-package deploys audit N package trees.

Scenario 2: Actions reports failure, the deploy actually landed — pm2 restart race​

The deploy itself did not fail; the second, colliding restart did — PM2 reported that race as a deploy error.

The symptom​

After a push, GitHub Actions "Deploy Server" fails at the pm2 restart step:

[PM2][ERROR] Process 3 not found → exit 1

But on the server: the code is the latest, pm2 list shows the process online, and the health check returns 200. All three deploy essentials pass — only Actions believes it failed.

Root cause: two channels restarting the same process​

Deploys had two trigger paths at the time:

1. The push trigger in deploy.yml;
2. A local /deploy script (deploy.sh).

One push made both paths run pm2 restart ccl-ext-api. When the restarts collided, one of them queried the process at the exact instant the other was replacing it, found nothing, logged Process not found, and exited 1 — PM2 reported the race as a failure.

Verification: the same second in pm2.log​

pm2.log is the hardest evidence. The log shows two kinds of records within the same second:

# one side: the restart completes, process online
Stopping → starting → online

# the other: the colliding restart finds no process
PM2 error: process already online

One restart "in flight" interleaving with another "querying" inside 1 second is the race signature. Combined with the three checks — code latest, process online, health 200 — the deploy landed and the Actions failure was noise.

The fix: drop the push trigger, single deploy channel​

The change removes the push auto-trigger from deploy.yml, keeping workflow_dispatch for manual fallback:

on:
  workflow_dispatch:

Between the two options — a concurrency lock versus collapsing to one channel — I chose the latter: a lock only makes the two channels queue up, leaving two deploy paths in place. A deploy should have exactly one entry point; who deployed what, when, should originate from a single place. After the change, the false alarms never returned.

For a different take on verifying what is actually live after a deploy, see: Frontend deployed but the site did not update? Troubleshooting stale builds.

Side by side: both false failures share one trait — a polluted signal source​

Put the two scenarios together and one trait surfaces immediately: what broke was never the deploy result, but the pipeline producing the signal.

CI's red and green are just the pipeline's output. When the pipeline itself is polluted (two lockfiles, two channels), the signal stops being trustworthy. Fix the pipeline first; read the signal after.

FAQ​

Why does pm2 report 'Process N not found' during a GitHub Actions deploy?​

Two deploy channels ran pm2 restart on the same process at the same time. One restart hits the instant the process is being replaced, finds nothing, and exits 1 — while pm2.log shows a successful restart and a 'process already online' error within the same second.

GitHub Actions shows the deploy as failed — how do I tell if it is a false alarm?​

Check 3 things: the code on the server is the latest, pm2 list shows the process online, and the health check returns 200. If all three pass and the failure point is the pm2 restart step, it is a concurrency false alarm; collapsing deploys to a single channel removes it.

What causes a pnpm frozen-lockfile error in CI when local installs pass?​

The root cause is pnpm-lock.yaml out of sync with package.json: running pnpm add in a subdirectory updates the child package declaration but leaves the root lockfile stale. Run pnpm install once at the workspace root, commit the regenerated lockfile, and check the specifiers diff in the CI log.

Clicking the "Extract this page" button in the browser extension throws Cannot access 'Hc' before initialization — while typecheck, build, and the regular test suite all pass, and the error never reproduces in local development.

I hit this while building an e-commerce automated data collection tool for a client — bulk-scraping product images, SKUs, prices, and reviews, cleaned and exported as structured data for inventory management and competitor analysis. "Extract this page" is the collection entry of that toolchain, and the content script module was what broke.

TL;DR​

A third-party classic script (vendored) contains an implicit global assignment (CandidateElement = function(...), assigned without declaration). Legal in a classic script — it silently creates a global — the line becomes a ReferenceError once the file is inlined into an ESM module graph that runs in strict mode. Module initialization aborts on the spot, and the namespace the downstream code receives through dynamic import sits in the temporal dead zone (TDZ). Fix: add one line var CandidateElement; at the top of the vendor file.

The symptom: green build, instant crash on click​

The error fires when the user clicks, not at page load:

TypeError: Cannot access 'Hc' before initialization

Three counterintuitive facts:

1. typecheck passes — nothing wrong at the type level;
2. build passes — the bundler does static analysis only, it never executes module code;
3. regular tests pass — no test imports the module graph completely.

The failing identifier is Hc, a 2-letter minified name, not any business symbol. Hold that thought; it matters for identification later.

Root cause: how an implicit global breaks the ESM module graph​

The offending line comes from a third-party library, at reader-finder.js:878:

// classic script semantics: assign without declaring = create a global, legal
CandidateElement = function(e, t) { ... }

The failure chain has 4 steps, each enabling the next:

Step 1: legal under classic script semantics. The file originally loaded via a <script> tag; in sloppy mode, assigning without declaring silently creates a global variable — the original author relied on exactly that.

Step 2: a minefield once inlined into ESM. The file got inlined into the extension's ESM module graph, and ESM code always runs in strict mode — the implicit global assignment now throws a ReferenceError, and module evaluation of that vendor module aborts immediately.

Step 3: the break spreads along the module graph. The content script content.js evaluates its inline module graph and stalls at the vendor module: earlier modules got their message listeners registered, but later module facades never executed — the graph is left half-initialized.

Step 4: the dynamic import lands in the TDZ. When the user clicks the button, code loads the namespace facade through dynamic import. Because of the step-3 break, that namespace is in the temporal dead zone, and touching it throws Cannot access 'Hc' before initialization — Hc being the renamed internal binding of the module that never finished initializing.

This explains every observation: static checks stay green because they never execute module code; the error appears on click because the dynamic import lives inside the click handler; and the identifier is minified because the broken binding was renamed by the bundler.

For the other high-frequency dynamic import pitfall (module not found), see: Node.js ESM dynamic import says module not found? Check the file extension.

The fix: one var declaration at the vendor file head​

No third-party logic changes — just turn the implicit global into an explicit declaration by adding at the top of the vendor file:

var ReaderArticleFinder;
var CandidateElement;

The assignment changes from "create a global" to "assign to a declared variable", which is legal in strict mode. Module initialization completes, and the facade downstream code imports dynamically works as expected.

ReaderArticleFinder in the same file was already handled this way — the same library planted the same trap twice; the first got fixed, the second (CandidateElement) slipped through.

Verification: reproduce it locally with Vitest​

Before fixing, make it reproduce on demand — otherwise every check means a production deploy. Regular tests never reach this path, but importing the module directly in Vitest (jsdom environment) does:

import { describe, it, expect } from 'vitest';

describe('vendor reader-finder strict-mode', () => {
  it('initializes the full module graph without a ReferenceError', async () => {
    const mod = await import('./lib/vendor/reader-finder');
    expect(mod).toBeDefined();
  });
});

This test reproduced the error before the fix and produced a stack with real file line numbers (reader-finder.js:878) — far more actionable than a minified Hc from production. It turned green after the fix.

The complete verification runs the extraction chain end to end: the module graph initializes fully and extraction actually works — both must pass to call it closed.

Regression guard​

The reproduction case became a permanent smoke test (extractor.test.ts), plus a rule for the repo: any new classic script vendored into the project must pass this test, or an equivalent strict-mode check, before landing.

Quick identification for this kind of TDZ error​

Next time Cannot access 'xxx' before initialization shows up, two traits tell you whether it is the same species:

Left column on both rows: suspect an implicit global in a vendored classic script — search for assign-without-declare and reproduce with a direct Vitest import. The other classic ESM migration error (CJS require ESM) is covered here: Node.js require nanoid throws ERR_REQUIRE_ESM? Alternatives after v5 went ESM-only.

FAQ​

Why does 'Cannot access before initialization' only appear at runtime when typecheck and build are green?​

Static checks and bundlers never execute module code, while the ReferenceError from an implicit global assignment fires only when the module actually initializes. If the offending module sits on a dynamic import chain, the error defers to the moment of interaction — in this case, clicking the extension button, with 0 errors at build time. Importing the module directly in Vitest (jsdom) reproduces it locally with real line numbers.

What does 'Cannot access xxx before initialization' have to do with the temporal dead zone (TDZ)?​

The namespace object returned by a dynamic import stays in the temporal dead zone once its dependency module's initialization broke, so touching any export throws. The giveaway is the identifier name: a 2-letter minified name like Hc instead of a business symbol means the break happened during module graph evaluation, not in your code.

How do I fix implicit globals in a vendored classic script?​

Declare each implicit global at the top of the vendor file with var (2 in this case: ReaderArticleFinder and CandidateElement) so the assignment targets a declared binding. One line removes the whole module-graph initialization break; run a strict-mode smoke test before any new vendor file lands.

cPanel 138 on AlmaLinux 10 installs smoothly and the panel opens — the real traps cluster in the close-out phase: security hardening, panel asset repairs, and domain mounting, each with counter-intuitive behavior waiting.

Encountered this while hardening a production cPanel server for a client — all three failure classes showed up during the close-out of the same AlmaLinux 10 machine. This post breaks them down by scenario.

TL;DR​

Three scenarios, three "looks right, does nothing" traps: TFA configured but login never asks for a code — the policy-wide switch is off, so per-user configuration is inert; frontend assets that look "missing" must be verified against the official manifest first — some never existed on that cpanelsync tree, and some are skipped because a polluted digest cache lets upcp fake success; Park domains rejected by the NS ownership check — when DNS lives on a cloud provider, the userdata include injecting a ServerAlias is the supported path.

Scenario one: "missing" WHM frontend libraries — verify before fixing​

While triaging panel oddities, /usr/local/cpanel/base/libraries turns out not to exist — do not rush to repair a defect. Upstream cPanel 138 simply has no such path. The real homes of the shared frontend libraries:

• base/frontend/jupiter/libraries/ — a symlink farm pointing at ../../../../3rdparty/share/<lib>, distributed by jupiter's own cpanelsync tree (sortablejs, ui-fonts, fontawesome, cldr)
• base/unprotected/libraries/ — same mechanism, hosting legacy libraries

To decide whether a file is genuinely missing, pull the official manifest instead of trusting one path:

curl -sO http://httpupdate.cpanel.net/cpanelsync/138/<tree>/.cpanelsync.bz2
bzcat .cpanelsync.bz2 | grep <target-path>
# Entry format: d===./path===755          (directory)
#               l===./link-name===777===target (symlink)

Second rule: base/ is not entirely cpanelsync-distributed. The v138 cpanel-* RPMs (bootstrap5, ace-editor, sortablejs and friends) install libraries directly into /usr/local/cpanel/3rdparty/share/<lib>/<version>, and cpanelsync only lays the symlinks into the theme trees. Verify the RPM side with rpm -V <package> and the cpanelsync side with the manifest — both.

If files are confirmed missing but upcp --sync reports success without restoring them, suspect the digest caches: /usr/local/cpanel/.cpanelsync.digest and the per-theme-tree digests. When an interrupted update pollutes them, --sync skips the missing files and still exits 0. Delete the affected digest and run upcp --force for a full reconciliation.

A successful --sync is not proof of complete files — judge by actual page loading: drive a headless Chromium to collect console errors and requests at 4xx or above; that is more honest than any exit code.

Two more operational traps on hardened machines:

• For root access, prefer the cloud assistant (aliyun ecs RunCommand) — out-of-band, SSH-free, audited by default. Pass the instance via --InstanceId.1 and feed CommandContent the raw script, not base64. Any temporary sudoers grant needs a self-cleaning /etc/cron.d entry; after cleanup, sudo -n whoami must answer a password is required to confirm the revoke took.
• Bulk-probing cpsrvd triggers rate limiting: a shell loop of individual curl calls degrades to all-000 responses after a few dozen requests, which reads like a mass 404. A single curl process fetching multiple URLs over keepalive behaves normally. And pkill -f matches your own bash -c command line — use the [] character-class trick or a plain PID.

A 200 from a WHM page does not mean an authenticated session — the login page returns 200 too. Assert on the <title> or a body fingerprint (the Two-Factor Authentication page's title, for instance).

Scenario two: the hardening chain of traps​

The goal: no root over SSH, TFA on the panel, a minimal sudo whitelist. Every step hides a precondition.

The TFA policy switch is a precondition. After twofactorauth_set_tfa_config writes a user's secret, the login form may never show the code step — twofactorauth_policy_status must report is_enabled = 1 (enabled via twofactorauth_enable_policy), otherwise WHM validates password only. Verified in a browser: before the policy, the password logs straight in; after, an "Enter the security code" page appears.

Two CLI details for TFA: the token parameter of twofactorauth_set_tfa_config is tfa_token, not code (passing code silently fails with "security code is invalid"); and the TOTP must be computed against the server clock — a 23-second skew crosses the window and the locally computed code is always rejected. The secret lands in /var/cpanel/authn/twofactor_auth/tfa_userdata.json.

The supported API path without root SSH is session plus cpsess prefix. After create_user_session and a curl cookie-jar login, API calls must carry the cpsess path: https://host:2087/cpsessNNN/json-api/<function>; hitting /json-api/ directly answers "Token denied". The service parameter of create_user_session is whostmgrd (with the d) — whostmgr and cpanel are rejected; only cpaneld, webmaild, whostmgrd are valid.

sudoers matches the entire command sequence exactly. Even the position of --output=json and the argument order are locked; any edit to the caller's command silently degrades to password authentication — which, with the opsuser password locked, is an outright refusal. Changing the command means changing the matching sudoers file.

Three whmapi1 details: the real path is /usr/local/cpanel/bin/whmapi1 (prefer it over the symlink); the default output is YAML, so --output=json before jq; and sethostname takes hostname, not domain — domain= silently passes an empty value and runs to no effect, and cPanel refuses whm./cpanel./webmail.-prefixed hostnames. On machines with the DNS role disabled, the trailing dnsadmin socket "Connection refused" is expected and harmless; after a rename, create_user_session URLs follow the new hostname automatically, and AutoSSL reissues the cpsrvd certificate within about a minute.

sshd drop-ins: first value wins. The Include sshd_config.d/*.conf in AlmaLinux's main sshd_config sits at the top, so drop-ins parse before the main body and sshd keeps the first occurrence — that is how a drop-in overrides the main file; among drop-ins, filename sort order decides (000- sorts before 00-). After any change: sshd -t, then sshd -T | grep -E 'permitrootlogin|passwordauthentication|allowusers' to confirm the effective values before reloading.

Host Access Control does nothing here. cPanel 138 + AlmaLinux 10 ships a cpsrvd that does not link libwrap (tcp_wrappers is gone from RHEL-line distributions); rules written to /etc/hosts.allow and a cpsrvd restart changed nothing in testing. Layer-3/4 allowlists belong in firewalld rich rules; leaving hosts.allow in place is harmless — it activates automatically if libwrap ever returns.

Three AlmaLinux 10 verification blind spots: last is always empty — systemd 256 dropped wtmp, login records live only in the journal (journalctl -u sshd as root); opsuser cannot execute /usr/bin/su (denied at the exec layer), so the root password can only be verified through a WHM form or the console; and /etc/ssh/sshd_config.d/, /etc/cron.d/* (mode 600), /var/cpanel/authn/ are unreadable to opsuser — hardening audits must run in WHM Terminal or VNC.

Scenario three: domain NS lives elsewhere, alias mounting rejected​

uapi Park park domain=test.xxx is refused: the domain's nameservers (hosted on a cloud DNS) are "not associated with this server" — cPanel validates that the domain's authoritative NS points at the machine, and in common China-hosting setups DNS lives on the provider, so this check can never pass.

The supported path, without touching the domain's NS, is a userdata include injecting a ServerAlias:

# one for http, one for https
/etc/apache2/conf.d/userdata/std/2_4/<user>/<domain>/alias.conf
/etc/apache2/conf.d/userdata/sssl/2_4/<user>/<domain>/alias.conf

(The ssl-side directory name varies by version — sssl/2_4 here, sometimes written ssl/2_4; trust the uncommented include line in httpd.conf.) Contents, one line:

ServerAlias test.xxx

Then /scripts/rebuildhttpdconf. A wrong directory name leaves the include line commented out — silently inert. Verify by inspecting whether the Include "...userdata..." lines in httpd.conf carry a comment prefix.

Confirm routing with httpd -S and by watching which vhost's domlog receives the requests.

One linkage note: domains attached via ServerAlias are not managed by AutoSSL, so no certificate is issued for them automatically — see cPanel AutoSSL Not Issuing? The Exclusion List and vhost Paths.

Frequently Asked Questions​

Why is ssh PermitRootLogin no not working?​

Because sshd takes the first value it parses. On AlmaLinux the Include directive sits at the top of the main sshd_config, so drop-in files under sshd_config.d/ are parsed before the main body — a drop-in overrides the main file, and among drop-ins the lexicographically first filename wins. Always verify with sshd -t followed by sshd -T | grep permitrootlogin to see the effective value before reloading.

How does WHM two-factor authentication work?​

Two layers must both be active: the server-wide policy switch (twofactorauth_enable_policy) and the per-user TOTP secret (twofactorauth_set_tfa_config). With the policy off, WHM keeps validating password only even when a secret exists. When configuring via CLI, the token parameter is tfa_token — not code — and the TOTP must be computed against the server clock; a skew of just 20-odd seconds crosses the window and gets rejected.

Why is my Apache ServerAlias not working on cPanel?​

The userdata include is most likely inert: a wrong directory name leaves the Include line commented out in httpd.conf, and the alias never loads. ServerAlias files belong in /etc/apache2/conf.d/userdata/std/2_4/<user>/<domain>/ and the ssl counterpart, followed by /scripts/rebuildhttpdconf. Verify by checking whether the Include userdata lines in httpd.conf carry a comment prefix, then confirm with httpd -S and the vhost domlogs.

Restoring a cPanel account from a cpmove archive on an Aliyun ECS instance: restorepkg reports success, httpd -S shows the namevhosts — and the browser greets every domain with the server's default page.

Encountered this while migrating a client's two China sites into a cPanel environment on Aliyun China — the very first public verification after the restore ran into this cloud architecture trap.

TL;DR​

On Aliyun ECS the NIC only carries a private IP; the public IP is edge NAT and never lands on the machine. Vhosts migrated from the old server bind that non-existent public address, never match at runtime, and all traffic drops into the wildcard default vhost. Fix all three IP stores (/etc/wwwacct.conf, /var/cpanel/users/, /var/cpanel/userdata/) and rebuild httpd.conf; write the private IP into /etc/mainip so new accounts start correct.

Symptoms​

Every migration step looks clean: restorepkg finishes without errors and the namevhosts are right there in the Apache config. But public verification falls flat — every domain lands on the same defaultwebpage redirect, and the site's domlog stays empty forever.

The clincher is a single static-file request: fetch a file that certainly exists and was never touched (say /some-real-page.html) and it returns 404 — with the error page styled from /var/www/html/*.shtml, the cPanel default site's directory. The real document root never received the request.

$ httpd -S | grep example.cn
203.0.113.10:80                   example.cn ...

The vhost binds 203.0.113.10 — the old server's public IP. This new machine's NIC has no such address.

Root causes​

The public IP is not on the machine. hostname -I returns only a private address like 172.28.100.10. Aliyun ECS public IPs are edge NAT: traffic arrives at Alibaba's gateway and is forwarded to the instance's private address — the public IP never appears on the NIC.

The restore inherited the old IP verbatim. The old server was a classic VPS with the public IP on the NIC, so its vhosts record exactly that address. cpmove brings the configuration over as-is, and every vhost Address now points at an address that does not exist locally.

Apache matches vhosts by IP:port. Requests arrive, match no namevhost, and fall into the *:80 default vhost (DocumentRoot /var/www/html). That single mechanism explains all three symptoms: the default page, the empty domlog, and the mismatched 404 styling.

Fix​

1. Confirm the NAT setup. One command:

hostname -I
# 172.28.100.10   ← private segment only: NAT confirmed

2. Rewrite all three IP stores — all of them, no exceptions. cPanel keeps IP information in three places serving different flows. Fixing only userdata rebuilds the vhosts correctly, but a stale wwwacct.conf writes the wrong address into every future account.

# 2a. Global default (read by account creation)
sed -i 's/^ADDR=.*/ADDR=172.28.100.10/' /etc/wwwacct.conf

# Default IP for new accounts — set this too
echo 172.28.100.10 > /etc/mainip

# 2b. Per-account record
sed -i 's/^IP=.*/IP=172.28.100.10/' /var/cpanel/users/example

# 2c. Bulk rewrite of userdata — the only input rebuildhttpdconf trusts
cd /var/cpanel/userdata/example
for f in *; do
  [ -f "$f" ] && sed -i 's/^ip: .*/ip: 172.28.100.10/' "$f"
done

The [ -f "$f" ] guard is not decorative: the userdata directory mixes in socket files like scope, and sed errors out on them without the check.

3. Rebuild and restart.

/scripts/rebuildhttpdconf
/scripts/restartsrv_httpd

4. Verify. httpd -S should now show namevhosts on the private IP; from the server, a request with an explicit Host header tells you which vhost answers:

curl -s -H "Host: example.cn" http://172.28.100.10/some-real-page.html -o /dev/null -w "%{http_code}\n"
# 200 ← no longer the default vhost's 404

Finish with an external visit to confirm the pages load and the domlog starts filling.

Edge cases and variants​

• Fresh installs: write /etc/wwwacct.conf ADDR and /etc/mainip before creating any account and the wrong-address problem never happens; this applies beyond restores.
• Not Aliyun-specific: AWS Elastic IPs and similar NAT-style public IPs behave the same. Anywhere the public IP is absent from hostname -I, cPanel's IP settings must use the private address.
• Local verification blind spot: testing from the server against 127.0.0.1 or the public address hits the default vhost and misleads — use the private IP with an explicit Host header.

Frequently Asked Questions​

Why does my cPanel site show the default page after a migration?​

Because the vhost address never matches. On Aliyun ECS the NIC only carries a private IP — the public IP lives on an edge NAT gateway. A restore inherits the old server's public IP, Apache matches vhosts by IP:port, no vhost ever matches, and everything falls into the wildcard default vhost. Rewriting three IP stores to the private IP and rebuilding httpd.conf fixes it.

How do I check which Apache vhost is serving my requests?​

Two checks give a verdict: the site's domlog stays completely empty, and a request for a static file that certainly exists returns 404 with an error page styled from the default document root (/var/www/html). Both together mean the default vhost is answering. Then compare httpd -S against hostname -I to see whether the bound address actually exists on the NIC.

Which cPanel IP settings need updating on a NAT cloud server?​

Three places together: ADDR in /etc/wwwacct.conf (global default), IP= in /var/cpanel/users/ account files, and every ip: field under /var/cpanel/userdata/ — then run /scripts/rebuildhttpdconf and restart httpd. For new accounts the default comes from /etc/mainip; writing the private IP there before migration prevents the problem from recurring.

You open your own website hosted on a cPanel server, and the browser warns "Your connection is not private" — the certificate details show an issuer that is a temporary hostname generated during the cPanel installation, with no relation to your domain.

Encountered this while migrating a client's two China sites into their fully managed hosting environment — long-expired certificates and browser security warnings were among the legacy issues we had to clear before sign-off.

TL;DR​

When the browser reports an untrusted self-signed certificate, the CA is usually fine — the real Let's Encrypt certificate was never issued. Check two things in order: first the AutoSSL per-account domain exclusion list (whmapi1 get_autossl_user_excluded_domains); once cleared, trigger issuance immediately with start_autossl_check_for_one_user. If issuance succeeds but the public still sees a self-signed certificate, look for a custom vhost pinning SSLCertificateFile. And note that already optimal does not mean full coverage — trust only the SAN of the certificate the public actually receives.

Symptoms​

Run a certificate check against the domain from an external machine:

$ openssl s_client -connect example.cn:443 -servername example.cn </dev/null 2>/dev/null \
    | openssl x509 -noout -subject -issuer -dates
subject=C = US, O = cPanel, L = Houston, ST = TX, OU = SSL Support, CN = 203-0-113-10.cprapid.com
issuer=C = US, O = cPanel, L = Houston, ST = TX, OU = SSL Support, CN = 203-0-113-10.cprapid.com
notBefore=May 21 00:00:00 2026 GMT
notAfter=Aug 19 00:00:00 2026 GMT

Three signals stack up: issuer identical to subject (self-signed), a CN that is the cprapid temporary hostname cPanel generated at install time (unrelated to the site), and validity dates matching the cPanel service certificate. In other words, visitors receive cPanel's self-signed service certificate — the Let's Encrypt certificate never reached the public.

The confusing part: inside WHM, AutoSSL looks perfectly healthy. Provider is Let's Encrypt, the daily task runs on schedule, and no errors appear. The AutoSSL log reveals the real reason:

User-excluded domains: 9 (mail.example.cn, webmail.example.cn, ...)

All 9 domains of the account are on the exclusion list — AutoSSL treats that as user intent, skips them every run, and wraps up with "already optimal".

Root causes​

Cause one: the domain exclusion list blocks issuance (primary). AutoSSL maintains a per-account exclusion list; domains on it never enter issuance. It neither errors nor warns — the daily task simply runs to completion, which is why the panel looks normal. The list gets populated in three common ways:

• Manual exclusions early on: the domain had no DNS or was still in testing, and the exclusion was never cleaned up
• cpmove migrations carry it over: restoring an archive brings the old server's exclusion state and old certificates to the new machine
• Newly created subdomains: cPanel adds a subdomain (together with its www.* form) to the list by default

Cause two: a custom vhost pins the certificate path. After clearing the list, the log confirmed a successful issuance — yet the public still received the self-signed certificate. Issued, but not served. This server carried a custom Apache include (a mirror vhost added for public IP routing) whose SSLCertificateFile was hardcoded to /var/cpanel/ssl/cpanel/cpanel.pem, cPanel's self-signed service certificate. Standard vhosts in cPanel's httpd.conf bind only the main IP, so all public traffic hit the mirror vhost and never saw the AutoSSL result.

The two causes stack: clearing only the list issues a certificate nobody sees; fixing only the vhost points the mirror at a certificate that was never issued. Fix one, then two — both are required.

Fix​

1. Locate from outside. Run the certificate check from an external machine (not on the server itself — see the warnings at the end). Once issuer equals subject is confirmed, query the exclusion list on the WHM server:

whmapi1 get_autossl_user_excluded_domains username=example

2. Clear the exclusion list. The domain parameter is repeatable, so all domains that need certificates can be allowed in one call:

whmapi1 remove_autossl_user_excluded_domains \
  username=example domain=example.cn domain=www.example.cn

Service subdomains such as mail or webmail that have no DNS record or do not need certificates are reasonably kept excluded — removing them only fills the log with DCV errors without producing any certificate.

3. Trigger issuance immediately. The daily task waits for the scheduler; a manual run executes now:

whmapi1 start_autossl_check_for_one_user username=example

Two naming details: there is no shorter variant without _for_one_user, and the parameter is username, not user. When unsure about function names, grep the module source:

grep -i autossl /usr/local/cpanel/Whostmgr/API/1/SSL.pm

The CLI equivalent is /usr/local/cpanel/bin/autossl_check --user=example. Logs land in /var/cpanel/logs/autossl/, one directory per timestamp; they contain binary characters, so filter with grep -a or strings before reading.

4. If issuance succeeded but the public still sees the old certificate, inspect custom vhosts. ACME requests succeeded and the certificate is on disk, yet visitors get the old one — traffic is not flowing through the standard vhost. Search the custom include for a pinned path:

grep -rn "SSLCertificateFile" /etc/apache2/conf.d/includes/post_virtualhost_global.conf

Replace the hardcoded cpanel.pem with the per-domain certificate path:

SSLCertificateFile /var/cpanel/ssl/apache_tls/example.cn/combined

Then restart with /scripts/restartsrv_httpd. This include file is custom configuration — cPanel does not overwrite it when rebuilding httpd.conf, so future renewals take effect automatically. One edit is enough.

5. Final verification. Re-run the step 1 command from outside: the issuer should now be a Let's Encrypt intermediate (R3/R10/R11 depending on LE rotation) and the SAN should include the site domain. Let's Encrypt certificates last 90 days; AutoSSL's daily task renews them from here on without further action.

Migration and subdomain variants​

• After a cpmove migration: the exclusion list and old certificate state arrive intact, and the old LE certificate's SAN usually covers only the apex and www. After a restore, clean the exclusion list and trigger issuance once — do not wait for the daily task.
• Newly created subdomains: cPanel auto-excludes them together with www.test.; remove the exclusion after creation. A www.test. without DNS resolution is better left excluded to avoid recurring DCV errors.
• Domains attached via ServerAlias: aliases injected through userdata includes are not managed by AutoSSL, and hand-editing parked_domains in userdata plus updateuserdomains gets silently dropped. The supported path is the official API:

uapi --user=example SubDomain addsubdomain domain=test rootdomain=example.cn dir=/home/example/public_html

Note the parameter is rootdomain — passing parentdomain is silently ignored and the API replies "You must specify a main domain".

Frequently Asked Questions​

Why is cPanel AutoSSL not issuing certificates?​

Check the certificate itself: if the issuer equals the subject and the CN is a cPanel temporary hostname, no real certificate was ever issued. The two most common causes are the per-account AutoSSL domain exclusion list, and a custom vhost pinning SSLCertificateFile to the cPanel self-signed service certificate. Let's Encrypt certificates are valid for 90 days; once issuance succeeds, AutoSSL's daily task renews automatically.

What are cPanel AutoSSL user excluded domains?​

A per-account list of domains that AutoSSL deliberately skips. Beyond manual exclusions, a cpmove restore carries the old server's list over, and newly created subdomains are auto-excluded (including www.test.*) — in our case all 9 domains of one account were on the list. Remove them with whmapi1 remove_autossl_user_excluded_domains, then trigger start_autossl_check_for_one_user instead of waiting for the daily run.

Why is cPanel AutoSSL not renewing certificates?​

"already optimal" in the AutoSSL panel only means no renewal is due — it does not confirm coverage. Domains on the exclusion list are skipped silently, so nothing is ever issued or renewed, and Let's Encrypt certificates expire after 90 days. Verify the live SAN with openssl s_client from an external machine, clean the exclusion list, and trigger issuance manually; the daily task takes over renewal afterwards.

You provision a fresh server with cPanel, the installer reports "complete" — and then account creation, database setup, or the sites themselves fail one after another. These failures share a trait: the place that errors is not the place that broke.

Encountered this while building a fully managed hosting environment on Aliyun China for a client — all three failure classes showed up on the same machine during provisioning, and each had to be cleared before the site migration could start.

TL;DR​

Fresh cPanel installs fail in three recurring ways, all wearing a mask of success: the installer reports complete while MariaDB never installed (a failed stage does not roll back); account creation is blocked by a chain of missing keys when /etc/wwwacct.conf is empty; and from China, httpupdate.cpanel.net crawls at roughly 50 KB/s — slow enough to cause the first failure in the first place. Accept a provisioning only after three checks: MariaDB RPMs present, mysql service active, client connects.

Scenario one: install reports complete, MariaDB is missing​

WordPress throws a Database Error, the mysql binary does not exist on the server, and systemctl is-active mysql returns inactive — while the cPanel installer reported success and the panel opens fine.

The install log's tail hides the real event:

(FATAL): The background process "SQL Databases and dependent apps" failed ... error number 127

The MariaDB RPM transaction in the SQL stage failed on download, but every later stage ran and finished anyway, and the final screen still said complete. Installation "success" does not mean the components are present — the installer neither rolls back nor blocks on a stage failure.

Fix order:

1. Confirm the gap: rpm -q MariaDB-server — most likely not installed
2. Install the full RPM set: MariaDB-server, MariaDB-client, MariaDB-devel, MariaDB-shared, MariaDB-common — every one of them
3. RPMs alone are not enough — /usr/local/cpanel/scripts/securemysql does not make restore tools connect; restorepkg fails with Missing: admin_mysql_password. Create /root/.my.cnf with a client-section password and run SET PASSWORD for root@localhost

Never trust the installer's exit status for acceptance. Run three checks:

rpm -qa | grep -i maria
systemctl is-active mysql
mysql -N -e "select version()"

All three green, the SQL stage is genuinely done.

Scenario two: first account creation hits a chain of missing keys​

restorepkg or manual account creation gets blocked once per run, in this order: Please setup a nameserver → Missing HOMEDIR → Missing DEFMOD → Missing LOGSTYLE → Missing SCRIPTALIAS.

The cause is direct: on a fresh WHM that never ran the Basic Setup wizard, /etc/wwwacct.conf is an empty file, and account creation validates it hard. The trap is the error mechanism — each run reports exactly one missing key, so patching one at a time costs five or more rounds.

Write the full standard key set in one shot:

cat > /etc/wwwacct.conf <<'EOF'
ADDR 172.28.100.10
CLUSTERED_DNS disabled
DEFMOD default
ETHDEV eth0
FTPHOMEDIR 0
HOMEDIR /home
HOMEMATCH home
LANG english
LOGSTYLE semicolon
MINUID 500
NS ns1.example-ns.com
NS2 ns2.example-ns.com
SCRIPT x3
SCRIPT x3parked
SCRIPT x3addon
SCRIPTALIAS y
EOF

Three details:

• ADDR takes the private IP, not the public one — on NAT architectures the public IP never lands on the NIC, and the consequences of binding it are covered in cPanel Sites Hit the Default Page? The Aliyun NAT vhost Trap
• whmapi1 set_nameserver takes the singular parameter nameserver (values: bind/powerdns/disabled), unlike the plural fields from get_nameserver_config; and the NS validation reads NS/NS2 from wwwacct.conf, not ns1/ns2 from cpanel.config
• When real DNS lives on cloud DNS, the NS values are nominal placeholders — pair with CLUSTERED_DNS disabled

For failed transfers, the details live in the JSON of /var/cpanel/transfer_sessions/<session>/master.log (search failure); note that view_transfer itself tails and blocks — do not get stuck in it during triage.

Scenario three: cpanel.net downloads at 50 KB/s from China​

Scenario one's RPM download failure usually traces back here: from an Aliyun Shanghai ECS, every mirror IP of httpupdate.cpanel.net measured about 50 KB/s (the international-site route in the same region was just as slow, ruling out any proxy transit benefit); the same source over a residential connection measured 1.8-6 MB/s.

The acceleration pattern lets the server borrow a faster line: a reverse dynamic SOCKS tunnel from a local machine, with proxychains-ng wrapping the installer on the server:

# Local machine: open a remote dynamic SOCKS port
ssh -N -R 1080 root@<server-ip>

# Server: with proxychains-ng installed, run the installer through the tunnel
proxychains4 -q sh latest

Measured lift: from 50 KB/s to 708 KB/s, about 14x. Three traps to avoid:

• The proxychains config must exempt localnet ranges (10/8, 172.16/12, 100.64/10, etc.) and drop proxy_dns — otherwise Aliyun internal mirror domains (mirrors.cloud.aliyuncs.com) get pushed into the tunnel and fail outright
• tinyproxy is incompatible with httpupdate.cpanel.net — it returns 404 reliably; do not use it as the tunnel exit
• Never clean up the installer with pkill -f "sh latest" — the pattern matches your own ssh session's command line and kills your connection (the source of exit code 255); kill by PID instead

Tear it down when done: the tunnel lives exactly as long as the local ssh process, and the server keeps no proxy configuration — uninstall proxychains-ng and delete its config after the install. If an interrupted install already left RPMs missing, cPanel's self-repair is /usr/local/cpanel/scripts/sysup — in our case a missing splitlogs binary had left httpd unable to start, fixed by sysup plus a manual RPM install.

Frequently Asked Questions​

How do I install MariaDB on a cPanel server?​

Install the full RPM set — MariaDB-server, MariaDB-client, MariaDB-devel, MariaDB-shared and MariaDB-common; a partial set passes rpm checks but breaks later steps. Then create /root/.my.cnf with a client-section password and run SET PASSWORD for root@localhost, otherwise restore tools fail with Missing: admin_mysql_password. Verify with rpm -qa | grep -i maria, systemctl is-active mysql, and mysql -N -e "select version()".

Why does a cPanel install or update fail without an error?​

Because the installer does not roll back. When one background stage fails (SQL Databases and dependent apps, error number 127 in our case), the remaining stages still run and the installer still reports success — the only trace is a (FATAL) line at the tail of the install log. Always grep the log tail for FATAL and run the three-component acceptance checks before trusting the completion message.

How can I speed up cPanel downloads in China?​

Direct throughput from a China cloud server to httpupdate.cpanel.net measures around 50 KB/s across all mirror IPs, while the same source reaches 1.8-6 MB/s over a residential line elsewhere. The workable pattern is a reverse dynamic SOCKS tunnel from a faster network plus proxychains on the installer — measured 14x faster at about 708 KB/s. Treat it as temporary: uninstall the proxy tooling when the install finishes.

While delivering a Go + Fyne desktop login tool for a client, we hit one pitfall in each phase: config parsing, SSH handshake, cross-platform packaging, and GUI verification. All five are solved, and every fix is reusable.

The tool was built for the fully managed China hosting engagement for an industry-leading manufacturer — after the WHM entrance was upgraded from a single password to layered gates, the team needed a way to log in without ever touching the root password. This small tool is the client side of that channel: click one button, get a one-time link into WHM. Small tool, but "small" does not mean pitfall-free.

TL;DR​

Scenario 1: SSH fingerprint check always fails? Look at the trailing equals sign​

The tool's first security layer pins the host key: the client only trusts the ed25519 fingerprint hard-coded in its config, so a hijacked machine can't intercept the one-time link. The fingerprint comes from ssh-keyscan on the server side.

The first live test failed every connection with host key fingerprint mismatch. Only by placing expected and actual values side by side was the difference visible — a single trailing equals sign:

ssh-keygen -lf  →  SHA256:tT5rFtRWhCcsvJg58hNOXt0rqYvSPmur6pL8xE2KxQ   (43 chars, no padding)
Go StdEncoding  →  tT5rFtRWhCcsvJg58hNOXt0rqYvSPmur6pL8xE2KxQ=          (44 chars, one =)

The root cause is an encoding spec detail: an ed25519 public key is 32 bytes, which base64-encodes to 44 characters, the last one being padding. ssh-keygen -lf strips the padding and prints 43 characters; Go's base64.StdEncoding pads by standard. Both are "correct" — stacked together, they never match.

Normalize both sides before comparing — strip trailing padding, unify the SHA256: prefix:

// Normalize a SHA256 fingerprint: strip padding, unify prefix, validate on the way
func normalizeFingerprint(fp string) (string, error) {
	fp = strings.TrimPrefix(fp, "SHA256:")
	fp = strings.TrimRight(fp, "=")
	if _, err := base64.RawStdEncoding.DecodeString(fp); err != nil {
		return "", fmt.Errorf("invalid sha256 fingerprint: %w", err)
	}
	return "SHA256:" + fp, nil
}

After normalizing, a unit test covers both spellings in, one spelling out, and the four live-test paths — valid login, wrong fingerprint, unauthorized key, black-hole timeout — all behave as expected.

If you'd rather not hand-roll it: x/crypto's ssh.FingerprintSHA256 already returns the unpadded form, matching ssh-keygen. The trap isn't in the library — it's in rolling your own encoding on one side only.

Scenario 2: Explicit timeout parses to 0? Embedded struct fields get shadowed​

The config struct looked like this: most fields live in a general Config, and timeouts wanted "default when absent" handling, so they were declared as outer pointer fields alongside the embedded struct:

type Options struct {
	Config                    // embedded: host, port, timeouts...
	ConnectTimeout    *int    `json:"connect_timeout_seconds"`
	CommandTimeout    *int    `json:"command_timeout_seconds"`
}

The intent: pointer fields detect "did the user set this", and defaults fill in when nil. The live test instead reported command timed out after 0s — the timeout value explicitly present in config.json parsed to 0.

The root cause is encoding/json's conflict rule: when multiple fields map to the same JSON key at different depths, the shallower (outer) one wins and the embedded struct's same-name field is never populated. So Options.Config kept zero-value timeouts, while the outer pointer did receive the explicit value — but the code only ever touched it in the "nil means default" branch. The explicit value landed in a field nobody read back. Silent loss.

The fix: stop trying to parse and probe defaults in one layer. Unmarshal the same data twice, each pass doing one job:

type Config struct {
	Host           string `json:"host"`
	Port           int    `json:"port"`
	ConnectTimeout int    `json:"connect_timeout_seconds"`
	CommandTimeout int    `json:"command_timeout_seconds"`
}

type timeoutOverrides struct {
	Connect *int `json:"connect_timeout_seconds"`
	Command *int `json:"command_timeout_seconds"`
}

func load(raw []byte) (*Config, error) {
	cfg := &Config{ConnectTimeout: 30, CommandTimeout: 15} // defaults
	if err := json.Unmarshal(raw, cfg); err != nil {
		return nil, err
	}
	var ov timeoutOverrides
	if err := json.Unmarshal(raw, &ov); err != nil {
		return nil, err
	}
	if ov.Connect != nil {
		cfg.ConnectTimeout = *ov.Connect
	}
	if ov.Command != nil {
		cfg.CommandTimeout = *ov.Command
	}
	return cfg, nil
}

Pass one fills the plain fields into Config; pass two probes absence with pointer-only fields and overrides defaults only when a value is explicit. A regression test asserts "explicit values survive" — this class of bug is silent by nature, and without a test watching it, the next refactor will bring it back.

JSON traps aren't limited to parsing. On the serialization side we previously hit a sneakier one: json.dumps with default=str silently turns a Python set into a string, and the in check then quietly returns wrong answers. The common thread: both happen where the type system can't see.

Scenario 3: fyne-cross reports go.mod requires go >= 1.26.0? The container toolchain is locked​

The Windows build goes through fyne-cross, and packaging failed with go.mod requires go >= 1.26.0. The container ships go 1.25.10 while x/crypto v0.57.0 demands go 1.26+ — and the container defaults to GOTOOLCHAIN=local, which disables Go 1.21's toolchain auto-switching. Whatever version ships in the image is what you're stuck with.

One-line fix — let the container fetch the toolchain it needs:

fyne-cross windows -tags migrated_fynedo -env GOTOOLCHAIN=auto

GOTOOLCHAIN=auto lets the go command download and switch to the version required by go.mod automatically. Future dependency bumps won't require touching the container. The flag is now baked into the project's build.sh.

Scenario 4: Goroutine UI updates not guaranteed? Threading is opt-in until 2.9​

Fyne 2.6 introduced a strict threading model where UI updates must go through fyne.Do. The easy-to-miss part: in v2.6–2.8 neither the model nor fyne.Do is enabled by default. Without explicitly opting in, mutating UI widgets from a background goroutine has no guaranteed behavior; the default flips only in v2.9.

This tool runs SSH handshakes and link fetches in background goroutines and then updates a status line — right inside that window. Enable it with a belt-and-suspenders pair:

# FyneApp.toml
[Migrations]
fyneDo = true

# plus the build tag
go build -tags migrated_fynedo .

One switch is runtime config, the other is compile-time; either alone is enough, and setting both guards against updating only one. On the code side, make it a habit:

go func() {
	link, err := fetchOneTimeLink()
	fyne.Do(func() {
		if err != nil {
			status.SetText("Failed: " + err.Error())
			return
		}
		status.SetText(link)
	})
}()

After upgrading to v2.9, none of this needs to change, and the migration markers can stay.

Scenario 5: WSLg screenshots of Fyne windows come out black? Verify with server-side evidence instead​

The acceptance criterion was "double-click the program, press the button, the browser opens the WHM login page." For pixel-level GUI verification we found that under WSLg, screenshotting a Fyne (OpenGL) window with scrot or import yields pure black frames — GL windows render outside the X11 pixel-capture path, so conventional screenshot tools get nothing.

There is no "fix" for this one — it's a workaround, stated as such:

• Keyboard events do get through: xdotool sending Tab + Return successfully activated the button, so GUI automation on the input side works;
• XTest mouse clicks did not work — don't burn time on them;
• Pixel-level verification is abandoned in favor of server-side evidence: the button triggers a privileged whmapi1 call on the server, so comparing the sudo log count before and after the click (21 → 22 in our run) proves "the GUI really drove the whole chain" — no screen pixels required.

For tools where the GUI is just a trigger and the real action happens server-side, a server log count is actually stronger evidence than a screenshot: it proves behavior, while a screenshot only proves appearance.

FAQ​

Why is my Go json.Unmarshal embedded struct field always zero?​

encoding/json resolves same-name JSON keys by depth: the shallower outer field wins and the embedded struct's same-name field is never populated. Unmarshal the same data twice — once into Config, once into a pointer-only probe struct — and copy explicit values over the defaults. Add a regression test asserting explicit values survive.

How do I update Fyne UI from a background goroutine in v2.6–2.8?​

In Fyne 2.6–2.8 the strict threading model and fyne.Do are opt-in: set fyneDo=true under [Migrations] in FyneApp.toml or build with -tags migrated_fynedo (default only from v2.9). Wrap every UI update in fyne.Do, otherwise behavior is not guaranteed.

Why does my Go SSH fingerprint check always fail?​

Padding: ssh-keygen prints 43 unpadded base64 characters for a 32-byte ed25519 key, while Go's base64.StdEncoding appends '='. Normalize both sides — strip trailing '=', unify the 'SHA256:' prefix — before comparing, or use x/crypto's ssh.FingerprintSHA256, which is already unpadded.

TL;DR​

Stopping campaigns is where marketplace ad budgets leak fastest: waiting one extra week on a keeper costs little, while keeping a real underperformer one week costs real money. Five checks — runtime, spend, continuity, learning period, true zero-inquiry — are not folklore. They are five rules that actually run inside a production judging codebase, each with an explicit numeric threshold: 16-day settlement, ¥100 weekly spend, 25% cost gap, 5-record samples, 3× spend. Four say "wait," one says "stop now." Every case below comes from one store's 40-week ad ledger, judged on settled data only.

The situation: half the "bad campaigns" on Monday's report were wronged​

Monday review: a campaign spent real money last week with zero inquiries, and your hand is already on the pause button. Hold it — in one industrial-supplies store's 40-week ledger (anonymized), most campaigns that "looked bad" were simply not done proving themselves: some hadn't run a full cycle, some had spent nothing at all, some had a delivery gap in the middle. A wrong stop costs twice: the killed campaign's future inquiries, and the money a real underperformer keeps burning while you hesitate. The cases below come from a campaign-verdict audit while building AI Operations.

Why: four checks say wait, one says stop​

Check 1: Has it run long enough? — only settled data counts​

Rule first: the judgment consumes settled weeks only — a calendar week clears the settlement line 16 days after it ends, and the line is strict: day 16 itself is not settled, the next day is. Nothing unsettled enters a verdict. Measurement shows 16 days is a floor, not the tail: one collection run wrote 27 new region rows into a week that had ended 29 days earlier — see Is 16 Days Enough for Marketplace Ad Data?.

In plain terms: take the new program below — across its 7 settled weeks, weekly cost per inquiry swung from ¥46 to ¥65. The eighth week, which has not cleared the line yet, currently reads ¥99 — it settles on September 8, and this article draws no conclusion from it. Judge the program on unsettled data, or on any single week you happen to pick, and the verdict is wrong either way.

(Technical note: inquiry attribution back-fills over time; the settlement line exists precisely to fence off "draft" data. A day-29 back-fill means the fence needs a watching period behind it, not just the line.)

Check 2: Has it spent enough? — thin spend is not judged​

Rule first: the rules turn "hasn't spent enough" into a hard gate — campaigns whose settled-week average spend sits below ¥100 are filed as "test" and receive no performance verdict; a first settled week with fewer than 5 inquiries plus leads is protected, not judged.

In plain terms: one "precision targeting" campaign appeared in the ledger twice — April and July — for 6 weekly rows and a lifetime spend of ¥0. It held a name on the report without spending a cent, never earning the right to be judged. Another veteran campaign (the store-wide self-serve program) fell to ¥90 a week in June — under a tenth of its ¥1,197 peak. At that scale the rules file it under "test" too; the word "stop" never comes up. When thin spend looks bad, the finding is "spent too little," not "performs badly."

(Technical note: a ¥0 weekly row means the campaign never passed the platform's delivery checks or is budget-throttled; a thin-spend sample is all noise, and noise drowns every ratio computed on it — the gate exists to block exactly that false signal.)

Check 3: Is delivery continuous? — interrupted campaigns get two paths​

Rule first: continuity is measured as delivery density — weeks actually delivered ÷ weeks spanned — and under 80% counts as interrupted. An interrupted campaign gets two paths only: if effective cost-per-acquisition runs more than 25% above benchmark, stop, with the reason stated as "the bidding model keeps re-learning"; below the threshold, the file is marked "optimize" and the only action is: restore continuity.

In plain terms: one measured 3-week gap dropped store-wide weekly spend from ¥1,960 to ¥281 → ¥90 → ¥281, with inquiries hitting zero in the middle. Once continuous delivery resumed, store-wide cost per inquiry jumped from ¥33 before the gap to ¥46 in the restart week — restarting after a gap is not starting from where you left off, which is exactly why the rule restores continuity before judging.

(Technical note: the traffic mix before and after a gap can differ, so grading the restart against the pre-gap baseline runs systematically optimistic; "keeps re-learning" refers to the bidding model falling back to cold start after every interruption.)

Check 4: Was the learning period honored? — protection lasts one extra week at most​

Rule first: the learning-period protection here is not a fixed number of weeks. It fires only when the first settled week is continuous but sample-poor — fewer than 5 inquiries plus leads — and grants at most one more week; from the second settled week on, there is no protection at all. From there, every week is measured by the same ruler: effective CPA above target by more than 25%, combined with a thin inquiry share (under 15%, or under 70% of the store's own level) — or spend still trending up (up more than 12% over two weeks with cost above target, or a rising 3-week slope) — means the stop tier; gaps past 50% with cost above target are judged even faster.

In plain terms: the same store's new program — the "Merchant growth" plan — launched in late June, and its human operators kept waiting: by press time it had 8 delivery weeks on the books, 7 of them settled. Weekly cost per inquiry ran ¥46–65 across the settled weeks — not one week back inside the store's own normal band of ¥25–31, with the cheapest week still nearly 50% above the band's ceiling. Over the same stretch, two other new programs in the same store, aimed at the same products — the "potential-customer harvest pack" and the "cross-border express program" — bought inquiries at ¥30 and ¥35 across the same 7 settled weeks. So neither "the market got expensive" nor "it hadn't started yet" holds. The 8 weeks of patience came from people, not from the rules: under the judging logic, from the second settled week on, this program had no protection and should have been measured every single week.

(Technical note: effective CPA = spend ÷ (quality inquiries + plain inquiries × 0.6 + raw leads × 0.1) — raw leads are worth little, they cannot prop up the denominator, and piles of junk leads cannot buy a cheap cost. Across the 7 settled weeks the program read ¥15,188 ÷ 236 inquiries = ¥64.4, 2.3× the store's historical median of ¥28.)

Check 5: Is it truly zero-inquiry? — the only stop-now tier​

Rule first: accumulated spend above 3× the target cost-per-acquisition with zero total inquiries is a hard stop; when no target is configured, the threshold degrades to 3× the campaign's own settled-week average spend. One tier fires even earlier: a closed-but-unsettled calendar week (Sunday passed, still inside the attribution window) spending past max(¥300, 3× the weekly average) with zero inquiries is an early hard stop — it does not wait for settlement. And the target is never hand-set — it is the median across the store's last 12 settled, computable weeks, updated automatically.

In plain terms: this is the one check you never hesitate on. Its real battlefield is the keyword layer: across 46 weeks, 78% of the same store's 1,641 keyword-week records produced no inquiry while absorbing 27% of keyword spend — see 78% of Keywords Never Brought an Inquiry.

(Technical note: the early stop dares to skip settlement because spend is real-time billing — fixed once written, zero drift measured on settled weeks — while inquiries are a conversion field that back-fills from zero; the pair of conditions, a high bar and a closed week, is what bounds the false-kill risk.)

The experiment and the data​

The five checks' thresholds at a glance (values live in the production judging code):

• Sample: one industrial B2B store (anonymized), campaign-by-week ad ledger from Nov 2025 to Aug 2026 — 40 weeks; the keyword layer covers 46 weeks and 1,641 keyword-week records of the same store.
• Calibers: inquiry cost = weekly spend ÷ weekly inquiries (cumulative uses total spend ÷ total inquiries); effective CPA = spend ÷ (quality inquiries + plain inquiries × 0.6 + leads × 0.1). The "normal band" is the store's own median weekly cost across the 28 normal weeks before the new programs launched (3 spring-festival weeks and 1 zero-inquiry gap week excluded) — ¥28 ±10%, i.e. ¥25–31.
• Settlement boundary: every verdict in this article reads settled weeks only. As of press time (Sep 4, 2026) the newest settled week is the week of Aug 10; the program's 8th week (week of Aug 17) settles on Sep 8 and appears here as an unsettled observation only.
• Judging code: every rule and threshold cited here was verified against the production judge (the thin-spend gate, the interrupted-delivery branch, the learning-period sample gate, and the two-tier stop plus hard stop); file- and function-level provenance is an internal record and stays out of the article.
• Anonymization: no store or campaign IDs appear; campaigns are referred to by their public platform program names.

What it's worth: two accounts​

The account of stopping late. Those 7 settled weeks of the "Merchant growth" program: ¥15,188 spent for 236 inquiries. At the store's own median of ¥28 across 28 normal weeks, the same 236 inquiries should have cost about ¥6,600 — 7 settled weeks of overpaying, roughly ¥8,600. Under the rules, protection lapsed at the second settled week and the program should have been measured weekly — every extra week of human patience was real money.

The account of not stopping. The 78% zero-inquiry keyword records carried ¥8,962 of real spend — 27% of the store's ¥33,417 keyword budget — without producing a single inquiry. Cutting them touches no campaign structure, and the money returns the same week.

For operators​

1. Runtime: judge on settled weeks only (+16 days after week end); when single weeks swing hard, only cumulative numbers count.
2. Spend: below a ¥100 settled-week average, fund it before judging it; a ¥0 weekly row is a delivery question, not a performance question.
3. Continuity: under 80% delivery density, restore continuous delivery first; after a gap, reset the baseline — never grade the restart against gap weeks.
4. Learning period: protection belongs to a sample-poor first settled week, one extra week at most; "give the new program time" stops being an argument at the second settled week.
5. True zero-inquiry: past 3× your target cost-per-acquisition with still zero inquiries — stop now, at campaign level and keyword level alike.

For developers​

1. Persist both granularities: campaign-by-week and keyword-by-week are separate tables — the keyword layer is where the stoppage money lives, and campaign-level views never see it.
2. Keep collection audit fields: re-collection rewrites historical weeks (measured: new rows arrived on day 29), so your pipeline must distinguish "what was visible then" from "settled data."
3. Keep thresholds in one place: gather every gate into a single configuration and keep the judging logic free of scattered magic numbers — tune thresholds without touching logic, and version every logic change.
4. Make the short-circuit order explicit: the five checks are not parallel options but a short-circuit chain — who judges first and what short-circuits what decides the verdict. The production judge's actual order:

1 Early stop  : closed-but-unsettled week spend > max(¥300, 3× weekly avg), zero inquiries → stop (runs first, skips settlement)
2 Hard stop   : lifetime spend > 3× target cost, zero inquiries → stop
3 Test gates  : settled weeks < 1 → test; weekly average spend < ¥100 → test
4 Gap branch  : delivery density < 80% → gap > 25% stops; no benchmark or gap below the line → optimize (restore continuity)
5 Learning    : first settled week only, continuous, inquiries + leads < 5 → test (one extra week at most)
6 Stop tier   : settled ≥ 2 (≥ 3 without a target): cost above target and gap > 50% → stop; gap > 25% with (thin inquiry share or no improvement) → stop
7 Keep tier   : gap ≤ 5% and cost ≤ target and continuous → keep
8 Fallback    : everything else → optimize
9 Store guard : if everything gets stopped, the biggest non-hard-stop spender is downgraded to optimize (or a sample-poor plan is kept when none qualifies)

FAQ​

How long should a B2B ad campaign run before judging it?​

Judgment reads settled weeks only: a calendar week clears the settlement line 16 days after it ends, and measured back-fill has arrived as late as day 29. The first settled week is judgeable, but a first week with fewer than 5 inquiries plus leads is protected for one extra week at most.

Which failure justifies stopping a campaign immediately?​

Lifetime spend past 3× the target cost-per-acquisition with zero total inquiries — the hard-stop tier, with the target auto-set to the median of the last 12 settled weeks. An earlier tier fires too: a closed-but-unsettled week spending past max(¥300, 3× its weekly average) with zero inquiries stops without waiting for settlement.

How do I judge a campaign after a delivery gap?​

Delivery density under 80% counts as interrupted: stop if cost runs more than 25% above benchmark (the bidding model keeps re-learning), otherwise restore continuity first. One measured 3-week gap pushed store-wide inquiry cost from ¥33 to ¥46.

All five checks are built into AI Operations — LLM-powered analysis that reads market trends, buyer behavior, and sales data to ground your operating decisions in numbers. It waits when waiting is right, and flags the stop a week early.

TL;DR​

July data from 18 audience packages in one store: two crowds with inquiry costs just 10% apart (¥38.9 vs ¥43.0) ran 4× apart in ROI (5.30 vs 1.25). Inquiry cost and ROI answer different questions — cost per inquiry says whether the traffic was bought expensively; ROI says whether the buyers were worth it — and either ruler alone, used in an environment month, will mislead you.

The situation: ranking by one metric produces fiction​

The comfortable way to read a crowd report is to sort it: by inquiry cost, cut the priciest; by ROI, cut the worst. In July's real data, those two sorts disagree completely. This reconciliation came out of a crowd-report audit while building AI Operations.

The data: one ruler prices traffic, the other grades it​

Across crowds (one month, 18 packages): inquiry costs spread ¥32–44, yet near-identical costs carried ROI from 1.25 to 5.30 — "store new-buyers" at ¥43.0 cost only 10% more than "cross-border buyers" at ¥38.9, and returned a quarter of the ROI. Inquiry cost measures what it takes to pull in one interested buyer; what those buyers then purchase, and at what value, is invisible to it.

Across months (the same store, seven months): January–June crowd inquiry costs held at ¥16–21 with ROI 8.4–18.5; in July, spend tripled (×3.1), cost doubled to ¥39, and ROI collapsed to 4.0. All 18 packages breached together — July was an environment month (platform competition, market-wide moves), not one crowd suddenly failing.

(Technical note: the report's ROI uses 15-day-attributed GMV — while measured attribution back-fill runs as late as day 29 after week end, see Is 16 Days Enough for Marketplace Ad Data?. That ROI only counts what landed inside 15 days: systematically low for long-cycle B2B buyers, and still drifting between months as the ledger finishes posting.)

What it's worth: the mis-pruning ledger​

Repricing by a single ruler in an environment month is wrong in both directions: June's good environment (ROI 18.5) inflates every crowd and hides the ones that genuinely need fixing; July's bad environment (ROI 4.0) condemns them all — including crowds that were merely dragged down by the month. Separating "environment" from "crowd" is what makes pruning precise: what deserves cutting deserves it in good months too; nothing gets cut for the weather.

Disciplines for operators​

1. Read both rulers together: inquiry cost prices the traffic; ROI grades it. Similar costs with multiples-apart ROI is an audience-selection problem — repricing cannot fix it.
2. No rankings in environment months: when store-wide crowd spend and costs move together (as in July), cross-audience comparisons are void that month.
3. Discount the ROI: 15-day-attributed ROI runs systematically low for B2B and drifts until settlement closes (when the platform locks the period's numbers and stops back-filling) — it has not earned the "sole benchmark" chair.
4. Reprice on consecutive trends: single months are noise; three months in one direction with real magnitude is a trend. The full monthly procedure lives in The 1688 Crowd Premium Monthly Method.

The judgment order for developers​

The discipline above ships as an automated pipeline in the crowd report. A comparable month = a finalized month (16 days after month-end, the attribution cutoff) with spend in that month; for cost-trend math, months with zero inquiries are filtered out as well. Checks run in dependency order, later checks overriding earlier conclusions:

(Technical note: the environment check sits after the trend check, not before — each crowd gets its own direction first, then the store-wide overlay flips directional calls to frozen; an environment month freezes the advice without swallowing state groups like low-spend or insufficient data. The freeze trigger is the inquiry-cost ratio, not spend: if spend doubles and inquiries double with it, cost hasn't moved and month-over-month self-comparison still works — freezing only when the efficiency baseline itself shifts. The 0.40 threshold comes from the measured split — normal months move 0.5%–27.6%, structure months 98%–116% — and 0.40 sits mid-band. And the 15-day ROI is display-only in this pipeline; it never enters the judgment.)

FAQ​

Should audience performance be judged on ROI or inquiry cost?​

Both, always: inquiry cost is the price of the traffic, ROI is its quality. Either one alone gets bent out of shape by environment months or single big orders.

Why can similar-cost audiences differ 4× in ROI?​

Inquiry cost only says whether the traffic was expensive, not whether the buyers convert. Audiences with different order sizes and paths turn the same inquiry price into very different GMV.

What to do in a month when all crowd costs jumped together?​

Call it an environment month — when store-wide crowd spend and costs double together, no cross-audience repricing; return to each crowd's own trend after the environment recovers.

That "two rulers + environment detection" method for crowd reports is built into AI Operations — LLM-powered analysis that automatically surfaces market trends, user behavior, and sales data to drive strategy. A crowd report deserves more than one sort button.

TL;DR​

One store's keyword ledger: of 1,641 keyword-week records, 1,277 (78%) produced zero inquiries while consuming ¥8,962 — 27% of total spend. The average over "keywords that did produce" comes to ¥32; the blended cost that counts every yuan of spend is ¥41 — a 28% understatement. The only correct formula is total spend ÷ total inquiries: not one yuan of zero-inquiry spend may vanish from the denominator.

The situation: the cost you see is the survivors' cost​

Keyword reports are organized per keyword, so your eyes land on keywords that produced inquiries — those have a cost to show. Zero-inquiry keywords display no cost, and so they quietly exit your field of view.

Their spend, however, left the account in full. This audit came out of a keyword-level check while building AI Operations.

The data: the missing 27%​

The naive average only bills the survivors — (technical note: this is textbook survivorship bias in ad data. Counting only producing samples donates the non-producing samples' spend for free; with 27% of the money missing from the denominator, the cost "improves" by two to three tenths.)

What it's worth: what 28% understatement does​

The understatement is not cosmetic — it cascades:

• Acquisition budget: budget set at ¥32 while reality is ¥41 leaves a ¥9-per-inquiry hole — across 824 inquiries, about ¥7,400
• Product go/no-go: a product line judged against an understated keyword cost reads "still viable" while truly underwater
• Pricing and margin: acquisition cost is the hidden floor of B2B quotes; a floor 28% too low cannot carry real deal prices

Disciplines for operators​

1. One base formula: cost = total spend ÷ total inquiries. Any "average" that excludes zero-inquiry samples is void on sight.
2. Keep a separate zero-inquiry watchlist, sorted by accumulated spend — this is the main battlefield of "check five: truly zero inquiries"; words that burn past a reasonable cost with nothing to show get stopped without ceremony.
3. Quality weighting is layer two: once the base is right, weight purchase-ready inquiries (pricing asks, sample requests, volume) above casual ones. No universal weights exist — derive them from your own deal path and freeze them, so months stay comparable.
4. The target line comes from your own history: normal months (holidays excluded) define the band — the method is in A Real Store-Wide Efficiency Alert, From a 40-Week Ledger.

FAQ​

What is the right way to calculate B2B ad inquiry cost?​

Total spend ÷ total inquiries — no exceptions. Leave zero-inquiry spend out of the denominator and the cost reads two to three tenths too low.

Should zero-inquiry keywords be paused?​

Check accumulated spend and observation window first: spend clearly above a reasonable cost with still zero inquiries means stop; freshly added keywords deserve a full settlement cycle.

Is quality-weighting inquiries still worth doing?​

Yes — as a second layer. Fix the base formula first (zero-inquiry spend must not vanish from the denominator), then grade purchase-ready vs casual inquiries.

That base-formula discipline is built into AI Operations — LLM-powered analysis that automatically surfaces market trends, user behavior, and sales data to drive strategy. One notch wrong on the cost formula, and everything downstream is wrong.