2 posts tagged with "Alibaba Cloud"

While using Aliyun Cloud Assistant RunCommand to start an strace sampler, a watch loop and a periodic sampler on a server, the background jobs were silently reaped the moment the call returned — the output was an empty file, or a half file with exactly one sample, and nothing ever logged an error.

Ran into this while providing managed hosting for a WordPress site in China — full-site migration, image CDN offload, daily backups and automated patrols.

TL;DR​

When a Cloud Assistant invocation ends, the agent cleans up the background processes derived from it, and nohup does not save them — it only ignores SIGHUP and redirects output, leaving the process in the original session, right inside the cleanup's reach. A new session via setsid is what escapes: setsid nohup cmd < /dev/null &. And before reading results back, verify the output file's line count and mtime — do not read an empty file as "the command never ran".

Symptoms: the call returns, the background jobs vanish​

Background jobs started with nohup cmd & inside one RunCommand invocation died in three different shapes:

All three share two traits: the invocation itself returned Success, and the server logged no error anywhere. Searching "nohup process disappeared" mostly points at SIGHUP — but nohup was already in place.

Root cause: nohup never changes the session, and cleanup sweeps by session​

nohup's contract is two lines long: ignore SIGHUP, redirect output to nohup.out. What it does not do is change the session — the process stays inside the session the Cloud Assistant agent created for this invocation. When the invocation finishes, the agent cleans up the processes derived from it, grouped by session, and nohup jobs fall squarely inside.

setsid is the semantics that actually addresses this: it puts the new process into a brand-new session, detached from the invocation's. Detached, the end-of-invocation sweep never reaches it — verified across all three job types, all of which survived once setsid went in.

One honest boundary: whether the agent sweeps by process group or by session was not verified at source level. But "leave the original session and you survive" held as a sufficient condition on every job tested, and that is enough.

Fix: the setsid three-piece form, plus output verification​

Detaching from session, terminal and foreground in one line:

# Long-running job started from inside a Cloud Assistant script (sampler example)
setsid nohup /usr/local/bin/sampler --interval 60 -o /tmp/samples.csv </dev/null >/tmp/sampler.log 2>&1 &

Do not drop </dev/null — a background job whose stdin still hangs on the invocation's terminal semantics can survive the cleanup and still stall.

Cloud Assistant is asynchronous in three steps (CreateCommand → InvokeCommand → DescribeInvocationResults); fetch results about 10 seconds after invoking, and let the script exit normally after spawning the background job — no waiting required.

Before reading results back, verify the job actually lived:

# Line count should match the expected sample count; mtime should keep advancing
wc -l /tmp/samples.csv
stat -c %Y /tmp/samples.csv

An empty file or a single sample means the job died — go check the setsid form, not the business command. This "spawn remotely, read back later, verify output first" loop, combined with Cloud Assistant's ActionTrail audit trail, is a complete emergency-diagnostics workflow.

The lesson also explains the architecture behind CCLee Server Sentinel, a Linux server monitoring service: collection runs through a resident agent and local cron on each machine, and backup scripts run locally without depending on a central server — a long-lived job's lifecycle should never be bound to the tail end of a single remote invocation.

FAQ​

Why did my nohup process disappear anyway?​

nohup does exactly two things: ignore SIGHUP and redirect output. The process stays in the original session, and Cloud Assistant cleans up the invocation's derived processes by session when the call ends — in one verified case strace, a sampler and a watcher all died while the invocation itself returned success with no error. A new session via setsid is what escapes the cleanup.

A nohup job quits after running for a while — what's going on?​

Three usual causes: the launcher's session being torn down, stdin still attached to a closed terminal, or a vanished redirect target. Check the timing: the sampler in this case stopped after exactly one sample, in sync with the invocation returning — that pinned the cause to invocation cleanup, not a crash.

How do I keep a script running in the background of a Cloud Assistant RunCommand?​

Use the three-piece form: setsid nohup cmd < /dev/null &. The new session is outside the end-of-invocation cleanup, so the job survives across calls. Before reading results, check the output file's line count and mtime — an empty or half-filled file means the job died, not that the command never ran.

Restoring a cPanel account from a cpmove archive on an Aliyun ECS instance: restorepkg reports success, httpd -S shows the namevhosts — and the browser greets every domain with the server's default page.

Encountered this while migrating a client's two China sites into a cPanel environment on Aliyun China — the very first public verification after the restore ran into this cloud architecture trap.

TL;DR​

On Aliyun ECS the NIC only carries a private IP; the public IP is edge NAT and never lands on the machine. Vhosts migrated from the old server bind that non-existent public address, never match at runtime, and all traffic drops into the wildcard default vhost. Fix all three IP stores (/etc/wwwacct.conf, /var/cpanel/users/, /var/cpanel/userdata/) and rebuild httpd.conf; write the private IP into /etc/mainip so new accounts start correct.

Symptoms​

Every migration step looks clean: restorepkg finishes without errors and the namevhosts are right there in the Apache config. But public verification falls flat — every domain lands on the same defaultwebpage redirect, and the site's domlog stays empty forever.

The clincher is a single static-file request: fetch a file that certainly exists and was never touched (say /some-real-page.html) and it returns 404 — with the error page styled from /var/www/html/*.shtml, the cPanel default site's directory. The real document root never received the request.

$ httpd -S | grep example.cn
203.0.113.10:80                   example.cn ...

The vhost binds 203.0.113.10 — the old server's public IP. This new machine's NIC has no such address.

Root causes​

The public IP is not on the machine. hostname -I returns only a private address like 172.28.100.10. Aliyun ECS public IPs are edge NAT: traffic arrives at Alibaba's gateway and is forwarded to the instance's private address — the public IP never appears on the NIC.

The restore inherited the old IP verbatim. The old server was a classic VPS with the public IP on the NIC, so its vhosts record exactly that address. cpmove brings the configuration over as-is, and every vhost Address now points at an address that does not exist locally.

Apache matches vhosts by IP:port. Requests arrive, match no namevhost, and fall into the *:80 default vhost (DocumentRoot /var/www/html). That single mechanism explains all three symptoms: the default page, the empty domlog, and the mismatched 404 styling.

Fix​

1. Confirm the NAT setup. One command:

hostname -I
# 172.28.100.10   ← private segment only: NAT confirmed

2. Rewrite all three IP stores — all of them, no exceptions. cPanel keeps IP information in three places serving different flows. Fixing only userdata rebuilds the vhosts correctly, but a stale wwwacct.conf writes the wrong address into every future account.

# 2a. Global default (read by account creation)
sed -i 's/^ADDR=.*/ADDR=172.28.100.10/' /etc/wwwacct.conf

# Default IP for new accounts — set this too
echo 172.28.100.10 > /etc/mainip

# 2b. Per-account record
sed -i 's/^IP=.*/IP=172.28.100.10/' /var/cpanel/users/example

# 2c. Bulk rewrite of userdata — the only input rebuildhttpdconf trusts
cd /var/cpanel/userdata/example
for f in *; do
  [ -f "$f" ] && sed -i 's/^ip: .*/ip: 172.28.100.10/' "$f"
done

The [ -f "$f" ] guard is not decorative: the userdata directory mixes in socket files like scope, and sed errors out on them without the check.

3. Rebuild and restart.

/scripts/rebuildhttpdconf
/scripts/restartsrv_httpd

4. Verify. httpd -S should now show namevhosts on the private IP; from the server, a request with an explicit Host header tells you which vhost answers:

curl -s -H "Host: example.cn" http://172.28.100.10/some-real-page.html -o /dev/null -w "%{http_code}\n"
# 200 ← no longer the default vhost's 404

Finish with an external visit to confirm the pages load and the domlog starts filling.

Edge cases and variants​

• Fresh installs: write /etc/wwwacct.conf ADDR and /etc/mainip before creating any account and the wrong-address problem never happens; this applies beyond restores.
• Not Aliyun-specific: AWS Elastic IPs and similar NAT-style public IPs behave the same. Anywhere the public IP is absent from hostname -I, cPanel's IP settings must use the private address.
• Local verification blind spot: testing from the server against 127.0.0.1 or the public address hits the default vhost and misleads — use the private IP with an explicit Host header.

Frequently Asked Questions​

Why does my cPanel site show the default page after a migration?​

Because the vhost address never matches. On Aliyun ECS the NIC only carries a private IP — the public IP lives on an edge NAT gateway. A restore inherits the old server's public IP, Apache matches vhosts by IP:port, no vhost ever matches, and everything falls into the wildcard default vhost. Rewriting three IP stores to the private IP and rebuilding httpd.conf fixes it.

How do I check which Apache vhost is serving my requests?​

Two checks give a verdict: the site's domlog stays completely empty, and a request for a static file that certainly exists returns 404 with an error page styled from the default document root (/var/www/html). Both together mean the default vhost is answering. Then compare httpd -S against hostname -I to see whether the bound address actually exists on the NIC.

Which cPanel IP settings need updating on a NAT cloud server?​

Three places together: ADDR in /etc/wwwacct.conf (global default), IP= in /var/cpanel/users/ account files, and every ip: field under /var/cpanel/userdata/ — then run /scripts/rebuildhttpdconf and restart httpd. For new accounts the default comes from /etc/mainip; writing the private IP there before migration prevents the problem from recurring.