# Standby Node Playbook

> Convert any fresh VPS to a Filter or Proxy role within 15 minutes.

> CAUTION: This playbook is an offline operational document. It contains WG mesh IPs and node topology details.
> It must NEVER be deployed to any server. Keep on Y:\ encrypted drive only (Rule 47).

---

## Prerequisites

| Requirement | Minimum |
|---|---|
| OS | AlmaLinux 9+ or Rocky Linux 9+ (preferred), Debian 12+ acceptable |
| nginx | Installed and running |
| WireGuard | Installed |
| Podman | Installed |
| Disk | 10GB free |
| RAM | 512MB minimum |
| SSH | Access via gabriella key, user admin with NOPASSWD sudo |
| Python | 3.9+ |
| Firewall | firewalld active (AlmaLinux) or iptables/nftables |

## Pre-Flight Checks

```bash
nginx -v
wg --version
podman --version
df -h /
free -m
sudo whoami
getenforce
```

---

## Step 1: Identity Setup (2 min)

### 1.1 Set hostname

```bash
sudo hostnamectl set-hostname <cover-name>
```

### 1.2 Create node identity

```bash
sudo mkdir -p /opt/svc /opt/sysc/engine/config
echo "<NODE_ID>" | sudo tee /opt/svc/node_id
```

### 1.3 Deploy SSH key

Ensure /home/admin/.ssh/authorized_keys contains the gabriella public key.

### 1.4 Verify SSH from Shield

```bash
ssh -p 9922 -i /home/admin/.ssh/gabriella_key admin@<new_node_wg7_ip>
```

---

## Step 2: WireGuard Mesh (3 min)

### 2.1 Generate wg7 keypair

```bash
sudo mkdir -p /etc/wireguard/keys
wg genkey | sudo tee /etc/wireguard/keys/wg7.key | wg pubkey | sudo tee /etc/wireguard/keys/wg7.pub
sudo chmod 700 /etc/wireguard/keys
sudo chmod 600 /etc/wireguard/keys/wg7.key
```

### 2.2 Create wg7.conf

```ini
[Interface]
Address = <ASSIGNED_WG7_IP>/24
ListenPort = 41652
PostUp = wg set %i private-key /etc/wireguard/keys/wg7.key

[Peer]
PublicKey = <SHIELD_WG7_PUBKEY>
Endpoint = 194.36.190.111:41820
AllowedIPs = 10.69.69.0/24
PersistentKeepalive = 25
```

Get Shield wg7 public key from Shield: sudo wg show wg7 public-key

### 2.3 Add peer on Shield

```bash
sudo wg set wg7 peer <NEW_NODE_PUBKEY> allowed-ips <NEW_NODE_WG7_IP>/32
sudo wg-quick save wg7
```

### 2.4 Bring up and enable

```bash
sudo wg-quick up wg7
sudo systemctl enable wg-quick@wg7
```

### 2.5 Verify

```bash
ping -c 3 10.69.69.1
sudo wg show wg7
```

---

## Step 3A: Filter Role (5 min)

### 3A.1 Deploy WAF config

Create /etc/nginx/conf.d/waf.conf:

```nginx
geo $allowed_proxy {
	default 0;
	<PROXY_WG7_IP>/32 1;
	127.0.0.1/32 1;
}
limit_req_zone $binary_remote_addr zone=waf_req:10m rate=25r/s;
limit_conn_zone $binary_remote_addr zone=waf_conn:10m;
map $http_user_agent $is_bot {
	default 0;
	~*(?:scrapy|python-requests|curl/|wget|httpclient|phantom|selenium|headless|Go-http|fasthttp|aiohttp) 1;
}
server {
	listen 80;
	server_name cleanearth.cc cleanpharma.cc _;
	if ($allowed_proxy = 0) {
		return 444;
	}
	server_tokens off;
	limit_req zone=waf_req burst=50 nodelay;
	limit_conn waf_conn 30;
	location ~* /(wp-admin|wp-login|xmlrpc\.php|\.env|phpmyadmin|admin\.php|\.git|autodiscover|owa|exchange) {
		proxy_pass http://127.0.0.1:9998;
		proxy_set_header Host "honeypot";
		proxy_set_header X-Trap-IP $remote_addr;
	}
	location ~* \.(bak|sql|tar|gz|zip|log|ini|conf|cfg|old|orig|swp)$ {
		return 444;
	}
	location / {
		if ($is_bot) {
			return 444;
		}
		proxy_pass http://10.69.69.1:80;
		proxy_http_version 1.1;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $http_x_real_ip;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
		proxy_set_header X-Store-ID "";
		proxy_hide_header X-Powered-By;
		proxy_buffering on;
		proxy_buffer_size 8k;
		proxy_buffers 16 8k;
		proxy_connect_timeout 5s;
		proxy_send_timeout 15s;
		proxy_read_timeout 30s;
	}
	error_page 502 503 504 @cf_error;
	location @cf_error {
		default_type text/html;
		return 503 '<html><body><h1>Service Unavailable</h1></body></html>';
	}
}
```

### 3A.2 Disable existing default config

```bash
sudo mv /etc/nginx/conf.d/webserver.conf /etc/nginx/conf.d/webserver.conf.backup 2>/dev/null
```

### 3A.3 SELinux (AlmaLinux/Rocky only)

```bash
sudo setsebool -P httpd_can_network_connect 1
```

### 3A.4 Test and reload nginx

```bash
sudo nginx -t && sudo nginx -s reload
```

### 3A.5 Deploy honeypot container

```bash
sudo mkdir -p /var/log/honeypot
sudo podman run -d --name blackhole-filter \
    -p 127.0.0.1:9998:80 \
    -v /var/log/honeypot:/var/log/honeypot:Z \
    --restart=always \
    docker.io/library/nginx:alpine
sudo podman generate systemd --name blackhole-filter --new | sudo tee /etc/systemd/system/container-blackhole-filter.service > /dev/null
sudo systemctl daemon-reload
sudo systemctl enable container-blackhole-filter.service
```

### 3A.6 Update active Proxy to route through this Filter

On the active Proxy:

```bash
sudo sed -i 's|proxy_pass http://<OLD_FILTER_IP>:80|proxy_pass http://<NEW_FILTER_WG7_IP>:80|g' /etc/nginx/conf.d/gateway.conf
sudo nginx -t && sudo nginx -s reload
```

### 3A.7 Firewall hardening

```bash
sudo firewall-cmd --add-rich-rule='rule family=ipv4 source address=<PROXY_WG7_IP>/32 port port=80 protocol=tcp accept' --permanent
sudo firewall-cmd --add-rich-rule='rule family=ipv4 source address=10.69.69.0/24 port port=9922 protocol=tcp accept' --permanent
sudo firewall-cmd --remove-service=http --permanent 2>/dev/null
sudo firewall-cmd --remove-service=https --permanent 2>/dev/null
sudo firewall-cmd --reload
```

### 3A.8 Sync vault key from Shield

```bash
VAULT_KEY=$(sudo cat /dev/shm/vault.key)
ssh -p 9922 -i /home/admin/.ssh/gabriella_key admin@<NEW_NODE_WG7_IP> \
    "echo '${VAULT_KEY}' | sudo tee /dev/shm/vault.key > /dev/null; sudo chmod 600 /dev/shm/vault.key"
```

### 3A.9 Update Shield dispatch

In /opt/svc/shield-receiver.py on Shield, update the filter entry in NODE_MAP:

```python
"filter": {"ip": "<NEW_NODE_WG7_IP>", "port": 9922},
```

Restart: `sudo systemctl restart shield-receiver`

---

## Step 3B: Proxy Role (5 min)

### 3B.1 Deploy cloudflare.conf

Create /etc/nginx/conf.d/cloudflare.conf with Cloudflare IP allowlist.
See System_Architecture.md section 16 for current IP list.

### 3B.2 Deploy gateway.conf

Key elements:
- SSL certificate and key paths
- proxy_pass to Filter WG IP:80 for both cleanpharma.cc and cleanearth.cc
- Honeypot proxy for bot paths (127.0.0.1:9999)
- include cloudflare.conf for IP allowlist

### 3B.3 Deploy SSL certificates

Copy from existing active Proxy or generate new:

```bash
sudo certbot certonly --nginx -d cleanpharma.cc -d cleanearth.cc
```

### 3B.4 Deploy honeypot container

```bash
sudo mkdir -p /var/log/honeypot
sudo podman run -d --name blackhole-proxy \
    -p 127.0.0.1:9999:80 \
    -v /var/log/honeypot:/var/log/honeypot:Z \
    --restart=always \
    docker.io/library/nginx:alpine
sudo podman generate systemd --name blackhole-proxy --new | sudo tee /etc/systemd/system/container-blackhole-proxy.service > /dev/null
sudo systemctl daemon-reload
sudo systemctl enable container-blackhole-proxy.service
```

### 3B.5 SELinux

```bash
sudo setsebool -P httpd_can_network_connect 1
```

### 3B.6 Firewall

```bash
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --add-rich-rule='rule family=ipv4 source address=10.69.69.0/24 port port=9922 protocol=tcp accept' --permanent
sudo firewall-cmd --reload
```

### 3B.7 Update DNS A records

On DNS node (margit, 10.69.69.5):

```bash
pdnsutil replace-rrset cleanearth.cc @ A 60 <NEW_PROXY_PUBLIC_IP>
pdnsutil replace-rrset cleanpharma.cc @ A 60 <NEW_PROXY_PUBLIC_IP>
pdns_control notify cleanearth.cc
pdns_control notify cleanpharma.cc
```

---

## Step 4: Heartbeat + Monitoring (3 min)

### 4.1 Deploy heartbeat

From Shield:

```bash
scp -P 9922 -i /home/admin/.ssh/gabriella_key admin@<EXISTING_NODE>:/opt/svc/heartbeat.sh /tmp/heartbeat.sh
sed -i 's/NODE_ID=.*/NODE_ID="<NEW_NODE_ID>"/' /tmp/heartbeat.sh
scp -P 9922 -i /home/admin/.ssh/gabriella_key /tmp/heartbeat.sh admin@<NEW_NODE_WG7_IP>:/tmp/heartbeat.sh
ssh -p 9922 -i /home/admin/.ssh/gabriella_key admin@<NEW_NODE_WG7_IP> \
    "sudo cp /tmp/heartbeat.sh /opt/svc/heartbeat.sh; sudo chmod 755 /opt/svc/heartbeat.sh"
```

### 4.2 Set up heartbeat cron

```bash
echo '*/1 * * * * root /opt/svc/heartbeat.sh' | sudo tee /etc/cron.d/heartbeat
```

### 4.3 Sync vault key

See Step 3A.8 above.

### 4.4 Verify heartbeat on Shield

```bash
curl -sf http://10.69.69.1:8401/api/status | python3 -m json.tool
```

---

## Step 5: Verification (2 min)

### 5.1 Traffic chain test

```bash
curl -sf -o /dev/null -w '%{http_code}' -H 'Host: cleanpharma.cc' http://<NEW_NODE_WG7_IP>:80
```

Expected: 200

### 5.2 Bot path test

```bash
curl -sf -o /dev/null -w '%{http_code}' -H 'Host: cleanpharma.cc' http://<NEW_NODE_WG7_IP>:80/wp-admin
```

Expected: 200 (from honeypot)

### 5.3 Unauthorized IP test

```bash
curl -sf -o /dev/null -w '%{http_code}' http://<NEW_NODE_PUBLIC_IP>:80/
```

Expected: 444 or connection refused

### 5.4 Full chain test

```bash
curl -sf -o /dev/null -w '%{http_code}' https://cleanpharma.cc/
curl -sf -o /dev/null -w '%{http_code}' https://cleanearth.cc/
```

Expected: 200

---

## Rollback Procedures

### Revert Filter Swap

1. On active Proxy: restore gateway.conf proxy_pass to original Filter IP
2. On Shield: revert NODE_MAP in shield-receiver.py, restart
3. On standby node: rename waf.conf to waf.conf.disabled, reload nginx

### Revert Proxy Swap

1. On DNS node: restore A records to original Proxy IP
2. On standby node: disable gateway.conf and SSL
3. On Shield: revert dispatch routing

---

## Quick Reference Card

| Action | Filter | Proxy |
|---|---|---|
| nginx config | waf.conf | gateway.conf + cloudflare.conf |
| Honeypot port | 9998 | 9999 |
| Upstream | Shield 10.69.69.1:80 | Filter WG IP:80 |
| Accepts from | Proxy WG IP only | Cloudflare IPs + Shield + WG |
| SELinux | httpd_can_network_connect=1 | httpd_can_network_connect=1 |
| DNS update | Not needed | Required (A records) |
| Shield dispatch | Update filter entry | Update proxy entry |
| SSL | Not needed | Required (certbot) |

---

*Last updated: 2026-06-15*
*Version: 1.0*
