# System Architecture - Clean Logistics Multi-Store Platform

> **CRITICAL**: Read this document BEFORE modifying ANY system. If a system already exists, DO NOT rebuild it. Fix it.
> This document was last updated **2026-06-19** (v4.1) - Full platform cleanup pass: all .bak files removed (Origin + all fleet nodes), inline onclick handlers converted to addEventListener, // comments stripped from 24 PHP files, mysql_root.key secured (root:root 400), backups_handler hardcoded password removed, orphaned DB views/tables dropped (nn_* views + cart_items + audit_log), documentation gaps resolved (cred.md SSH port, NTK timestamp, rules.md collision).

---

## 0. Brand Architecture

This platform manages multiple storefronts under the **Clean** brand family.

| Brand            | Domain           | Role                                                  | Status  |
| ---------------- | ---------------- | ----------------------------------------------------- | ------- |
| **Clean Earth**  | `cleanearth.cc`  | Original store - supplements, peptides, wellness      | Active  |
| **Clean Pharma** | `cleanpharma.cc` | Second storefront - pharma-grade supplements          | Active  |
| **Clean Supply** | `cleansupply.cc` | Third store - dropshipping / Amazon pipeline          | Planned |
| **NoName**       | admin dashboard  | Multi-store eCommerce management system (this system) | Active  |

> The admin dashboard is branded **NoName eCommerce Management System**. Source code is completely brand-agnostic - store names (`store_name`, `tagline`, `store_email`) are encrypted at rest in the `stores` table via `SecureVault::encrypt()` (AES-256-GCM) and decrypted at runtime via `Crypt::render()` in `init.php`. Platform labels use `NoName` - never CleanPharma, CleanEarth, Clean Logistics, or any store-specific name.
>
> **DNS/domain management endpoint**: `cleanearth.cc` - this is the authoritative DNS zone managed by Node 001 (PowerDNS). Failover writes target this zone first.
>
> **Backup download endpoint**: `dl.cleanpharma.cc` - hosted on Node 001, used for the backup pipeline temporary download links.

---

## 0.1. Payment Methods (Current)

| Method                                 | Status       | Processor                      | Notes                                                                           |
| -------------------------------------- | ------------ | ------------------------------ | ------------------------------------------------------------------------------- |
| Stripe (credit/debit card)             | **Active**   | Stripe live keys loaded        | Primary payment method                                                          |
| Interac e-Transfer                     | **Active**   | Engine email matching          | Canadian customers                                                              |
| Cryptocurrency (BTC/ETH/USDC/USDT/XRP) | **Disabled** | Coinbase Commerce bridge ready | `/var/www/private/coinbase_bridge.php` created; add API key to `.env` to enable |
| Monero (XMR)                           | **Removed**  | -                              | Removed 2026-06-16; privacy coin, not appropriate                               |
| TRON / USDT-TRC20                      | **Removed**  | -                              | Removed 2026-06-16 with Monero                                                  |

> When re-enabling crypto: set `COINBASE_COMMERCE_API_KEY` in `/var/www/private/.env`.

---

## 1. Directory Structure

```
Y:\5\ -> /var/www/ (ORIGIN server, Node 005 / 558201)
| -- dashboard/ # Admin dashboard (authenticated, HTTPS :8443)               |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- index.php # Single entry point, CSP, nav, tab router                                      |                                                                                                          |                                                                                                                                        |
|                                                                            | -- login.php # Admin authentication                                                          |                                                                                                          |                                                                                                                                        |
|                                                                            | -- logout.php # Session destroy                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- core/ # Core systems                                                                      |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- bootstrap.php # Loads shared libs, defines constants                                                  |                                                                                                                                        |
|                                                                            |                                                                                              | -- action_chain.php # Transactional CRUD with auto field encryption                                      |                                                                                                                                        |
|                                                                            |                                                                                              | -- tabs.php # Tab registry and routing                                                                   |                                                                                                                                        |
|                                                                            |                                                                                              | -- auth_chain.php # Session auth, rotation, idle check                                                   |                                                                                                                                        |
|                                                                            |                                                                                              | -- rbac.php # Role-based access control                                                                  |                                                                                                                                        |
|                                                                            |                                                                                              | -- csrf.php # CSRF token generation/enforcement                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- events.php # Audit event recorder                                                                     |                                                                                                                                        |
|                                                                            |                                                                                              | -- icons.php # SVG icon library (40+ custom methods, NO emoji)                                           |                                                                                                                                        |
|                                                                            | +-- ui.php # UI helpers (renderIcon, alert, e())                                             |                                                                                                          |                                                                                                                                        |
|                                                                            | -- tabs/ # Tab routers                                                                       |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- store.php # Store tab -> modules/store/*                                                              |                                                                                                                                        |
|                                                                            |                                                                                              | -- topology.php # Topology canvas                                                                        |                                                                                                                                        |
|                                                                            |                                                                                              | -- fleet.php # Fleet control                                                                             |                                                                                                                                        |
|                                                                            |                                                                                              | -- tools.php # Node tools (SVG icons, decrypted labels)                                                  |                                                                                                                                        |
|                                                                            |                                                                                              | -- mesh_security.php # Mesh security (SecureVault decrypted)                                             |                                                                                                                                        |
|                                                                            |                                                                                              | -- backups.php # Full backup pipeline tab                                                                |                                                                                                                                        |
|                                                                            | +-- ... # Other tabs                                                                         |                                                                                                          |                                                                                                                                        |
|                                                                            | -- modules/                                                                                  |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- store/                                                                                                |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          | -- listings.php # PRODUCT SYSTEM (THE ONLY ONE)                                                                                        |
|                                                                            |                                                                                              |                                                                                                          | -- store_config.php # Store configuration                                                                                              |
|                                                                            |                                                                                              |                                                                                                          | -- store_data.php # Store variable setup                                                                                               |
|                                                                            |                                                                                              |                                                                                                          | -- blocks.php # Storefront block builder                                                                                               |
|                                                                            |                                                                                              | +-- row_builder.php # Row layout builder                                                                 |                                                                                                                                        |
|                                                                            |                                                                                              | -- fleet/                                                                                                |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          | -- topology_map.php # Network topology canvas                                                                                          |
|                                                                            |                                                                                              | +-- fleet_control.php                                                                                    |                                                                                                                                        |
|                                                                            |                                                                                              | -- backups/                                                                                              |                                                                                                                                        |
|                                                                            |                                                                                              | +-- backups_handler.php # DB dump + archive + delete pipeline                                            |                                                                                                                                        |
|                                                                            | +-- ...                                                                                      |                                                                                                          |                                                                                                                                        |
|                                                                            | -- assets/                                                                                   |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- css/                                                                                                  |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          | -- dashboard.css # Master stylesheet (+1726 lines; includes extracted Rule 20 classes, modal-view, topo-*, ep-*, auth utility classes) |
|                                                                            |                                                                                              |                                                                                                          | -- auth.css # Login page styles (auth.css, 9.5 KB - covers login.php through login_totp.php)                                           |
|                                                                            |                                                                                              |                                                                                                          | -- credentials.css # Credentials tab styles                                                                                            |
|                                                                            |                                                                                              |                                                                                                          | -- emails.css # Email module styles                                                                                                    |
|                                                                            |                                                                                              |                                                                                                          | -- infrastructure.css                                                                                                                  |
|                                                                            |                                                                                              |                                                                                                          | -- logs-live.css                                                                                                                       |
|                                                                            |                                                                                              |                                                                                                          | -- mfa-setup.css                                                                                                                       |
|                                                                            |                                                                                              |                                                                                                          | -- money.css                                                                                                                           |
|                                                                            |                                                                                              |                                                                                                          | -- safety.css                                                                                                                          |
|                                                                            |                                                                                              | +-- topology.css                                                                                         |                                                                                                                                        |
|                                                                            | +-- js/                                                                                      |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- dashboard.js # Core dashboard JS                                                                      |                                                                                                                                        |
|                                                                            | +-- topology.js # Topology canvas renderer                                                   |                                                                                                          |                                                                                                                                        |
|                                                                            | -- system/ # POST action handlers                                                            |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- post.php # Generic create                                                                             |                                                                                                                                        |
|                                                                            |                                                                                              | -- modify.php # Generic update                                                                           |                                                                                                                                        |
|                                                                            |                                                                                              | -- verify.php # Password re-verification (zero-trust)                                                    |                                                                                                                                        |
|                                                                            | +-- delete.php # Generic delete                                                              |                                                                                                          |                                                                                                                                        |
| +-- api/                                                                   |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- mesh-api.php # Mesh security API                                                          |                                                                                                          |                                                                                                                                        |
|                                                                            | -- commands.php # Fleet command queue API                                                    |                                                                                                          |                                                                                                                                        |
|                                                                            | -- heartbeat.php # Heartbeat receiver API                                                    |                                                                                                          |                                                                                                                                        |
|                                                                            | -- fleet/topology.php # Topology CRUD API                                                    |                                                                                                          |                                                                                                                                        |
| +-- payments/etransfer_webhook.php # E-transfer auto-matching              |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          |                                                                                                                                        |
| -- public/ # Public-facing storefront (customer, HTTPS :443)               |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- pages/ # Page templates (12 files)                                                        |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- home.php # Product catalog / storefront                                                               |                                                                                                                                        |
|                                                                            |                                                                                              | -- product.php # Individual product detail                                                               |                                                                                                                                        |
|                                                                            |                                                                                              | -- checkout.php # Full checkout (3 payment tabs)                                                         |                                                                                                                                        |
|                                                                            |                                                                                              | -- payment.php # Post-payment confirmation/tracking                                                      |                                                                                                                                        |
|                                                                            |                                                                                              | -- track.php # Order tracking page                                                                       |                                                                                                                                        |
|                                                                            |                                                                                              | -- account.php # User account portal                                                                     |                                                                                                                                        |
|                                                                            |                                                                                              | -- reviews.php # Product reviews                                                                         |                                                                                                                                        |
|                                                                            |                                                                                              | -- faq.php # FAQ page                                                                                    |                                                                                                                                        |
|                                                                            |                                                                                              | -- support.php # Support contact                                                                         |                                                                                                                                        |
|                                                                            |                                                                                              | -- privacy.php # Privacy policy                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- terms.php # Terms of service                                                                          |                                                                                                                                        |
|                                                                            | +-- exclude.php # Self-exclusion                                                             |                                                                                                          |                                                                                                                                        |
|                                                                            | -- modules/                                                                                  |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- cart-drawer.php # Slide-out cart drawer HTML                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- welcome.php # Welcome/age-gate module                                                                 |                                                                                                                                        |
|                                                                            |                                                                                              | -- account/ # Account sub-modules                                                                        |                                                                                                                                        |
|                                                                            | +-- checkout/ # Checkout sub-modules (10 files)                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- block_agree.php, block_cart.php, block_credit.php                                                     |                                                                                                                                        |
|                                                                            |                                                                                              | -- block_etransfer.php, block_shipping.php                                                               |                                                                                                                                        |
|                                                                            |                                                                                              | -- block_summary.php, etransfer.php, stripe.php                                                          |                                                                                                                                        |
|                                                                            | +-- (block_crypto.php removed 2026-06-16 - crypto payment disabled)                          |                                                                                                          |                                                                                                                                        |
|                                                                            | -- system/ # Public bootstrap & utilities (16 files)                                         |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- init.php # Bootstrap: sessions, DB, encryption, auth, CSRF, store                                     |                                                                                                                                        |
|                                                                            |                                                                                              | -- App.php # Core utility (ok/err, bridgeSend, rate limit)                                               |                                                                                                                                        |
|                                                                            |                                                                                              | -- payment_bridge.php # PaymentBridge class (HMAC to ENGINE)                                             |                                                                                                                                        |
|                                                                            |                                                                                              | -- email_bridge.php # Email sending bridge                                                               |                                                                                                                                        |
|                                                                            |                                                                                              | -- store_secrets.php # StoreSecrets (wallet addresses, stripe keys)                                      |                                                                                                                                        |
|                                                                            |                                                                                              | -- auth.php, config.php, csrf.php, db.php, encrypt.php                                                   |                                                                                                                                        |
|                                                                            |                                                                                              | -- security.php, obfuscate.php, telemetry.php, trust.php                                                 |                                                                                                                                        |
|                                                                            |                                                                                              | -- labels.php, image.php                                                                                 |                                                                                                                                        |
|                                                                            | -- api/ # Public API endpoints                                                               |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- index.php # Main API router (route-based: group.action)                                               |                                                                                                                                        |
|                                                                            |                                                                                              | -- router.php # SPA page router (returns HTML + CSRF)                                                    |                                                                                                                                        |
|                                                                            |                                                                                              | -- purchase-bridge.php # Legacy order creation endpoint                                                  |                                                                                                                                        |
|                                                                            |                                                                                              | -- support-bridge.php # Support ticket bridge                                                            |                                                                                                                                        |
|                                                                            |                                                                                              | -- welcome_beacon.php # Welcome tracking                                                                 |                                                                                                                                        |
|                                                                            |                                                                                              | -- create_orders.sql # Order schema                                                                      |                                                                                                                                        |
|                                                                            | +-- handlers/ # API handlers (13 files)                                                      |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              | -- order.handler.php # Order placement (place_order, track, finalize)                                    |                                                                                                                                        |
|                                                                            |                                                                                              | -- checkout.handler.php # Payment processing - calls PaymentBridge::submit(); order prefix NN- (was CE-) |                                                                                                                                        |
|                                                                            |                                                                                              | -- callback.handler.php # Payment webhooks (succeeded/failed/crypto/etransfer)                           |                                                                                                                                        |
|                                                                            |                                                                                              | -- auth.handler.php, account.handler.php, product.handler.php                                            |                                                                                                                                        |
|                                                                            |                                                                                              | -- review.handler.php, support.handler.php, trust.handler.php                                            |                                                                                                                                        |
|                                                                            |                                                                                              | -- telemetry.handler.php, security.handler.php                                                           |                                                                                                                                        |
|                                                                            |                                                                                              | -- page.handler.php, render.handler.php                                                                  |                                                                                                                                        |
| +-- assets/                                                                |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- css/ # Public stylesheets                                                                 |                                                                                                          |                                                                                                                                        |
| +-- js/ # Client-side JS (22 files)                                        |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- bridge.js # Core API client (post, purchase, csrf, formatMoney)                           |                                                                                                          |                                                                                                                                        |
|                                                                            | -- cart.js # Cart IIFE (localStorage-based, 220 lines)                                       |                                                                                                          |                                                                                                                                        |
|                                                                            | -- product-cards.js # Product card interactions (pills, qty, add-to-cart)                    |                                                                                                          |                                                                                                                                        |
|                                                                            | -- router.js # SPA router                                                                    |                                                                                                          |                                                                                                                                        |
|                                                                            | -- navigation.js # Nav active state                                                          |                                                                                                          |                                                                                                                                        |
|                                                                            | -- category-nav.js # Category navigation                                                     |                                                                                                          |                                                                                                                                        |
|                                                                            | -- checkout/checkout.js # Full checkout flow (485 lines)                                     |                                                                                                          |                                                                                                                                        |
|                                                                            | -- checkout/payment.js # Alt/legacy checkout                                                 |                                                                                                          |                                                                                                                                        |
|                                                                            | -- track-events.js # Order tracking UI                                                       |                                                                                                          |                                                                                                                                        |
|                                                                            | -- telemetry.js # Event telemetry                                                            |                                                                                                          |                                                                                                                                        |
|                                                                            | -- watchdog.js # Session watchdog                                                            |                                                                                                          |                                                                                                                                        |
|                                                                            | -- anti-scrape.js # Anti-scraping                                                            |                                                                                                          |                                                                                                                                        |
|                                                                            | -- screenshot-protection.js                                                                  |                                                                                                          |                                                                                                                                        |
|                                                                            | -- fingerprint.js # Browser fingerprinting                                                   |                                                                                                          |                                                                                                                                        |
|                                                                            | -- fraud-detect.js # Fraud detection                                                         |                                                                                                          |                                                                                                                                        |
| +-- ... (product-events, account-events, reviews-events, faq-events, etc.) |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          |                                                                                                                                        |
| -- shared/ # Shared libraries (dashboard + public)                         |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- encryption.php # SecureVault (sodium secretbox, domain subkeys)                           |                                                                                                          |                                                                                                                                        |
|                                                                            | -- db.php # Database class (PDO wrapper, AUTO-DECRYPT on fetch)                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- credentials.php # AES-256-GCM credential vault                                            |                                                                                                          |                                                                                                                                        |
|                                                                            | -- config_loader.php # DB/encryption config loader                                           |                                                                                                          |                                                                                                                                        |
|                                                                            | -- image_processor.php # Image upload, EXIF strip, resize                                    |                                                                                                          |                                                                                                                                        |
|                                                                            | -- public_id.php # Public ID generation                                                      |                                                                                                          |                                                                                                                                        |
|                                                                            | -- request_signer.php # HMAC request signing for inter-node                                  |                                                                                                          |                                                                                                                                        |
| +-- status_codes.php # All status codes (order: 2001-2010, etc.)           |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          |                                                                                                                                        |
| -- private/ # Secrets (NEVER in version control)                           |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- .env # (DEPRECATED) Only contains `sk` master secret key                                  |                                                                                                          |                                                                                                                                        |
|                                                                            | -- credentials.enc # AES-256-GCM encrypted JSON vault (pii_key, db_pass, stripe_pk, wallets) |                                                                                                          |                                                                                                                                        |
| +-- (other plaintexts) # Deprecated, fully migrated to credentials.enc     |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          |                                                                                                                                        |
| -- engine-scripts/ # S3RV1C3agent + node tools                             |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- s3rv1c3agent # Main agent daemon (bash, 418 lines, 10s poll loop)                         |

> **Updated 2026-07-02**: The s3rv1c3agent Bash engine has been replaced by the compiled C++ Engine (669427-engine). The architectural behavior described here remains accurate but the implementation is now C++, not Bash.                                                                                                          |                                                                                                                                        |
|                                                                            | -- mailctl # Agent variant (406 lines)                                                       |                                                                                                          |                                                                                                                                        |
|                                                                            | -- se_client.sh # Shared HTTP/HMAC client library                                            |                                                                                                          |                                                                                                                                        |
|                                                                            | -- svcbeat # Heartbeat sender                                                                |                                                                                                          |                                                                                                                                        |
|                                                                            | -- svcboot # Node bootstrap (tmpfs + WireGuard)                                              |                                                                                                          |                                                                                                                                        |
|                                                                            | -- hello # 3-phase node initialization                                                       |                                                                                                          |                                                                                                                                        |
|                                                                            | -- paymail # Payment status checker tool                                                     |                                                                                                          |                                                                                                                                        |
|                                                                            | -- cmdmail # Command receiver tool                                                           |                                                                                                          |                                                                                                                                        |
|                                                                            | -- dbwrite # DB writer (parameterized)                                                       |                                                                                                          |                                                                                                                                        |
|                                                                            | -- deploy.sh # Deployment script                                                             |                                                                                                          |                                                                                                                                        |
|                                                                            | -- engine.json # Engine config                                                               |                                                                                                          |                                                                                                                                        |
| +-- sysd-engine.service # Systemd unit for C++ daemon                      |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          |                                                                                                                                        |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|                                                                            | -- config/engine.json # Engine daemon config                                                 |                                                                                                          |                                                                                                                                        |
|                                                                            | -- firmware/ # Node firmware                                                                 |                                                                                                          |                                                                                                                                        |
|                                                                            | -- scripts/ # Engine scripts                                                                 |                                                                                                          |                                                                                                                                        |
| +-- watch/ # File watchers                                                 |                                                                                              |                                                                                                          |                                                                                                                                        |
|                                                                            |                                                                                              |                                                                                                          |                                                                                                                                        |
+-- uploads/products/ # Product images (obfuscated filenames)
```

---

## 2. Network Topology - 8 Nodes (All Active)

### Node Inventory

| Node  | ID     | Role    | Cover Name   | WG IP      | Public IP      | SSH Port | Location  | Status                             |
| ----- | ------ | ------- | ------------ | ---------- | -------------- | -------- | --------- | ---------------------------------- |
| 001   | 199335 | DNS     | margit       | 10.69.69.5 | 80.78.19.57    | 9922     | Sweden    | Active                             |
| 002   | 244939 | Proxy   | srv244939.hostlocal.net | 10.69.69.3 | 107.189.23.194 | 9922     | Amsterdam | Active                             |
| 003   | 333941 | Filter  | filter       | 10.69.69.4 | 91.229.239.29  | 9922     | Moldova   | Active                             |
| 004   | 414342 | Shield  | vps-82a41f.hosted-by.app | 10.69.69.1 | 194.36.190.111 | 9922     | Amsterdam | Active (HUB)                       |
| 005   | 558201 | Origin  | customer-558201.1984.is | 10.69.69.2 | 89.127.235.207 | 9922     | Iceland   | Active                             |
| 006   | 669427 | Engine  | srv1653421.hstgr.cloud | 10.69.69.6 | 76.13.222.157  | 9922     | Malaysia  | Active                             |
| 007   | 773816 | Email   | -            | -          | 76.13.222.157  | -        | Malaysia  | DECOMMISSIONED (2026-06-19)        |
| 008   | 869377 | Vault   | vps-c91b28.cloudzy.com | 10.69.69.8 | 104.194.144.66 | 9922     | London    | Active — Credential vault          |
| 009   | 714820 | Payment | srv156742.hosttoname.com | 10.69.69.7 | 45.143.166.118 | 9922     | Holland   | Active — Payment processing        |

> [!IMPORTANT]
> 8 nodes. 8 six-digit IDs. All active. No legacy entries exist in the database or documentation.

### Zero-Trust Traffic Chain (MANDATORY)

```
Internet -> Cloudflare DNS
 |
 v
Proxy nginx (cloudflare.conf: CF IPs + Shield + WG subnet ONLY -> deny all)
 | WireGuard tunnel shale (255.17.69.x/30)
 v
Filter nginx (geo $allowed_proxy - 255.17.69.4/30 ONLY)
 | WAF rules, bot detection, rate limiting, per-domain config
 | proxy_pass http://10.69.69.1:80 (wg7 mesh)
 v
Shield nginx
 | proxy_pass http://10.69.69.2:80 (wg7 mesh)
 v
Origin (serves public storefront + dashboard)
```

> [!CAUTION]
> **YOU MUST route: Proxy -> Filter -> Shield -> Origin**
> Direct Proxy -> Origin is FORBIDDEN (Rule 30).
> Direct Proxy -> Shield (bypassing Filter) is FORBIDDEN.

### WireGuard Tunnels

| Tunnel          | Interface | Subnet         | Port  | Purpose                      | Nodes           | Status   |
| --------------- | --------- | -------------- | ----- | ---------------------------- | --------------- | -------- |
| wg7 mesh        | wg7       | 10.69.69.0/24  | 41820 | Hub-and-spoke (Shield hub)   | ALL             | ACTIVE   |
| Proxy<->Filter  | shale   | 255.17.69.4/30 | 49616 | Live traffic Proxy->Filter   | 002, 003        | ACTIVE   |
| Engine bridge   | lantern   | 172.28.41.0/30 | 39471 | Origin<->Engine direct link  | 005, 006        | ACTIVE   |
| Rain (vault)    | rain      | 172.23.83.0/30 | 34800 | Shield<->Engine vault relay  | 004, 006        | ACTIVE   |
| Gravel (dl)     | gravel    | 172.29.11.0/30 | 37747 | Engine<->DNS download pipe   | 006, 001        | ACTIVE   |
| Umbrella        | umbrella  | 10.83.47.0/30  | 38471 | Engine<->Shield side channel | 004, 006        | ACTIVE   |
| Auth challenge  | compass   | 172.28.47.0/28 | 42783 | Challenge-response to Vault  | ALL, 007        | ACTIVE   |
| AI inference    | thread    | 172.28.45.0/30 | 44619 | AI inference pipeline        | 005, 006        | ACTIVE   |
| Store 1         | pebble    | 10.69.70.0/30  | 39847 | Store 1 traffic              | 002, 005        | ACTIVE   |
| Store 2         | timber    | 10.69.71.0/30  | 40153 | Store 2 traffic              | 002, 005        | ACTIVE   |
| DNS zones       | fossil    | 172.29.13.0/30 | 43291 | DNS zone management          | 005, 001        | ACTIVE   |

### Shield Relay Rule (Rule 30)
All inter-node commands to/from Origin (558201) MUST route through Shield (414342):
```
Proxy(244939) -> Filter(333941) -> Shield(414342) -> Origin(558201) CORRECT
Proxy(244939) -> Origin(558201) FORBIDDEN
Proxy(244939) -> Shield(414342) -> Origin(558201) (bypass Filter) FORBIDDEN
```

### Heartbeat-Based Failover

All 8 nodes run `svc-heartbeat.service` reporting to Shield (:8401) every 30s.
Vault lockdown triggers after 3 consecutive missed Vault pings (admin sudo disabled, auto-recovery).

|                     | Normal                      | On Proxy Failure                        |
| ------------------- | --------------------------- | --------------------------------------- |
| DNS A record        | `107.189.23.194` (Proxy)    | DNS failover to maintenance page        |
| Filter upstream     | `255.17.69.5` (Proxy WG)    | Filter blocks — maintenance response    |
| TTL during failover | 300s                        | 60s (auto during failover)              |

### Credential Rotation Engine

`/opt/apache/engine/tools/credential_rotation.sh` on Shield. Centralized credential rotation triggered by heartbeat failure detection or manual invocation.

| Action | Command | Effect |
| --- | --- | --- |
| Rotate single node | `credential_rotation.sh rotate <node_id>` | Rotates admin+root passwords, MySQL password |
| Rotate all nodes | `credential_rotation.sh rotate all` | Full fleet rotation (excludes admin.ppk) |
| Vault lockdown | `credential_rotation.sh vault-lockdown` | Disables admin sudo on ALL non-vault nodes |
| Vault restore | `credential_rotation.sh vault-restore` | Re-enables admin sudo fleet-wide |
| Node lockdown | `credential_rotation.sh lockdown <node_id>` | Disables admin sudo on single node |
| Node unlock | `credential_rotation.sh unlock <node_id>` | Re-enables admin sudo on single node |

Rotation events logged to `sec_key_rotations`, `sec_password_rotations`, and `sec_lockdown_log` tables on Origin DB.

### hello.sh — Vault Challenge-Response Gate

`/opt/svc/hello.sh` deployed on all 8 nodes. Provides vault connectivity verification and self-lockdown.

| Mode | Trigger | Effect |
| --- | --- | --- |
| `check` | Each heartbeat cycle (30s) | TCP ping to Vault (10.69.69.8). On failure: disables sudo + password auth. On recovery: auto-restores. |
| `challenge` | Sensitive operations | Full HMAC challenge-response using `/dev/shm/vault.key` against Vault HTTPS endpoint |
| `status` | Manual query | Reports vault reachability and current lockdown state |

**Lockdown cascade**: Node heartbeat.sh → `vault_check()` (ICMP ping, 3 miss threshold) → `hello.sh check` (TCP handshake, immediate) → sudo disabled + password auth disabled.

### Heartbeat Watchdog (Centralized)

`/opt/apache/engine/tools/heartbeat_watchdog.sh` on Shield, cron every 60s (`/etc/cron.d/heartbeat-watchdog`).

**Function**: Queries `nodes.last_heartbeat` on Origin DB. If any node's heartbeat is stale (>120s), increments `consecutive_misses`. On 3 consecutive misses, triggers lockdown via `credential_rotation.sh`. Auto-restores on recovery (resets `consecutive_misses` to 0).

**Vault-specific**: If Vault (869377) goes stale, triggers `vault-lockdown` (all nodes lose sudo). On recovery, triggers `vault-restore`.

---

### WireGuard Subnet Audit - 2026-06-19

> [!IMPORTANT]
> Full live audit conducted 2026-06-19. All nodes accessed via Origin fleet key. Results below reflect real-time state.

#### Active Interfaces per Node

| Node           | Interface | WG IP            | Subnet           | Purpose                           | Status |
| -------------- | --------- | ---------------- | ---------------- | --------------------------------- | ------ |
| DNS (001)      | wg7       | 10.69.69.5/24    | 10.69.69.0/24    | Main mesh (via Shield)            | ACTIVE |
| DNS (001)      | gravel    | 172.29.11.1/30   | 172.29.11.0/30   | DNS<->Engine download pipe        | ACTIVE |
| Proxy (002)    | wg7       | 10.69.69.3/24    | 10.69.69.0/24    | Main mesh                         | ACTIVE |
| Proxy (002)    | shale   | 255.17.69.5/30   | 255.17.69.4/30   | Proxy<->Filter traffic            | ACTIVE |
| Filter (003)   | wg7       | 10.69.69.4/24    | 10.69.69.0/24    | Main mesh                         | ACTIVE |
| Filter (003)   | shale   | 255.17.69.6/30   | 255.17.69.4/30   | Filter<->Proxy traffic            | ACTIVE |
| Shield (004)   | wg7       | 10.69.69.1/24    | 10.69.69.0/24    | Hub - all spokes                  | ACTIVE |
| Shield (004)   | rain      | 172.23.83.1/30   | 172.23.83.0/30   | Shield<->Engine vault relay       | ACTIVE |
| Shield (004)   | umbrella  | 10.83.47.1/30    | 10.83.47.0/30    | Shield<->Engine side channel      | ACTIVE |
| Origin (005)   | wg7       | 10.69.69.2/24    | 10.69.69.0/24    | Main mesh                         | ACTIVE |
| Origin (005)   | lantern   | 172.28.41.1/30   | 172.28.41.0/30   | Origin<->Engine direct            | ACTIVE |
| Engine (006)   | wg7       | 10.69.69.6/24    | 10.69.69.0/24    | Main mesh                         | ACTIVE |
| Engine (006)   | lantern   | 172.28.41.2/30   | 172.28.41.0/30   | Engine<->Origin direct            | ACTIVE |
| Engine (006)   | rain      | 172.23.83.2/30   | 172.23.83.0/30   | Engine<->Shield vault             | ACTIVE |
| Engine (006)   | gravel    | 172.29.11.2/30   | 172.29.11.0/30   | Engine<->DNS download             | ACTIVE |
| Engine (006)   | umbrella  | 10.83.47.2/30    | 10.83.47.0/30    | Engine<->Shield side channel      | ACTIVE |
| Vault (008)    | wg7       | 10.69.69.8/24    | 10.69.69.0/24    | Main mesh                         | ACTIVE |
| Payment (009)  | wg7       | 10.69.69.7/24    | 10.69.69.0/24    | Main mesh                         | ACTIVE |

**Shield wg7 peer count: 8** (Origin .2, Proxy .3, Filter .4, DNS .5, Engine .6, Payment .7, Vault .8, admin .10)

---


## 3. Encryption System (SecureVault)

**Class**: `SecureVault` (aliased as `Encryption` and `Crypt`)
**File**: `Y:\5\shared\encryption.php`
**Algorithm**: libsodium `crypto_secretbox` (XSalsa20-Poly1305)
**Key Derivation**: `sodium_crypto_kdf_derive_from_key` from 256-bit master key (KDF context: `cleanpii`)
**Key Source**: `/var/www/private/.env` -> `credentials.enc` (AES-256-GCM) -> `pii_key`
**Max Size**: **UNLIMITED** - sodium_crypto_secretbox has no practical limit.

### Domain Subkeys
| ID  | Domain  | Usage                                   |
| --- | ------- | --------------------------------------- |
| 0   | default | General encryption                      |
| 1   | user    | User PII (email, name, phone, address)  |
| 2   | order   | Order data, tracking numbers, addresses |
| 3   | review  | Customer reviews                        |
| 4   | log     | Audit logs                              |
| 5   | payment | Payment data, wallet addresses          |
| 6   | faq     | FAQ content                             |
| 7   | product | Product titles, prices, descriptions    |

### Key Methods
```php
Encryption::encrypt($plaintext, $domain) // -> base64 ciphertext
Encryption::decrypt($encoded) // -> plaintext (auto-detects domain tag)
Encryption::render($value) / ::d($value) // -> decrypt or return as-is
Encryption::e($value) // -> decrypt + htmlspecialchars
Encryption::isEncrypted($value) // -> bool
Encryption::encryptFields($data, $fields, $domain)
Encryption::decryptFields($data, $fields)
```

### Auto-Decrypt in Database
`Database::$autoDecrypt = true` - ALL fetched values are auto-decrypted if they match encrypted format.

> [!IMPORTANT]
> `bootstrap.php` explicitly sets `Database::$autoDecrypt = false` in the dashboard.
> All tab modules must call `SecureVault::render()` explicitly to decrypt values for display.

### Engine Vault System (AES-256-CBC)
Separate vault for engine tools on Node 006:
- **Encrypted tools**: `/opt/apache/engine/vault/encrypted/*.vault`
- **Decryption key**: On SHIELD at `/opt/apache/engine/config/.k7x9m2/v.dat`
- **Active vault files**: `stripe.vault`, `origin_mysql.vault`, `gmail_address.vault`, `gmail_etransfer.vault`
- **Removed vault files** (2026-06-16): `tron_privkey.vault`, `trongrid_api.vault`, `tron_address.vault`, `xmr_wallet_pass.vault`, `xmr_rpc_login.vault`, `xmr_address.vault`, `xmr_viewkey.vault`

### Engine Services - Node 006 (2026-06-16 state)
| Service             | Port                    | Status       | Notes                                                 |
| ------------------- | ----------------------- | ------------ | ----------------------------------------------------- |
| `669427-engine`     | 127.0.0.1:8301          | Active       | C++ payment/email daemon                              |
| `sysd-engine`       | 127.0.0.1:8302          | Active       | Security heartbeat agent (moved from 8301)            |
| `pay-relay`         | 172.28.41.2:8443        | Active       | HMAC gateway; DNAT: 10.69.69.6:8443->172.28.41.2:8443 |
| `postfix`           | 127.0.0.1:25/587        | Active       | Outbound SMTP; from: noreply@cleanearth.cc            |
| `dovecot`           | 0.0.0.0:993/995/110/143 | Active       | IMAP/POP3 mail storage                                |
| `monero-wallet-rpc` | -                       | **Disabled** | Removed 2026-06-16                                    |
| `tron-signer`       | -                       | **Disabled** | Removed 2026-06-16                                    |

---

## 4. Product System (SINGLE FILE)

> [!CAUTION]
> There is ONE product management file: `listings.php`.
> `products.php` was DELETED. DO NOT recreate it. EVER.

**File**: `Y:\5\dashboard\modules\store\listings.php`
**Router**: `Y:\5\dashboard\tabs\store.php` -> default case loads `listings.php`

### Product Data Hierarchy
```
Product (products table)
 |-- title / name -> Encrypted (domain=product)
 |-- base_price -> Encrypted (domain=product)
 |-- notes / description -> Encrypted (domain=product)
 |-- unit_label -> Encrypted (domain=product)
 |-- variant_label -> Encrypted (domain=product)
 |-- image_url / image_path -> Non-sensitive (public URL)
 |-- sku, slug, status -> Non-sensitive metadata
 |
 +-- Quality Variants (product_qualities table)
 |-- quality_name -> Encrypted (domain=product)
 |-- base_price -> Encrypted (domain=product)
 |-- image_path -> Non-sensitive
 |-- display_order
 |
 +-- Packaging Options (product_units table)
 |-- unit_amount -> Encrypted (e.g. "1 Unit", "5 Pack")
 |-- unit_price -> Encrypted (overrides variant price)
 +-- display_order
```

### Actions
| Action         | Route                                        | Purpose                             |
| -------------- | -------------------------------------------- | ----------------------------------- |
| `list`         | `?view=store&id=X`                           | Product card grid (store dashboard) |
| `new_product`  | `?view=store&id=X&action=new_product`        | Create product form                 |
| `edit_product` | `?view=store&id=X&action=edit_product&pid=Y` | Edit product form                   |
| `delete`       | POST with `listing_action=delete`            | Delete + cascade                    |
| `quick_upload` | POST with `api&action=quick_upload`          | AJAX image upload                   |

---

## 5. Customer Cart System

> [!IMPORTANT]
> Cart is 100% client-side localStorage. NO server-side cart. NO session cart. NO database cart.

**File**: `Y:\5\public\assets\js\cart.js` (220 lines)
**Storage**: `localStorage.getItem('cart')` / `setItem('cart', JSON.stringify(items))`

### Price Flow (Product -> Cart -> Checkout)
```
DB (encrypted) -> home.php (decrypts server-side) -> HTML data-attributes (plaintext floats)
 -> product-cards.js reads data attributes -> Cart.add() stores in localStorage
 -> checkout.js reads from localStorage -> Shows to customer
 -> On submit: cart sent to order.handler.php
 -> SERVER RE-VALIDATES ALL PRICES from DB (does NOT trust client prices)
```

---

## 6. Checkout & Payment Flow

### Payment Methods
| Method         | Provider        | How                                           |
| -------------- | --------------- | --------------------------------------------- |
| Credit/Prepaid | Stripe Elements | Card -> Stripe PaymentMethod -> PaymentIntent |
| E-Transfer     | Interac         | Customer sends to email, provides reference   |

### Order Placement Flow
```
1. Customer fills checkout form
2. checkout.js validates: cart, agreements (3 checkboxes), shipping info
3. For Stripe: stripe.createPaymentMethod() -> gets pm_id
4. POST to api/index.php { route: 'order.place_order', cart, shipping, pm_id/txhash/ref }
5. order.handler.php:
 a. Parses cart JSON (max 50 items)
 b. RE-FETCHES each product from DB, RE-DECRYPTS prices
 c. Decomposes variant_id (composite: rawVid % 1000 = qualityId)
 d. Creates order with ALL PII encrypted (Crypt::encrypt(..., 'order'))
 e. Creates order_items with encrypted fields
 f. Generates order_ref (32 hex) + tracking_number (CP + 10-digit)
 g. PaymentBridge::submit() -> ENGINE node for payment processing
6. Returns: order_ref, order_id, amount, client_secret/wallet_address/etransfer_email
7. For Stripe: stripe.confirmCardPayment(client_secret)
8. On success: clears localStorage cart, shows confirmation with tracking
```

-- Crypto payments removed 2026-06-16. Stripe and e-Transfer only.

---

## 7. Payment Bridge -> ENGINE

### PaymentBridge Class
**File**: `Y:\5\public\system\payment_bridge.php`
**ENGINE URL**: Primary: `http://10.69.69.6:8443` (from `.env` key `payment_node_url`); Fallback: `http://172.28.41.2:8443` (via `lantern`). **NOT port 8301** (that is the raw C++ TCP bridge used by `p1a2y3m4_bridge.php`, not this class).
**Auth**: HMAC-SHA256 signed requests (timestamp + nonce) + HMAC response verification enabled (2026-06-18 audit).
**Method**: `PaymentBridge::submit(string $action, array $payload): array` - replaces former raw curl calls.

### ENGINE HTTP API (C++ Daemon: `669427-engine`)
| Endpoint               | Method | Purpose                      |
| ---------------------- | ------ | ---------------------------- |
| `/a/health`            | GET    | Service health               |
| `/a/hb`                | POST   | Heartbeat receiver           |
| `/a/cmd/recv`          | POST   | Command queue for agents     |
| `/a/payment/init`      | POST   | Payment initiation           |
| `/a/payment/status`    | POST   | Transaction confirmation     |
| `/a/payment/cancel`    | POST   | Refunds                      |
| `/a/payment/callback`  | POST   | Payment webhook (nonce+HMAC) |
| `/a/email/send`        | POST   | Email queue                  |
| `/a/fulfillment/start` | POST   | Fulfillment initiation       |
| `/a/checkout/config`   | GET    | Payment config               |

### ENGINE Services (Node 006)
| Service           | Bind Address    | User     | Purpose                                                |
| ----------------- | --------------- | -------- | ------------------------------------------------------ |
| monerod           | 0.0.0.0:18081   | xmr      | Monero daemon - **DISABLED/REMOVED** (2026-06-16)      |
| monero-wallet-rpc | 127.0.0.1:18082 | xmr      | Wallet RPC - **DISABLED/REMOVED** (2026-06-16)         |
| tron-signer       | 127.0.0.1:9090  | mail-svc | USDT-TRC20 signing - **DISABLED/REMOVED** (2026-06-16) |
| ollama            | 127.0.0.1:11434 | -        | Mistral 7B (podman container)                          |
| mariadb           | 3306            | -        | engine database                                        |
| postfix           | 587             | -        | Outbound SMTP                                          |
| dovecot           | 110,143         | -        | IMAP                                                   |
| opendkim          | 8891            | -        | DKIM signing                                           |

### Engine Security
- **Constant-time HMAC comparison**: Uses `libsodium` (`sodium_memcmp`)
- **Strict TLS Verification**: `CURLOPT_SSL_VERIFYPEER = 1L`, `CURLOPT_SSL_VERIFYHOST = 2L` on ALL external API calls
- **Buffer Limits**: Hard-coded 64KB maximum payload size
- **Rate Limiting**: Token Bucket IP-based rate limiting per session
- **Zero Shell Invocation**: `popen()` and `system()` completely purged
- **Process Deception**: Native `prctl(PR_SET_NAME)` and `argv` masking

---

## 8. Shield - Central Command Hub

Shield (Node 004) is the hub of the entire infrastructure. All inter-node communication MUST transit through Shield.

### Shield Services
| Service         | Bind       | Port | Purpose                                     |
| --------------- | ---------- | ---- | ------------------------------------------- |
| keyserv         | 10.69.69.1 | 8302 | Vault key distribution (HMAC authenticated) |
| keyserv-alt     | 10.69.69.1 | 8303 | Alt key endpoint                            |
| shield-receiver | 10.69.69.1 | 8401 | Heartbeat receiver from all nodes           |
| dl-auth         | 10.69.69.1 | 8445 | Download authentication service             |
| s3rv1c3agent    | -          | -    | Orchestrator daemon                         |
| nginx           | -          | 80   | Reverse proxy to Origin                     |

### Shield Assets
- `/home/admin/.ssh/gabriella_key` - SSH key to ALL nodes
- `/etc/bridge_hmac.key` - Bridge HMAC for cross-node auth
- `/opt/apache/vault/master.key` -> `/dev/shm/vault.key` - Vault master key
- Per-node disguised tool directory (see TOOLING_MAP.md)
- `/opt/ghost/tools/*.enc` - Encrypted ghost tools (12 total): blackhole, dns, filter, ping, probe, service, wg-status, integrity, kernel, network, vault, backup
- `/opt/ghost/run.sh` - Ghost runner (SHA-256 hash verification before decrypt, AES-256-CBC, tmpfs execute, shred)
- `/opt/ghost/.runner_hash` - SHA-256 manifest for all 12 tools (chmod 600)
- `/opt/ghost/tools/*.sha256` - Per-tool SHA-256 sidecar files
- `/opt/svc/node_id` - Node identity file (contains `414342`)
- `/etc/wireguard/keys/` - Separated WG private keys (chmod 600, chmod 700 dir)
- `/opt/sysc/dl/dl_pass` - DL server auth password

### Blackhole Chain Architecture

Blackhole containers run ONLY on the **user-facing traffic chain** — never on Engine, Origin, or Vault.

```
Internet → Proxy (maintenance) → Filter (geo/bot ban) → Shield (self-exclusion/trust) → Origin
              │                       │                        │
              ▼                       ▼                        ▼
         maint-page              geo-block                blackhole-nginx
         (per-store)             bot-ban                  self-exclusion
                                 nftables drop            engine removal
                                                          trust score fail
```

| Node | Responsibility | Method | Container Port |
|------|---------------|--------|----------------|
| **Proxy** (244939) | Maintenance mode pages (per-store toggle) | Podman `maint-page` + `blackhole-proxy` | 9999, 8444, 65000 |
| **Filter** (333941) | Geolocation blocks, bot bans, scanner traps | nftables + Podman (NEEDS DEPLOY) | 9998 (planned) |
| **Shield** (414342) | Self-exclusion, engine removal, trust score failures | Podman `blackhole-nginx` + `self-exclusion` | 9997, 8444, 9443 |
| **Vault** (869377) | ❌ **NEVER** — credential vault only | N/A | N/A |
| **Engine** (669427) | ❌ **NEVER** — processing node only | N/A | N/A |
| **Origin** (558201) | ❌ **NEVER** — data origin only | N/A | N/A |

### Podman Containers — Shield (LIVE)

| Container | Port | Purpose | Status |
|-----------|------|---------|--------|
| `blackhole` | `127.0.0.1:9997` | Honeypot trap (wp-admin, .env, .git scanners) | ACTIVE |
| `sinkhole` | `0.0.0.0:8444` | Self-exclusion / engine removal / trust fail blackhole | ACTIVE |
| `exclusion-svc` | `127.0.0.1:9443` | Self-exclusion service (gunicorn) | ACTIVE |

- nginx `relay.conf` routes bot probes (`/wp-admin`, `/.env`, `/.git`) → `127.0.0.1:9997`
- nginx `relay.conf` routes excluded/failed-trust users → `sinkhole` upstream on `127.0.0.1:8444`

> [!WARNING]
> `blackhole` (9997) is NOT set to auto-restart. Container exited after 25h on 2026-06-20. Must add Podman systemd unit for persistence.

### Shield Dispatch Architecture (LIVE)

```
Origin Dashboard
 | HTTPS POST signed with bridge_hmac.key
 v
Shield (10.69.69.1:8401) - SINGLE GATEWAY
 |- Validates HMAC signature
 |- Identifies target node (by node_id)
 |- SSHes to target using gabriella_key
 |
 |--> Engine (10.69.69.6) -> sudo tvault <tool> [args]
 |--> Proxy (10.69.69.3) -> sudo bash /opt/maint/<script> [args]
 |--> Vault (10.69.69.8) -> vault access via compass tunnel
 |--> DNS (10.69.69.5) -> sudo pdnsutil [cmd]
 +--> Filter (via Proxy jump) -> sudo ftool [tool] [args]
 ssh -J 10.69.69.3 admin@91.229.239.29
```

### Shield Firewall State
```
Port 80: rich-rule -> 10.69.69.0/24 (WG mesh) ONLY
Port 8302: keyserv - WG mesh only
Port 8401: shield-receiver - WG mesh only
Port 8445: dl-auth - WG mesh only
Port 9922: SSH - restricted
Port 41820: WireGuard wg7
```

---

## 9. Filter Pipeline & WAF System

### Filter Tool System (`ftool`)
**Location:** `/opt/filter/bin/ftool`
**Vault:** `/opt/filter/vault/*.enc` (AES-256-CBC, key in `/dev/shm/filter.vault.key`)
**Domain configs:** `/etc/nginx/domains/<domain>.conf`
**Rule files:** `/opt/filter/rules/<domain>.rules`

### ftool Commands
```bash
# Domain management
ftool filter-domain set cleanearth.cc --rate 30r/s --burst 60 --store-id 1
ftool filter-domain set cleanpharma.cc --rate 25r/s --burst 50 --store-id 2
ftool filter-domain list
ftool filter-domain remove olddomain.com

# Per-domain WAF rules
ftool filter-rule cleanearth.cc block-ip 1.2.3.4
ftool filter-rule cleanearth.cc block-ua "scrapy|python-requests"
ftool filter-rule cleanearth.cc block-path "/api/old-endpoint"
ftool filter-rule cleanearth.cc trust-ip 192.168.1.100
ftool filter-rule cleanearth.cc list

# Proxy failover
ftool filter-failover status
```

### Filter Firewall State (nftables - migrated 2026-06-14)
```
Backend: nftables v1.0.9 (replaces firewalld - disabled)
Policy: default DROP on input chain
Loopback: accept
Established: accept
Blacklist: dynamic set with 1h auto-timeout
ICMP: rate-limited 5/second
Port 80: 255.17.69.4/30 (Proxy tunnel) ONLY
Port 8380: 255.17.69.4/30 + 255.17.69.5/32 (keyserv)
Port 9922: 194.36.190.111 (Shield) + 80.78.19.57 (DNS) + 80.96.113.0/24 (local)
Port 9443: 10.69.69.0/24 (WG mesh)
Port 41920: WireGuard (wg1)
Port 49616: WireGuard (shale - Proxy<->Filter)
Port 49619: WireGuard (additional)
All other: 172.20.69.0/30 full accept
Config: /etc/nftables.conf (chmod 600)
```

### Blackhole — Filter (NEEDS DEPLOY)
- **No Podman installed on Filter** — currently using nftables drop rules only
- **DESIGN**: Should have `blackhole-filter` on `127.0.0.1:9998` for:
  - Geolocation-blocked visitors (serve localized block page)
  - Bot/scanner ban pages (serve honeypot instead of silent drop)
  - Known bad user-agent blackhole
- Previous container was never redeployed after Filter node rebuild (2026-06-18)

### Podman Containers — Proxy (LIVE)

| Container | Port | Purpose | Status |
|-----------|------|---------|--------|
| `blackhole` | `127.0.0.1:9999` | Bot probe honeypot | ACTIVE |
| `sinkhole` | `0.0.0.0:8444` | Blackholed user traffic sink | ACTIVE |
| `maint-svc` | `127.0.0.1:65000` | Per-store maintenance pages | CRASHED |

- nginx `gateway.conf` routes bot probes → `127.0.0.1:9999`
- nginx `gateway.conf` routes excluded users → `sinkhole` upstream on `127.0.0.1:8444`

> [!WARNING]
> `maint-svc` container crashed with `worker_processes directive not allowed` error. Needs config fix before restart.

### Stale Tunnel Cleanup (Session 2026-06-14)
- **EarT4** (Filter->old Origin 93.95.226.207): **Removed and disabled from boot**
- Shield `wg7.conf` AllowedIPs for Filter peer: **Fixed** (was missing `10.69.69.4/32`)
- Shield firewall: **Port 80 opened** for WG subnet `10.69.69.0/24`

---

## 10. DNS / Download Server (margit - Node 001)

### DNS
**Server:** PowerDNS on margit (80.78.19.57)
**Zones:** `cleanpharma.cc`, `cleanearth.cc`, and all subdomains

### DL Server
**Domain:** `dl.cleanpharma.cc`
**DocumentRoot:** `/var/www/downloads/`
**Web Server:** nginx (port 443 SSL, Let's Encrypt cert)
**Auth:** nginx `auth_request` -> Shield `dl-auth` service at `10.69.69.1:8445`

#### DL Authentication
- **Service:** Python HTTP server on Shield (`/usr/local/bin/dl-auth`)
- **Bind:** `10.69.69.1:8445` (WG mesh only)
- **Auth methods:**
 - HTTP Basic Auth -> user: `dl`, password from `/opt/sysc/dl/dl_pass`
 - URL token: `?token=<password>`
- **Password rotation:** `GET /dl/rotate` (loopback only)
- **Status check:** `GET /dl/status`

#### Adding a New Store Domain
```bash
# 1. On Filter - configure domain
ftool filter-domain set newstore.com --rate 20r/s --burst 40 --store-id 3

# 2. On DNS (margit) - add A record
pdnsutil add-record newstore.com @ A 300 107.189.23.194
pdnsutil add-record newstore.com www A 300 107.189.23.194
pdns_control notify newstore.com

# 3. On Proxy (webserver.conf) - already catches all server_names, no change needed

# 4. On Origin - add store in DB, configure vhost in PHP
```

---

## 11. Vault-Encrypted Tool System

### Execution Flow
```
ShieldRelay::exec('payment', 'store --domain X --store-id Y')
 |
 v (via /var/lib/php/sessions/.pending/ file relay)
s3rv1c3agent on Engine picks up .req file
 |
 v
/root/oomd-report.sh exec-raw "store --domain X --store-id Y"
 |
 v
tvault exec store-provision -- --domain X --store-id Y
 |
 |- Shield keyserv (10.69.69.1:8302)
 | HMAC-signed request using /dev/shm/vault.key
 | Returns: node-specific AES key (128 hex chars)
 |
 ||- Decrypt per-node tool dir vault/store-provision.enc
 | AES-256-CBC, PBKDF2 100000 iterations
 | Output: /dev/shm/.te/store-provision.<PID> RAM only
 |
 v
bash /dev/shm/.te/store-provision.<PID> --domain X --store-id Y
 |
 v (tool executes, configures Proxy + Origin)
shred -fuz /dev/shm/.te/store-provision.<PID> wiped from RAM
```

### Vault Key Bootstrap
```
/opt/apache/vault/master.key (Shield, persistent, 600)
 |
 v (vault-init.service on boot)
/dev/shm/vault.key RAM, cleared on reboot, re-loaded by service
 |
 |-- Used by keyserv to VERIFY HMAC signatures from nodes
 +-- Used by tvault on each node to SIGN key requests
```

> [!IMPORTANT]
> `/dev/shm/vault.key` must be the SAME on Shield AND every node.
> It's the shared HMAC secret. Re-sync after any key rotation.

### Runner Dispatch (`/root/oomd-report.sh`)
| Tool word   | Maps to                       | Purpose                   |
| ----------- | ----------------------------- | ------------------------- |
| `store`     | `tvault exec store-provision` | New domain nginx pipeline |
| `backup`    | `tvault exec backup-runner`   | Build/status/rotate/purge |
| `dl-backup` | `tvault exec dl-backup`       | Direct archive tool       |
| `service`   | systemctl wrapper             | Start/stop/status         |
| `probe`     | system info                   | disk/mem/uptime           |
| `ping`      | JSON pong                     | Relay health check        |

---

## 12. Backup System

### Dashboard Backup Tab
**Handler:** `Y:\5\dashboard\modules\backups\backups_handler.php`
**Storage:** `/opt/backups/archives/bk_YYYYMMDD_HHMMSS/`

**What gets backed up:**
- `store` database -> `store.sql` (mariadb-dump)
- `/var/www/dashboard/` -> `dashboard.tar.gz`
- `/var/www/shared/` -> `shared.tar.gz`
- `/var/www/public/` -> `public.tar.gz`
- `/etc/wireguard/` -> `wireguard.tar.gz`

### Cross-Node Backup to DL Server
Files can be SCP'd across the WG mesh to margit for off-site storage:
```
Origin (10.69.69.2) --SCP via wg7---> margit (10.69.69.5):/var/www/downloads/
Engine (10.69.69.6) --SCP via wg7---> margit (10.69.69.5):/var/www/downloads/
```

> [!NOTE]
> Engine node has no `gabriella_key` - for SCP transfers, the key must be temporarily
> injected via SSH session, used for SCP, then removed. Origin has the key at
> `/home/admin/.ssh/gabriella_key`.

---

## 13. Order & Fulfillment System

### Order Status Flow
```
2001 Pending -> 2002 Paid -> 2003 Processing -> 2004 Packed -> 2005 Shipped -> 2006 Delivered
 2007 Cancelled
 2008 Refunded
 2009 Disputed
 2010 Hold
```
`StatusCode::canTransitionOrder()` enforces forward-only flow.

### Tracking
- Format: `CP` + 10-digit random (Canada Post style)
- E-transfer reference: `CP-XXXX-XXXX`
- Customer tracking page: `Y:\5\public\pages\track.php`

---

## 14. Database Tables - Complete Inventory

### Core Commerce
| Table                             | Purpose                            | Customer Data?                         |
| --------------------------------- | ---------------------------------- | -------------------------------------- |
| `products`                        | Product catalog                    | No                                     |
| `product_types`                   | Product type variants              | No                                     |
| `product_qualities`               | Quality tier variants (A, AA, AAA) | No                                     |
| `product_units`                   | Packaging options per quality      | No                                     |
| `categories`                      | Product categories                 | No                                     |
| `orders`                          | Customer orders                    | YES - ALL PII encrypted (order domain) |
| `order_items`                     | Line items per order               | YES - encrypted                        |
| `order_status_history`            | Order stage transitions            | No                                     |
| `payments`                        | Payment records                    | AUDIT NEEDED                           |
| `payment_methods`                 | Stored payment tokens              | token_enc encrypted                    |
| `payment_logs` / `payment_ledger` | Payment audit trail                | AUDIT NEEDED                           |
| `coupons`                         | Discount codes                     | No                                     |
| `cart_items`                      | Server-side cart (legacy?)         | AUDIT                                  |

### Shipping
| Table             | Purpose          | Encryption Status                                     |
| ----------------- | ---------------- | ----------------------------------------------------- |
| `shipments`       | Shipping records | tracking_number_enc, label_data_enc, api_response_enc |
| `shipping_labels` | Encrypted labels | label_pdf_enc, label_zpl_enc, dimensions_enc          |

### Users
| Table    | Purpose           | Encryption Status                     |
| -------- | ----------------- | ------------------------------------- |
| `users`  | Customer accounts | MIXED - has both plain + _enc columns |
| `admins` | Admin accounts    | AUDIT                                 |

### Support
| Table              | Purpose  | Encryption Status                                  |
| ------------------ | -------- | -------------------------------------------------- |
| `support_messages` | Messages | MIXED - has subject + subject_enc, body + body_enc |
| `support_tickets`  | Tickets  | MIXED - has subject + subject_enc                  |
| `support_replies`  | Replies  | message plaintext                                  |

### Security
| Table                        | Purpose                           |
| ---------------------------- | --------------------------------- |
| `audit_log`                  | Action audit trail                |
| `fleet_nodes` / `nodes`      | Node registry                     |
| `fleet_heartbeats`           | Node health                       |
| `ip_blacklist` / `blacklist` | Blocked IPs                       |
| `rate_limits`                | Rate limiting                     |
| `threat_intel_cache`         | Threat intel                      |
| `login_attempts`             | Failed logins                     |
| `obfuscation_map`            | DOM element mapping               |
| `sec_tool_runs`              | Agent tool executions (encrypted) |
| `sec_lockdown_state`         | Lockdown events                   |
| `sec_events`                 | Security events                   |
| `sec_bash_integrity`         | File integrity                    |
| `sec_canary_triggers`        | Honeypot triggers                 |
| `sec_key_rotations`          | Key rotations                     |
| `fleet_commands`             | Queued node commands              |
| `nn_fleet_routing`           | Fleet topology                    |
| `merkle_anchors`             | Audit integrity                   |
| `deception_tokens`           | Deception tracking                |
| `honeypot_hits`              | Trap access logs                  |
| `visitor_sessions`           | Behavioral tracking               |

### Finance
| Table                 | Purpose             | Encryption Status     |
| --------------------- | ------------------- | --------------------- |
| `wallets`             | Crypto wallets      | address_enc encrypted |
| `wallet_transactions` | Crypto TXs          | tx_hash_enc encrypted |
| `invoices`            | Invoices            | AUDIT                 |
| `revenue_daily`       | Revenue aggregation | No (aggregate)        |

### Content
| Table          | Purpose            |
| -------------- | ------------------ |
| `faqs`         | FAQ entries        |
| `reviews`      | Product reviews    |
| `testimonials` | Testimonials       |
| `social_posts` | Social media       |
| `terms`        | Legal pages        |
| `pages`        | CMS pages          |
| `newsletters`  | Newsletter entries |
| `email_log`    | Email audit trail  |
| `ui_labels`    | UI label system    |
| `settings`     | System settings    |

---

## 15. Encryption Gaps - REMEDIATED 2026-06-14

> [!NOTE]
> All P0/P1 encryption gaps below were remediated in the security hardening session 2026-06-14.

### `users` Table
| Field         | Status      | Remediation                               |
| ------------- | ----------- | ----------------------------------------- |
| `email`       | Login index | `encrypted_email` column active           |
| `name`        | Resolved    | `name_enc` active                         |
| `phone`       | Resolved    | `phone_enc` active                        |
| `username`    | Resolved    | `encrypted_username` active               |
| `first_name`  | Resolved    | `first_name_enc` added, plaintext NULLed  |
| `last_name`   | Resolved    | `last_name_enc` added, plaintext NULLed   |
| `address`     | Resolved    | `address_enc` added, plaintext NULLed     |
| `city`        | Resolved    | `city_enc` added, plaintext NULLed        |
| `postal_code` | Resolved    | `postal_code_enc` added, plaintext NULLed |
| `last_ip`     | Resolved    | `last_ip_enc` added, plaintext NULLed     |

### `payments` Table
| Field              | Status   | Remediation                                 |
| ------------------ | -------- | ------------------------------------------- |
| `gateway_txn_id`   | Resolved | `gateway_txn_id_enc` added, data migrated   |
| `gateway_response` | Resolved | `gateway_response_enc` added, data migrated |
| `payer_email`      | Resolved | `payer_email_enc` added, data migrated      |

### `support_messages` / `support_tickets` / `support_replies`
| Field                      | Status   | Remediation                                  |
| -------------------------- | -------- | -------------------------------------------- |
| `support_messages.subject` | Resolved | Plaintext column dropped, `subject_enc` only |
| `support_messages.body`    | Resolved | Plaintext column dropped, `body_enc` only    |
| `support_tickets.subject`  | Resolved | Plaintext column dropped, `subject_enc` only |
| `support_replies.message`  | Resolved | `message_enc` added, data migrated           |

---

## 16. Proxy cloudflare.conf (Updated 2026-06-14)

Proxy uses `cloudflare.conf`:

```nginx
# Cloudflare IPv4
allow 173.245.48.0/20;
allow 103.21.244.0/22;
allow 103.22.200.0/22;
allow 103.31.4.0/22;
allow 141.101.64.0/18;
allow 108.162.192.0/18;
allow 190.93.240.0/20;
allow 188.114.96.0/20;
allow 197.234.240.0/22;
allow 198.41.128.0/17;
allow 162.158.0.0/15;
allow 104.16.0.0/13;
allow 104.24.0.0/14;
allow 172.64.0.0/13;
allow 131.0.72.0/22;
# Cloudflare IPv6
allow 2400:cb00::/32;
allow 2606:4700::/32;
allow 2803:f800::/32;
allow 2405:b500::/32;
allow 2405:8100::/32;
allow 2a06:98c0::/29;
allow 2c0f:f248::/32;
# Internal: Shield gateway + WG mesh admin
allow 194.36.190.111;
allow 10.69.69.0/24;
deny all;
```

> [!NOTE]
> Shield IP (`194.36.190.111`) and WG subnet (`10.69.69.0/24`) were added on 2026-06-14
> because user traffic routes via Shield VPN - Shield's IP was being denied by cloudflare.conf.

---

## 17. Navigation Structure

### Left Sidebar
- **GLOBAL**: Overview
- **STORES**: Clean Earth, Clean Pharma (clicking -> `?view=store&id=X` -> listings.php)
- **OPERATIONS**: Fulfillment, Users, Support, FAQ, Reviews, Marketing, Settings
- **FINANCE**: Money, Proofs

### Right Sidebar (Service Mesh)
- Fleet Control, Topology, Node Console, Logs
- Security, Traffic Intel, Devices, Obfuscation, Packages, Safety
- Mesh Security, Watchdog
- **Backups** - Full backup pipeline tab

### Store Navigation
- **NO DROPDOWN** - clicking store name goes directly to product listing
- Store tab has: PRODUCTS | CONFIGURATION sub-tabs

---

## 18. Boot Persistence - Critical Services

### Engine (Node 006)
| Service               | Purpose                        | Boot State                        |
| --------------------- | ------------------------------ | --------------------------------- |
| 669427-engine         | C++ payment/fulfillment daemon | enabled                           |
| pay-relay             | Payment relay                  | enabled                           |
| mariadb               | Engine database                | enabled                           |
| postfix               | Outbound SMTP                  | enabled                           |
| ~~monero-wallet-rpc~~ | ~~XMR wallet RPC~~             | **disabled** - Removed 2026-06-16 |
| sshd                  | SSH access                     | enabled                           |
| firewalld             | Firewall                       | enabled                           |
| crond                 | Cron jobs                      | enabled                           |
| wg-quick@wg7          | WG mesh tunnel                 | enabled                           |

### Shield (Node 004)
| Service                  | Purpose                                | Boot State |
| ------------------------ | -------------------------------------- | ---------- |
| s3rv1c3agent             | Orchestrator daemon                    | enabled    |
| dl-auth                  | DL authentication service              | enabled    |
| vault-init               | Loads master.key -> /dev/shm/vault.key | enabled    |
| nginx                    | Reverse proxy to Origin                | enabled    |
| sshd / firewalld / crond | Standard                               | enabled    |
| wg-quick@wg7             | WG mesh tunnel                         | enabled    |

### Origin (Node 005)
| Service                  | Purpose        | Boot State |
| ------------------------ | -------------- | ---------- |
| nginx                    | Web server     | enabled    |
| php-fpm                  | PHP processing | enabled    |
| mariadb                  | Store database | enabled    |
| sshd / firewalld / crond | Standard       | enabled    |
| wg-quick@wg7             | WG mesh tunnel | enabled    |

---

## 19. What NOT to Do

> [!CAUTION]
> 1. DO NOT create a second product file. `listings.php` is the ONLY one.
> 2. DO NOT add dropdown submenus to store nav. They were removed.
> 3. DO NOT use external CDNs, npm, or third-party libraries (Rule 1).
> 4. DO NOT store plaintext customer data. Everything encrypted at rest.
> 5. DO NOT rebuild systems that already exist. Trace the code first.
> 6. DO NOT embed inline CSS/JS in PHP files (Rule 20). Use assets/ files.
> 7. DO NOT let Origin communicate directly with Proxy (Rule 30 - must go through Shield).
> 8. DO NOT trust client-sent prices. Always re-validate from DB.
> 9. DO NOT bypass the traffic chain. It's Proxy -> Filter -> Shield -> Origin. Always.
> 10. DO NOT use emoji above U+1F300 in dashboard (Rule 18). Use `Icons::` SVG methods.
> 11. DO NOT create new WireGuard tunnels without documenting them in the WG tunnel table.
> 12. DO NOT SSH directly from Engine/Proxy to other nodes for commands - route through Shield dispatch.
> 13. DO NOT create separate JS command systems per tab. ALL node commands route through `ncCmd()` in `fleet.js` -> `node_cmd.php`. One function. One endpoint. Always.
> 14. DO NOT change permissions on `/var/www/private/keys/fleet`. It MUST be `root:nginx 640`. If it reverts to `root:root 600`, PHP-FPM (nginx user) loses SSH access and ALL fleet commands silently fail across every tab.
> 15. DO NOT create tab-specific API endpoints that duplicate `node_cmd.php` logic. Add new actions as cases inside `node_cmd.php`.

---

## 19b. Zero-Trust Security Services

### svcauth - SSH Port Auto-Rotation
- PAM hook at `/etc/pam.d/sshd`
- Rotates SSH port from pool [9922, 30222, 33022] on login without session gate token
- 30-second grace period, updates SELinux + firewalld + sshd_config
- Deployed: 7/8 nodes (Payment pending)

### svcwatch - File Integrity Watchdog
- systemd service monitoring /etc/shadow, sshd_config, authorized_keys
- SHA-256 baseline hashes at /run/se/state/file_hashes
- Auto-rotates passwords + re-locks shadow on tamper detection
- Reports to Origin via POST /api/mesh-api.php?action=report-watchdog-alert
- Deployed: 7/8 nodes

### chattr +i Protection
- Immutable flag on /etc/shadow prevents rescue mode password changes
- svcwatch re-applies if removed

### Tooling Obfuscation (2026-06-20)
- All tool directories, daemon binaries, and session scripts use per-node disguised names
- TOOL_ID system maps generic purpose to per-node filename
- Source of truth: TOOLING_MAP.md (local only, never deployed)

---

## 20. Dashboard Command Architecture (Unified - 2026-06-18)

> [!IMPORTANT]
> There is ONE command system. Everything flows through it. Breaking this pattern is what caused Fleet Control, Infrastructure, and Node Console to all fail simultaneously.

### The Single Command Path

```
Any dashboard button (any tab)
 | data-nc="node-cmd" data-node="{id}" data-action="{action}"
 v
fleet.js -> ncCmd(nodeId, action) THE ONLY JS COMMAND FUNCTION
 | POST /api/fleet/node_cmd.php
 | { node, action, service, _csrf }
 v
node_cmd.php THE ONLY API COMMAND ENDPOINT
 | AuthChain::isAuthenticated()
 | RBAC::enforce('manage.fleet')
 | PIN check (for destructive actions)
 | runNodeCmd($nodeId, $sshCmd)
 v
SSH as nginx -> admin@{wg_ip} -p 9922 FLEET KEY REQUIRED (see below)
 v
Node executes command, returns output
```

### Fleet SSH Key - CRITICAL

| Property          | Value                                                                                   |
| ----------------- | --------------------------------------------------------------------------------------- |
| **Path**          | `/var/www/private/keys/fleet`                                                           |
| **Owner**         | `root:nginx`                                                                            |
| **Mode**          | `640` (`rw-r-----`)                                                                     |
| **Why nginx**     | PHP-FPM runs as `nginx` user - needs read access to connect to nodes                    |
| **Verify**        | `ls -la /var/www/private/keys/fleet` -> must show `-rw-r----- root nginx`               |
| **Fix if broken** | `chown root:nginx /var/www/private/keys/fleet && chmod 640 /var/www/private/keys/fleet` |

> [!CAUTION]
> If the key is `root:root 600`, PHP-FPM cannot read it. Every single SSH-dependent feature fails **silently** - no error shown to user, buttons appear to do nothing, stats show `-`. This was the root cause of Fleet Control, Infrastructure, and engine uptime all failing simultaneously on 2026-06-18.

### node_cmd.php - Supported Actions

| Action                   | SSH Command                                 | Notes                                                                                                     |          |            |
| ------------------------ | ------------------------------------------- | --------------------------------------------------------------------------------------------------------- | -------- | ---------- |
| `health_check`           | `uptime; df -h /; free -m`                  | Node vitals                                                                                               |          |            |
| `service_status`         | `systemctl status {service}`                | Service state                                                                                             |          |            |
| `service_restart`        | `sudo systemctl restart {service}`          | PIN required                                                                                              |          |            |
| `service_stop`           | `sudo systemctl stop {service}`             | PIN required                                                                                              |          |            |
| `service_start`          | `sudo systemctl start {service}`            | PIN required                                                                                              |          |            |
| `logs_tail`              | `sudo journalctl -u {service} -n 50`        | Last 50 lines                                                                                             |          |            |
| `network_test`           | `ping -c3` to mesh IPs                      | Reachability                                                                                              |          |            |
| `wg_status`              | `sudo wg show`                              | WireGuard peers                                                                                           |          |            |
| `firewall_status`        | `sudo nft list ruleset`                     | nftables/firewalld                                                                                        |          |            |
| `update_check`           | `sudo yum check-update`                     | Pending updates                                                                                           |          |            |
| `processes`              | `ps aux --sort=-%cpu`                       | Top processes                                                                                             |          |            |
| `uptime`                 | `uptime -p`                                 | Node uptime                                                                                               |          |            |
| `ssh_status`             | `sudo sshd -T \                             | grep -E 'port\                                                                                            | permit'` | SSH config |
| `ssh_open` / `ssh_close` | firewall-cmd to open/close SSH              | PIN required                                                                                              |          |            |
| `kernel_harden`          | sysctl hardening params                     | PIN required                                                                                              |          |            |
| `engine_config_show`     | `cat /opt/apache/engine/config/engine.json` | Engine only                                                                                               |          |            |
| `engine_restart_svc`     | `sudo systemctl restart sysd-engine`        | Engine only, PIN                                                                                          |          |            |
| `tool_run`               | `/opt/ghost/run.sh {tool} {action}`         | Run ghost tool on node; allowlist: svcbeat/svcfile/svcfw/svclock/svclog/svcpass/svcperm/svcrotate/svctrap |          |            |

**To add a new action**: Add a `case` to `node_cmd.php`. Do NOT create a new PHP file.

### engine_panel.js - Engine Control Panel

- Polls `api/system/engine_status.php` every 10 seconds for KPI values (uptime, blacklist, DNS mode, DDOS)
- `engine_status.php` uses **SSH** (not curl) to get live stats - requires fleet key to be `root:nginx 640`
- Button clicks (`engine_health`, `engine_logs`, `engine_config`, `engine_restart`) post to `node_cmd.php` with node `669427`
- Output renders in `#nc-engine-output` div (appended to `engine_panel.php`)

### api/infrastructure.php (Created 2026-06-18)

- `GET ?action=status` -> reads node stats from DB, returns `{total, healthy, stale, dead, nodes[]}`
- `POST action=integrity_scan|wg_status|probe_system|ping_all` -> SSHes to target nodes, returns results
- **Uses same fleet key** - same SSH pattern as `node_cmd.php`
- `infrastructure.js` was previously 404/405 because this file did not exist

### api/system/heartbeat.php - Session Keepalive

- Browser endpoint only - refreshes `$_SESSION['last_activity']` for authenticated admin sessions
- **NOT a node heartbeat** - node heartbeats go to `api/heartbeat.php` (different file)
- Returns `401` for unauthenticated requests (expected for mesh nodes hitting it)
- On `401` from a mesh IP (`10.69.69.x`): writes structured event to `system_events` table
- `system_events` table: `event_type, source_ip, node_role, endpoint, detail, created_at`

### cspNonce Variable - Dashboard Templates

- `index.php` declares `$cspNonce` (NOT `$nonce`)
- All tab files included via `require_once` inherit this scope
- Use `$cspNonce ?? ''` in tab files for nonce attributes on `<script>` and `<link>` tags
- **Wrong**: `<?= $nonce ?>` -> PHP Warning: Undefined variable
- **Correct**: `<?= $cspNonce ?? '' ?>`

### account.handler.php - Column Aliases (2026-06-18)

The `orders` and `support_tickets` tables use encrypted/renamed columns. Queries must alias:

```sql
-- orders: column is tracking_carrier, not carrier
SELECT tracking_carrier AS carrier FROM orders ...

-- support_tickets: column is subject_enc, not subject 
SELECT subject_enc AS subject FROM support_tickets ...
-- PHP then decrypts: Crypt::d($r['subject'])
```

---

## 21. Pending Architecture Items

### Build Queue - Completed 2026-06-14 (Full Alignment Session) + v3.2 Updates 2026-06-14
| #   | Item                                                                  | Priority | Status                                                                                                                 |
| --- | --------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------- |
| 1   | Shield dispatch endpoint (`POST /dispatch` on `10.69.69.1:8401`)      | CRITICAL | DONE - integrated into `shield-receiver.py` (HMAC, nonce, SSH routing to all nodes)                                    |
| 2   | Ghost `blackhole` tool (maintenance mode + exclusion + honeypot)      | HIGH     | DONE - `/opt/ghost/tools/blackhole.enc`, runner at `/opt/ghost/run.sh`                                                 |
| 3   | Ghost `dns` tool (pdnsutil failover wired to dispatch)                | HIGH     | DONE - `/opt/ghost/tools/dns.enc`, calls pdnsutil on DNS node                                                          |
| 4   | Ghost `filter` tool (ftool passthrough via jump to Filter)            | HIGH     | DONE - `/opt/ghost/tools/filter.enc`, SSH jump via Proxy                                                               |
| 5   | `security_control.php` backend (Origin API -> Shield dispatch)        | HIGH     | VERIFIED - file already present on Origin at `/var/www/dashboard/api/security_control.php` with correct HMAC signing   |
| 6   | `proxy-watchdog` update (route through Shield, not direct SSH)        | MEDIUM   | VERIFIED - `proxy-watchdog.sh` on Engine already uses `shield_dispatch()` to POST to `http://10.69.69.1:8401/dispatch` |
| 7   | `blackhole-manage` update (replace direct SSH with Shield dispatch)   | MEDIUM   | DONE - `blackhole-manage.enc` on Engine rebuilt with Shield dispatch API, old direct-SSH version backed up             |
| 8   | Store DB security controls migration                                  | MEDIUM   | - Deferred: not an infrastructure item; tracked in DB migration backlog                                                |
| 9   | Per-store nginx maintenance mode toggle on Proxy nodes                | MEDIUM   | DONE - `/opt/maint/stores/store_N.json` files managed by ghost-blackhole via Shield dispatch; nginx reloads on write   |
| 10  | Engine vault-init.service (boot persistence for vault key sync)       | LOW      | VERIFIED - `vault-init.service` enabled on Engine; loads HMAC key to `/dev/shm/vault.key` at boot                      |
| 11  | Filter wg7 AllowedIPs integration (currently reached via Proxy jump)  | LOW      | BY DESIGN - Filter reachable via SSH jump through Proxy; wg7 not present on Filter (intentional)                       |

### Additional Items Resolved in This Session
| #   | Item                                                                                           | Resolution                                                                                                                                         |
| --- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| A   | `ghost-dispatch.service` (socat) conflicted with `shield-receiver.service` on port 8401        | Disabled + masked `ghost-dispatch.service`                                                                                                         |
| B   | `s3rv1c3agent.service` on Filter had wrong SELinux contexts (`user_home_t`)                    | Relabeled unit + binary to correct contexts; service now `enabled + running`                                                                       |
| C   | Engine `wg-quick@wg7` was in `failed` state (interface existed before service start)           | Cycled interface; service now `active (exited)`                                                                                                    |
| D   | Engine `669427-engine-agent.sh` emitting malformed service JSON (`mem=[not set]`)              | Fixed `service_metric()` to sanitize non-numeric values; JSON now valid                                                                            |
| E   | `monerod.service` on Engine was disabled with LMDB corruption (MDB_BAD_TXN)                    | Service enabled; LMDB data.mdb corrupted - disk at 90% with 77G blockchain, insufficient space to repair. Needs disk expansion or blockchain prune |
| F   | Proxy `s3rv1c3agent` (binary 244939) sending heartbeats; Shield /api/status confirmed HEALTHY  | Verified operational                                                                                                                               |
| G   | Engine `669427-engine-agent` sending heartbeats; Shield /api/status confirmed HEALTHY          | Verified operational                                                                                                                               |

### v3.2 Session Completed Items
| #   | Item                                               | Status                                                                                                 |
| --- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| F7  | WG private key separation (Origin, Engine, Shield) | DONE - keys in `/etc/wireguard/keys/` (chmod 600), PostUp loads key, no inline PrivateKey in any .conf |
| T2  | 9 remaining ghost tools built and encrypted        | DONE - all 12 tools in `/opt/ghost/tools/`, runner hash manifest created, SHA-256 sidecars generated   |
| T2b | run.sh hash verification                           | DONE - hash check before decryption prevents tampered tool execution                                   |
| T3  | nftables dynamic firewall on Filter                | DONE - firewalld disabled, nftables active, dynamic blacklist set with 1h timeout                      |
| T4  | Podman blackhole containers                        | DONE - running on Proxy (9999) and Shield (9997). Filter (9998) NOT deployed (no Podman on rebuilt Filter node). Both nodes also have blackhole-nginx on 8444 for excluded users. |
| T5  | hp-trap boot persistence on Engine                 | DONE - `/etc/wireguard/hp-trap.conf` saved, `wg-quick@hp-trap` enabled                                 |
| O6  | wg10 dead tunnel on Shield                         | DONE - zero handshakes confirmed, backed up to wg10.conf.disabled, service disabled                    |

### Remaining Open Items
| #   | Item                                               | Notes                                                                                                                                                                                                                                          |
| --- | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| R3  | Shield self-heartbeat stale                        | Shield 414342 shows high age_sec at startup; heartbeat thread sends heartbeat to itself every 60s. Add explicit self-heartbeat logic to `shield-receiver.py`                                                                                   |
| R4  | HB parse errors from Proxy heartbeat               | `mem=[not set]` in service JSON for `wg-quick@wg7` (exited services return no MemoryCurrent). Fixed in `heartbeat.sh` service_metric()                                                                                                         |

### Ghost Tool Inventory (Complete - v3.2)
| Tool      | File                             | Size  | Purpose                                           |
| --------- | -------------------------------- | ----- | ------------------------------------------------- |
| blackhole | `/opt/ghost/tools/blackhole.enc` | 5776B | Maintenance mode, user exclusions, honeypot stats |
| dns       | `/opt/ghost/tools/dns.enc`       | 2176B | pdnsutil failover on DNS node                     |
| filter    | `/opt/ghost/tools/filter.enc`    | 2944B | WAF rule management via ftool                     |
| ping      | `/opt/ghost/tools/ping.enc`      | 368B  | Health check - node identity + vitals JSON        |
| probe     | `/opt/ghost/tools/probe.enc`     | 1072B | System resource info - disk, memory, CPU, kernel  |
| service   | `/opt/ghost/tools/service.enc`   | 2352B | Whitelisted systemctl wrapper per-node            |
| wg-status | `/opt/ghost/tools/wg-status.enc` | 1696B | WireGuard tunnel status with peer handshake info  |
| integrity | `/opt/ghost/tools/integrity.enc` | 3392B | BLAKE2b file integrity scanner with baseline      |
| kernel    | `/opt/ghost/tools/kernel.enc`    | 1696B | Kernel module audit, rootkit indicator check      |
| network   | `/opt/ghost/tools/network.enc`   | 3360B | Firewall rule management and network status       |
| vault     | `/opt/ghost/tools/vault.enc`     | 2848B | Vault key integrity verification                  |
| backup    | `/opt/ghost/tools/backup.enc`    | 3888B | Backup pipeline (DB dump + code + wireguard)      |

### Encryption Gap Remediation - COMPLETED 2026-06-14
- All `users` plaintext PII columns NULLed, `_enc` columns populated
- `payments` encrypted columns added and migrated
- `support_messages`/`support_tickets` plaintext columns dropped
- `support_replies.message_enc` added and migrated
- `admins` table: pending audit (separate work item)

---

## 21. Three-Database Architecture (2026-06-20 Audit)

> **AUDIT FINDING**: The platform operates across 3 databases, not 2 as previously documented.

| Database     | Alias   | Host             | Node             | Tables | With Data | Purpose                                        |
| ------------ | ------- | ---------------- | ---------------- | ------ | --------- | ---------------------------------------------- |
| `store`      | noname  | 89.127.235.207   | Origin (558201)  | 112    | 27        | eCommerce, dashboard, security, reporting      |
| `engine`     | -       | 76.13.222.157    | Engine (669427)  | 6+     | 1 (13 rows) | Node telemetry, AI analysis, WG management   |
| ~~`s3rv1c3_admin`~~ | - | ~~194.36.190.111~~ | ~~Shield (414342)~~ | 0 | **DOES NOT EXIST** | ~~Tool logging~~ (BROKEN — redirected to Engine) |

### CRITICAL FIX: log_to_db.sh (TOOL_ID f6a9)

**Before (BROKEN):** `log_to_db.sh` targeted `s3rv1c3_admin` on Shield (10.69.69.1). Shield has **no MariaDB/MySQL** installed. Every tool execution log across all 8 nodes was silently failing.

**After (FIXED):** Redirected to `engine` on Engine (10.69.69.6). Table changed from `tool_executions` to `sec_tool_audit` with obfuscated columns per AUDIT_CODES.md.

### Origin Schema Drift

SCHEMA_MAP.md (June 11 dump) had 75 tables + 5 views. Live DB now has 112 tables — 37 tables were added since the dump. INSTALL_ALL.sql has been regenerated to include all 112 tables + 6 views.

### Orphaned Tables (~47)

Tables that exist in the database but are not referenced by any application code. These are infrastructure for designed-but-not-yet-wired features (marketing suite, fulfillment notifications, advanced security). They should be KEPT in the schema.

---

## 22. Mistral AI Security Analyzer (9d00)

> **Node**: Engine (669427) only  
> **Model**: Ollama Mistral 7B (local inference — no external API calls)

### Architecture

```
sec_tool_audit (Engine DB)
 | Tool execution telemetry from all 8 nodes
 v
C++ Engine Daemon (sysd-engine)
 | Aggregates, patterns, anomaly detection
 v
Ollama Mistral 7B (local)
 | Analyzes: execution patterns, auth failures,
 | config drift, network anomalies
 v
engine_alerts → engine_suggestions
 | Alert + remediation proposal
 v
Dashboard Review (Origin, via lantern)
 | Admin approve/reject workflow
```

### Engine DB Tables

| Table                | Purpose                                          | Status  |
| -------------------- | ------------------------------------------------ | ------- |
| `wg_config`          | WireGuard mesh configuration                     | Active (13 rows) |
| `sec_tool_audit`     | Obfuscated tool execution log                    | NEW     |
| `engine_alerts`      | Mistral AI security alerts                       | NEW     |
| `engine_suggestions` | Remediation proposals with approval workflow     | NEW     |
| `wg_port_allowlist`  | WireGuard port rotation pool                     | NEW     |
| `wg_subnet_pool`     | Per-store subnet allocation                      | NEW     |

### store_domains is a VIEW

Code references `store_domains` in `init.php`, `security.php`, `product.handler.php`. In the database it's a VIEW over the `stores` table (`SELECT *.columns FROM stores`). This is working correctly but not obvious from code.

---

*Last updated: 2026-06-21*
*Document version: 5.1*
*v5.1 (2026-06-21): Credential rotation engine deployed on Shield. hello.sh vault challenge-response gate deployed to 7/8 nodes. Heartbeat watchdog (stale detection + auto-lockdown) deployed on Shield with cron. Heartbeat receiver fixed (PHP DB updates, SSH key permissions). Runaway udevd killed on Proxy, NetworkManager disabled on Vault. DNS heartbeat timestamp fixed.*
*v5.0 (2026-06-21): Complete legacy purge. 8 active nodes with 6-digit IDs, zero legacy in DB or docs. Node naming consolidated (Proxy, Vault). All stale references removed fleet-wide. Topology, heartbeat, dispatch, and documentation fully aligned.*
*v4.3 (2026-06-21): Podman container docs corrected. Vault node role assigned. Heartbeat fleet-wide deployment completed.*
*v4.2 (2026-06-20): Full system audit — 3-database architecture documented, schema drift noted.*
*v4.0 (2026-06-19): Full Arsenal Audit — live SSH audit of all active nodes. Fleet SSH key permission fixed. PHP lint clean.*
*v3.9 (2026-06-19): Stripe checkout fixed. Filter heartbeat cron deployed. All nodes online.*
*v3.8 (2026-06-19): WG cleanup complete. Postfix confirmed. Payment bridge verified.*
*v3.6 (2026-06-18): Filter node deployed (91.229.239.29, Moldova). Traffic chain restored.*
*v3.2–3.5: Ghost tools (12/12). WG key separation. nftables. Podman blackholes. Dashboard tab repairs.*
