# SERVICE PIPELINES - NoName Platform
> **Version**: 2.1 (2026-06-20)
> **Location**: `Y:\documents\source\SERVICES.md` (LOCAL ONLY per OPS-09)
> **Rule**: This document describes every service pipeline end-to-end

---

## SYSTEM OVERVIEW

NoName runs across 3 database instances and 8 nodes with a clear separation of concerns:

### Architecture Philosophy

```
+-------------------------------------------------------------+
| ADMIN DASHBOARD (Origin)                                |
| Oversight - Configuration - Decision-making - Reporting |
| Human operator manages POLICY - not execution           |
+------------------------+------------------------------------+
 | lantern (172.28.41.0/30)
 v
+-------------------------------------------------------------+
| C++ ENGINE (Engine Node 669427)                      |
| Autonomous Infrastructure Management - AI-Assisted   |
| Ollama Mistral 7B - Node Health - Payment Processing |
| s3rv1c3agent data flows INTO engine_db for analysis  |
| Engine makes OPERATIONAL decisions autonomously      |
+-------------------------------------------------------------+
 ^ ^ ^
 HMAC POST HMAC POST HMAC POST
 | | |
 +-----+--+ +-----+--+ +-----+--+
 |s3rv1c3 | |s3rv1c3 | |s3rv1c3 | (RAM-resident on ALL nodes)
 |agent | |agent | |agent |
 +--------+ +--------+ +--------+
 Shield Proxy Filter ... etc
```

**Key distinction:**
- **s3rv1c3agent** = RAM-resident service on each node. Monitors, reports, executes. Sends ALL telemetry to Engine's C++ database (`engine_db`).
- **C++ Engine (sysd-engine)** = The autonomous brain. Uses Ollama Mistral 7B for anomaly detection and decision-making. Manages node health, rotations, payment processing WITHOUT dashboard involvement.
- **Admin Dashboard** = Human oversight layer. Reviews Engine's decisions, manages policy/configuration, handles business logic (orders, products, support). Does NOT execute infrastructure commands directly - delegates to Engine via lantern.

### Database Separation

| Database                   | Host         | Node             | Purpose                                                 | Managed By      |
| -------------------------- | ------------ | ---------------- | ------------------------------------------------------- | --------------- |
| `store` (historical alias: `noname`) | MariaDB 10.5 | Origin (558201)  | eCommerce, dashboard, security, reporting               | Dashboard PHP   |
| `engine_db`                | MariaDB 10.5 | Engine (669427)  | Node telemetry, agent data, AI analysis, autonomous ops | C++ Engine      |
| `payment_db`               | MariaDB 10.5 | Payment (714820) | Stripe webhooks, payment audit, PCI-isolated records    | Payment service |

### Connection Architecture
- **Origin DB**: PHP connects via PDO, credentials in `/var/www/private/db_app.secret` (user: k3m7z)
- **Engine DB**: C++ daemon connects locally, s3rv1c3agents POST data via HMAC-signed HTTP
- **Payment DB**: Isolated on Payment node, Engine connects via `rain` tunnel
- **Cross-node**: Origin->Engine queries traverse `lantern` tunnel (172.28.41.0/30)

### Subnet-Per-Store Architecture

Each store gets its own WireGuard subnet for traffic isolation (per NET-02):

| Store          | WG Subnet     | WG Interface | Proxy Bind IP | Kill Switch           |
| -------------- | ------------- | ------------ | ------------- | --------------------- |
| cleanpharma.cc | 10.69.70.0/30 | `pebble`      | 10.69.70.2    | `wg-quick down pebble` |
| cleanearth.cc  | 10.69.71.0/30 | `timber`      | 10.69.71.2    | `wg-quick down timber` |
| [future store] | 10.69.72.0/30 | `wg-s3`      | 10.69.72.2    | -                     |

**Benefits:**
- nginx vhosts bind to specific WG IPs - no server_name conflicts
- Firewalld zone per store - compromised store can't leak traffic
- Per-store rate limiting at WG interface level
- One-command store takedown without affecting others
- `stores.wg_subnet` column already exists in schema

---

## 1. PAYMENT PIPELINE

### Flow: Customer -> Stripe -> Fulfillment

```
Customer Browser
 |
 v
[1] Stripe.js (client-side tokenization)
 | Card data NEVER touches our servers (PCI SAQ-A)
 | Creates payment_method token
 v
[2] checkout.handler.php (Origin)
 | Validates: CSRF, session, cart, address
 | Creates order in `orders` table (status: pending)
 | Encrypts PII via SecureVault
 v
[3] PaymentBridge::submit() (Origin)
 | payment_bridge.php
 | HMAC-signs the request
 | POST to Engine via lantern (172.28.41.2:8443)
 v
[4] C++ Engine Daemon (Engine, port 8301/8443)
 | Validates HMAC signature
 | Calls Stripe API (charge/payment_intent)
 | Stripe API key IP-restricted to Engine public IP

> **CORRECTION (2026-07-02)**: Stripe API key is stored ENCRYPTED on Payment node (714820) at `/opt/payment/config/.stripe_key`. The decryption key is on Vault (869377), accessed via the dedicated Payment↔Vault WG tunnel. The key is IP-locked to Payment's public IP. See Section 10 for authoritative Stripe architecture. Engine does NOT hold an active Stripe key.
 | Records in payment_audit DB
 v
[5] Engine Response -> Origin
 | Returns: {success, stripe_id, amount}
 | Origin updates order status -> paid (2002)
 | Creates payment_ledger entry
 v
[6] Fulfillment Trigger
 | Order moves through pipeline:
 | pending(2001) -> paid(2002) -> processing(2003) ->
 | packed(2004) -> shipped(2005) -> delivered(2006)
 | Each stage: fulfillment_stages table + email trigger
 v
[7] Customer Notification
 | Email queued in email_queue (status 8001)
 | Engine picks up via SMTP bridge
 | Postfix delivers
```

### Payment Methods

| Method        | Handler                                   | Engine Involvement            | Status             |
| ------------- | ----------------------------------------- | ----------------------------- | ------------------ |
| Stripe        | `PaymentBridge::submit('stripe', ...)`    | Full - Stripe API call        | Active             |
| e-Transfer    | `PaymentBridge::submit('etransfer', ...)` | Partial - Gmail inbox polling | Active             |
| Crypto (XMR)  | -                                         | -                             | Removed 2026-06-16 |
| Crypto (USDT) | -                                         | -                             | Removed 2026-06-16 |

### Key Files

| File                   | Location                        | Purpose                                   |
| ---------------------- | ------------------------------- | ----------------------------------------- |
| `payment_bridge.php`   | `/var/www/public/system/`       | PaymentBridge class - HMAC POST to Engine |
| `checkout.handler.php` | `/var/www/public/api/handlers/` | Checkout flow - validates, creates order  |
| `stripe.secret.php`    | `/var/www/private/`             | Stripe secret key (never in code)         |
| Engine daemon          | `/opt/apache/engine/bin/`       | C++ binary, port 8301/8443                |

### Security Controls
- Stripe API key IP-restricted to Engine public IP only

> **CORRECTION (2026-07-02)**: Stripe API key is stored ENCRYPTED on Payment node (714820) at `/opt/payment/config/.stripe_key`. The decryption key is on Vault (869377), accessed via the dedicated Payment↔Vault WG tunnel. The key is IP-locked to Payment's public IP. See Section 10 for authoritative Stripe architecture. Engine does NOT hold an active Stripe key.
- Payment data only on Engine - never touches Origin, Proxy, Filter, DNS
- Card numbers tokenized by Stripe.js (client-side) - PCI SAQ-A
- PaymentBridge uses HMAC-SHA256 on every request
- rain tunnel (172.23.83.0/30) dedicated to payment traffic only

---

## 2. EMAIL PIPELINE

### Flow: Trigger -> Queue -> Engine -> Delivery

```
Email Trigger (Origin)
 | Order status change, verification, support reply
 v
[1] email_queue INSERT (Origin DB)
 | recipient_enc: SecureVault encrypted
 | body_enc: SecureVault encrypted
 | status_code: 8001 (QUEUED)
 v
[2] Email Bridge (Origin -> Engine)
 | email_bridge.php polls queue
 | HMAC-authenticated POST to Engine
 | Via lantern tunnel
 v
[3] Engine SMTP Relay (Engine)
 | Postfix (port 587, outbound only)
 | Dovecot (IMAP 110/143 for inbox)
 | OpenDKIM (port 8891 for signing)
 | Domain: configured per store
 v
[4] Delivery
 | email_queue.status_code -> 8003 (SENT)
 | email_log entry created
 | Bounce handling -> status 8005
 |
[FALLBACK] email_bridge_fallback.php
 | If Engine unreachable, attempts direct SMTP
 | from Origin (degraded mode)
```

### Email Types

| Type             | Trigger           | Template         | Priority   |
| ---------------- | ----------------- | ---------------- | ---------- |
| `account_verify` | Registration      | verify.html      | High (5)   |
| `order_confirm`  | Payment confirmed | order.html       | High (5)   |
| `shipping`       | Order shipped     | shipping.html    | Normal (5) |
| `delivery`       | Order delivered   | delivery.html    | Normal (5) |
| `password_reset` | Reset request     | reset.html       | High (5)   |
| `review_request` | Post-delivery     | review.html      | Low (3)    |
| `fulfillment`    | Stage change      | fulfillment.html | Normal (5) |
| `support`        | Support reply     | support.html     | High (5)   |
| `exclusion`      | Self-exclusion    | exclusion.html   | High (5)   |

### Key Files

| File                        | Location                            | Purpose                       |
| --------------------------- | ----------------------------------- | ----------------------------- |
| `email_bridge.php`          | `/var/www/dashboard/modules/fleet/` | Queue processor, Engine relay |
| `email_bridge_fallback.php` | `/var/www/dashboard/modules/fleet/` | Degraded-mode direct SMTP     |
| `email_queue` table         | Origin DB                           | Encrypted email queue         |
| `email_log` table           | Origin DB                           | Send history                  |

---

## 3. DNS PIPELINE

### Architecture

```
Internet DNS Query
 v
[1] PowerDNS (DNS Node 199335)
 | Authoritative for: cleanearth.cc, cleanpharma.cc
 | Port 53 (UDP + TCP)
 | DNSSEC signed
 v
[2] Zone Management
 | Dashboard -> DNS tab
 | API: mesh-api.php?action=dns-update
 | Changes pushed via Shield relay
 v
[3] Record Types
 | A: Points to Proxy (107.189.23.194)
 | MX: Points to Engine mail
 | TXT: SPF, DKIM, DMARC records
 | CNAME: www -> apex
```

### DNS Security
- DNS node (199335) rebuilt fresh 2026-06-20 - Ubuntu 24.04 on Njalla (1 Core / 1.5GB RAM / 15GB Disk / 1.5TB Traffic)
- Hostname: p3plcpnl0842 (obfuscated - appears as GoDaddy cPanel server)
- Zone transfers disabled (single authoritative server)
- DNSSEC for tamper detection
- DNS-over-HTTPS not exposed (internal only)

### Adding a New Store
1. Add DNS records (A -> Proxy, MX -> Engine)
2. Add store entry in `stores` table
3. Configure nginx vhost on Proxy
4. Generate SSL cert via Let's Encrypt
5. Update store_domains view

---

## 4. Service Daemon (TOOL_ID: a1f0) - Per-Node Service Agent

### What It Is
The service daemon (TOOL_ID: a1f0) is a **bash daemon** (418 lines, version 4.0.0-hardened) that runs on ALL fleet nodes in the per-node disguised tool directory. It is the primary mesh command executor and heartbeat agent. Despite the name suggesting a binary, it's a hardened bash script.

**Systemd Unit**: per-node disguised systemd unit (see TOOLING_MAP.md)
**Runs as**: user `mail-svc`
**Environment**: `NODE_ID=<node_int>`, `ENGINE_URL=http://10.69.69.6:8301`

### Main Loop (every 10s)

```
while true:
 1. heartbeat() - POST HMAC-signed heartbeat to Engine
 2. process_engine_commands() - Poll Engine for commands via /cmd/recv
 3. autonomous_loop() - Run critical tools if Engine unreachable >300s
 4. self_integrity_check() - SHA-256 verify own tooling files every 60s
 sleep 10
```

### Strict Whitelist (39 commands ONLY)

The agent uses an explicit case/switch - NEVER passes user input to `bash -c`. Only these commands are permitted:

```
hb_init, hb_send, hb_send_all,
dm_init, dm_check, dm_loop,
ca_init, ca_plant, ca_monitor,
kr_init, kr_rotate, kr_rotate_all,
ld_status, ld_activate, ld_deactivate, ld_escalate, ld_kill_switch,
bs_init, bs_generate, bs_install,
fm_init, fm_check, fm_monitor,
lm_init, lm_monitor,
pr_rotate, per_rotate, cr_rotate, fr_rotate,
ww_init, ww_monitor,
health_poll, event_send, vault_fetch, payment_check,
cmd_recv, decrypt, ai_query
```

Each command maps to a specific tool in the per-node disguised tool directory (paths are per-node, see TOOLING_MAP.md):
| Command                     | Maps To (TOOL_ID)           | Purpose                         |
| --------------------------- | --------------------------- | ------------------------------- |
| `hb_send`                   | TOOL_ID: a1f0-hb (svcbeat)  | Heartbeat to Engine             |
| `health_poll`               | TOOL_ID: a1f0-hp (pollmail) | Node health check               |
| `vault_fetch <ip>`          | TOOL_ID: a1f0-vf (getmail)  | Fetch from vault (IP validated) |
| `event_send`                | TOOL_ID: a1f0-es (logmail)  | Log security event              |
| `cmd_recv`                  | TOOL_ID: a1f0-cr (cmdmail)  | Receive mesh command            |
| `ai_query <q>`              | TOOL_ID: a1f0-aq (askmail)  | Ollama AI query                 |
| `payment_check <id> <type>` | TOOL_ID: a1f0-pc (paymail)  | Payment verification            |
| `decrypt <path>`            | TOOL_ID: a1f0-dc (unmail)   | Decrypt vault file              |

### Lockdown System

3 consecutive HMAC failures trigger lockdown level 2:
- Creates `/var/run/s3rv1c3agent.lockdown`
- ONLY `ld_*` and `hb_send` commands execute during lockdown
- Level 3: Self-integrity check failure -> all tool execution stops

### Autonomous Mode

When Engine is unreachable for >300 seconds:
1. Creates `/var/run/s3rv1c3agent.autonomous` flag
2. Creates `/run/se/state/agent_rotating` token (15 min TTL)
3. Stops polling Engine for commands
4. Self-executes SAFE maintenance only: `hb_send, dm_check, fm_check, lm_monitor, health_poll`
5. **Does NOT execute `kr_rotate_all` or `pr_rotate`** — credential rotation requires Engine coordination or human approval
6. Resumes normal operation when heartbeat succeeds
7. Removes `/run/se/state/agent_rotating` token and `/var/run/s3rv1c3agent.autonomous` flag on resume

### Per-Node Agent Configuration

| Node                 | Disguised Name          | Systemd Unit                    | Specific Duties                            |
| -------------------- | ----------------------- | ------------------------------- | ------------------------------------------ |
| **Shield** (414342)  | `httpd-prefork-mgr`     | `httpd-prefork-mgr.service`     | WG hub, relay dispatch, HMAC validation    |
| **Origin** (558201)  | `php-opcache-gc`        | `php-opcache-gc.service`        | DB management, fleet dispatch, email queue |
| **Proxy** (244939)   | `nginx-cache-gc`        | `nginx-cache-gc.service`        | Reverse proxy, cache, SSL certs            |
| **Filter** (333941)  | `nft-chainmon`          | `nft-chainmon.service`          | WAF/nftables, blackhole, DDoS              |
| **DNS** (199335)     | `unbound-resolvd`       | `unbound-resolvd.service`       | PowerDNS, zone health, DNSSEC              |
| **Engine** (669427)  | `postfix-qmgr-aux`      | `postfix-qmgr-aux.service`      | C++ daemon, payment, SMTP                  |
| **Vault** (869377)   | `stunnel-cert-refresh`  | `stunnel-cert-refresh.service`  | Credential vault, key distribution         |
| **Payment** (714820) | `journal-coalesce`      | `journal-coalesce.service`      | Stripe API, payment processing             |

### Three Communication Pathways

#### Pathway 1: SSH Relay Chain (ShieldRelay - `shield_relay.php`)
Origin -> SSH (fleet.key) -> Shield -> SSH (reader.key) -> Spoke node

Each node has **disguised runner scripts**:
| Node   | Primary Runner              | Backup Runner                                             |
| ------ | --------------------------- | --------------------------------------------------------- |
| origin | `/root/nginx-maint.sh`      | `/usr/lib/systemd/scripts/nginx-cache-gc.sh`              |
| shield | `/root/coredump-handler.sh` | `/usr/lib/systemd/coredump.conf.d/coredump-gc.sh`         |
| dns    | `/root/resolved-sync.sh`    | `/usr/lib/systemd/resolved.conf.d/stub-cache-gc.sh`       |
| proxy  | `/root/networkd-diag.sh`    | `/usr/lib/systemd/network/carrier-bond-check.sh`          |
| filter | `/root/journal-compact.sh`  | `/usr/lib/systemd/journal-upload.conf.d/vacuum-rotate.sh` |
| engine | `/root/oomd-report.sh`      | `/usr/lib/systemd/oomd.conf.d/pressure-cleanup.sh`        |

#### Pathway 2: File-Based Relay (`fleet-relay.sh` on Origin)
PHP (ShieldRelay) writes `.req` JSON to `/var/lib/php/sessions/.pending/`:
1. fleet-relay.sh polls directory every 200ms
2. Reads JSON: `{id, tool, args, node, ts, hmac}`
3. Verifies HMAC against hello session at `/var/lib/php/sessions/.gc_state`
4. Executes via runner: `bash /root/nginx-maint.sh exec-raw <tool> <args>`
5. Writes `.res` response file
6. PHP polls for `.res` (50ms intervals)

#### Pathway 3: REST API (`mesh-api.php` on Origin)
Node-to-Origin reporting via HTTPS. Auth: WG IP (10.69.69.x) OR HMAC signature.

| Endpoint                | Method   | Purpose                     |
| ----------------------- | -------- | --------------------------- |
| `node-health`           | GET      | Aggregate fleet health      |
| `tool-status`           | GET/POST | Tool execution history      |
| `security-events`       | GET      | Security event feed         |
| `lockdown-state`        | GET      | Lockdown status per node    |
| `canary-triggers`       | GET      | Canary token detections     |
| `bash-integrity`        | GET      | File integrity results      |
| `key-rotations`         | GET      | Key rotation history        |
| `report-tool-run`       | POST     | Node reports execution      |
| `report-heartbeat`      | POST     | Node sends heartbeat        |
| `report-event`          | POST     | Node reports security event |
| `report-watchdog-alert` | POST     | Watchdog tamper alert       |

### Heartbeat System - Three Layers

| Layer             | File                                         | Interval   | Target                      | Method       |
| ----------------- | -------------------------------------------- | ---------- | --------------------------- | ------------ |
| svcbeat           | Per-node tool directory (see TOOLING_MAP.md) | 30s        | Engine 10.69.69.6:8301/a/hb | HMAC POST    |
| mesh_heartbeat    | Per-node tool directory (see TOOLING_MAP.md) | 60s (cron) | Origin mesh-api.php         | HMAC POST    |
| FleetOrchestrator | `fleet_orchestrator.php`                     | Passive    | Updates `nodes` table       | PHP receiver |

**Freshness thresholds**: <120s = green, <600s = yellow, >600s = red/offline

### SE Client Library (`se_client.sh`)
Shared by ALL `*mail` tools (askmail, cmdmail, getmail, logmail, paymail, pollmail, unmail):
- Reads vault key from `/dev/shm/vault.key` (memory-only, never on disk)
- Signs: `HMAC-SHA256(timestamp:nonce:METHOD+PATH+BODY, vault_key)`
- Headers: `X-SE-T` (timestamp), `X-SE-N` (nonce), `X-SE-S` (signature), `X-SE-I` (node ID)
- Retry: exponential backoff, max 3 retries

### Session Challenge System (TOOL_ID: b2e0)
Zero-trust session manager (298 lines) at per-node disguised path (see TOOLING_MAP.md):
1. `check-in <purpose> <scope> <ttl>` - creates session (default TTL: 900s, max: 7200s)
2. Creates JSON at `/opt/binary/session/active/<session_id>.json`
3. Sets `HELLO_SESSION_ID` env var
4. All tools call `require_active_session` to validate
5. `check-out` or TTL expiry closes session
6. Full audit trail in `/opt/binary/session/history/`

### Command Lifecycle (End-to-End)

```
[1] Dashboard UI -> control_command_create.php
 Admin selects node, command, priority (8010-8013)
 |
[2] FleetOrchestrator::executeRestricted()
 Whitelist check (systemctl, wg, nginx, uptime, df, free, ps, etc.)
 Injection detection via regex
 |
[3] ShieldRelay::exec()
 Origin -> SSH -> Shield -> SSH -> Target node
 For origin: local relay via file-based IPC
 |
[4] Target node runner script
 exec-raw <tool> <args> dispatched to handler
 |
[5] s3rv1c3agent (independent path)
 Polls Engine /cmd/recv for HMAC-signed commands
 Verifies whitelist, executes, records to DB
 |
[6] Result return
 node_commands.status -> 5013 (CMD_COMPLETED)
 sec_tool_runs entry created
 Dashboard updates via AJAX polling
```

---

## 5. C++ ENGINE DAEMON - Autonomous Infrastructure Brain

### Role
The C++ Engine (`sysd-engine`) is the **autonomous management brain** of the fleet. It receives ALL telemetry from s3rv1c3agents across every node, stores it in `engine_db`, analyzes it using Ollama Mistral 7B, and makes operational decisions WITHOUT requiring dashboard/human intervention.

**The dashboard OVERSEES - the Engine OPERATES.**

### Architecture

```
Engine Node (669427)
 |
 |-- [sysd-engine] (disguised as systemd-nscd)
 | Binary: /usr/local/sbin/sysd-engine
 | Launch: --node-id 669427 --role f --log-level INFO
 | --key /var/lib/sysc-engine/master.key
 | --config /opt/sysc/engine/config/engine.json
 | Listens: 127.0.0.1:8301 (mesh API - localhost only)
 | Listens: 0.0.0.0:8443 (HTTPS PaymentBridge - WG only)
 | DB: engine_db (MariaDB, local socket)
 |
 |-- AUTONOMOUS SERVICES:
 | |
 | |-- [Node Management] - Receives ALL s3rv1c3agent data
 | | Heartbeat ingestion (every 30s from all nodes)
 | | Health analysis: CPU, memory, disk, WG peers
 | | Anomaly detection via Ollama Mistral
 | | Auto-rotation triggers (passwords, ports)
 | | Lockdown escalation decisions
 | | Command dispatch to agents via /cmd/recv
 | |
 | |-- [AI Analysis] - Ollama Mistral 7B (127.0.0.1:11434)
 | | Podman container, local inference ONLY
 | | Analyzes: security events, traffic patterns, anomalies
 | | Generates: rotation recommendations, threat assessments
 | | Queries via: s3rv1c3agent `ai_query` command
 | |
 | |-- [PaymentBridge] endpoint (:8443)
 | | Stripe charge/intent/refund via rain tunnel
 | | e-Transfer verification (Gmail inbox polling)
 | | HMAC validation on every request
 | |
 | |-- [Fulfillment API] (:8301)
 | | Order stage transitions (2001->2006)
 | | Tracking number management
 | | Inventory updates -> Origin DB via lantern
 | |
 | +-- [Email SMTP Relay]
 | Postfix (outbound :587)
 | OpenDKIM signing (:8891)
 | Dovecot inbox (:110, :143)
 |
 +-- SECURITY:
 |-- HMAC validation on ALL inbound requests
 |-- WG-only access - port 8301 is localhost-only
 |-- Stripe key in SecureVault (.vault files)
 |-- Decryption key sourced from Vault node (869377)
 +-- Audit trail in engine_db
```

### Engine Decision Flow (Autonomous)

```
s3rv1c3agent (any node) -> HMAC POST -> Engine :8301
 |
 v
engine_db: INSERT heartbeat/event/alert
 |
 v
Engine analysis loop (every 60s):
 |-- Check: any node missed >5 heartbeats? -> trigger failover
 |-- Check: watchdog alert received? -> query Mistral for threat assessment
 |-- Check: HMAC failure count > threshold? -> lockdown escalation
 |-- Check: disk/cpu critical? -> alert dashboard + auto-remediate
 +-- Check: payment failure pattern? -> pause payment bridge, alert admin
 |
 v
Autonomous actions (no dashboard needed):
 |-- Send rotation command to affected agent
 |-- Escalate lockdown level on compromised node
 |-- Quarantine suspicious agent (stop accepting its heartbeats)
 |-- Log decision + reasoning to engine_db.decisions table
 +-- Push summary to Dashboard via lantern (for human review)
```

### engine_db Schema (Node Management)

| Table                   | Purpose                                          |
| ----------------------- | ------------------------------------------------ |
| `agent_heartbeats`      | All heartbeat data from all nodes (30s interval) |
| `agent_commands`        | Commands dispatched to agents + responses        |
| `agent_events`          | Security events reported by agents               |
| `agent_alerts`          | Watchdog and integrity alerts                    |
| `engine_decisions`      | Autonomous decisions with AI reasoning           |
| `engine_rotations`      | Password/port rotations triggered by engine      |
| `ai_queries`            | Ollama Mistral query log + responses             |
| `ai_threat_assessments` | AI-generated threat analysis                     |
| `payment_transactions`  | Stripe/e-Transfer audit trail                    |
| `payment_webhooks`      | Stripe webhook log                               |

### Engine Vault System
The Engine stores secrets as `.vault` files encrypted with AES-256-CBC:
- `stripe.vault` - Stripe secret key (DEPRECATED — Stripe vault file is on Payment node, not Engine. See Section 10.)
- `origin_mysql.vault` - Origin DB credentials
- `gmail_address.vault` - Notification sender
- `gmail_etransfer.vault` - e-Transfer inbox credentials

Decryption key: `/opt/apache/engine/config/.k7x9m2/v.dat` (sourced from Vault node)

---

## 6. ECOMMERCE SYSTEM (PUBLIC SITE)

### Architecture

```
Customer Browser
 v
[Proxy] nginx reverse proxy (244939)
 | SSL termination, CF headers, cache
 v
[Filter] WAF / ModSecurity (333941)
 | nftables rules, blackhole container
 | Request inspection, bot detection
 v
[Shield] VPN Hub / Bastion (414342)
 | WG mesh hub, HMAC validation
 | Relay to Origin
 v
[Origin] Web Server (558201)
 | PHP-FPM, MariaDB
 | Serves: public storefront + admin dashboard
 |
 |-- Public Site (/var/www/public/)
 | Single entry: index.php -> bootstrap
 | Product listing: listings.php (ONE file, Rule COM-05)
 | Cart: 100% client-side (localStorage)
 | Checkout: checkout.handler.php -> PaymentBridge
 | User auth: login/register/verify
 | DOM: fully obfuscated (COD-01)
 |
 +-- Admin Dashboard (/var/www/dashboard/)
 WG-only access (NET-04)
 3-phase auth: PIN -> user/pass -> token sequence
 RBAC: 4 tiers (Viewer/Operator/Admin/Owner)
 Tabs: Fleet, Security, Orders, Products, Support, ...
```

### Dashboard System (Separate from Store)

The dashboard is a completely separate application from the public store:

| Aspect    | Public Store                  | Admin Dashboard                |
| --------- | ----------------------------- | ------------------------------ |
| Path      | `/var/www/public/`            | `/var/www/dashboard/`          |
| Access    | Internet (via proxy chain)    | WG mesh only                   |
| Auth      | Customer login (email/pass)   | 3-phase admin auth             |
| DB user   | `k3m7z` (read-heavy)          | `k3m7z` (full access)          |
| RBAC      | None (customer)               | 4-tier (Viewer->Owner)         |
| DOM       | Obfuscated                    | Internal (not obfuscated)      |
| Bootstrap | `public/system/bootstrap.php` | `dashboard/core/bootstrap.php` |

---

## 7. ENCRYPTION AT REST

### Encryption Layers

| Layer                 | Algorithm                       | Key Location                              | Data Type                        |
| --------------------- | ------------------------------- | ----------------------------------------- | -------------------------------- |
| Customer PII          | XSalsa20-Poly1305 (SecureVault) | `/dev/shm/vault.key` (Shield)             | Names, emails, addresses, phones |
| Payment data          | AES-256-CBC (Engine vault)      | `/opt/apache/engine/config/.k7x9m2/v.dat` | Stripe tokens, amounts           |
| Credential records    | AES-256-CBC + PBKDF2            | Master rotation key                       | Passwords, SSH keys, MySQL creds |
| Static store data     | Hex-encoded AES-256             | `/var/www/shared/static.key`              | Product names, categories        |
| Dashboard credentials | AES-256-GCM                     | Vault master key                          | Admin user data                  |
| Inter-node HMAC       | SHA-256                         | Per-service HMAC key                      | Request authentication           |
| WireGuard             | ChaCha20-Poly1305               | Per-node private keys                     | All inter-node traffic           |

### SecureVault Domain Subkeys (encryption.php)

> **Source of truth**: `Y:\5\var\www\shared\encryption.php` - class `SecureVault`

| ID  | Domain    | Constant         | Used For                                             |
| --- | --------- | ---------------- | ---------------------------------------------------- |
| 0   | `default` | `DOMAIN_DEFAULT` | Fallback, general-purpose encryption                 |
| 1   | `user`    | `DOMAIN_USER`    | User-generated content, profiles, reviews            |
| 2   | `order`   | `DOMAIN_ORDER`   | Order details, tracking numbers, notes               |
| 3   | `review`  | `DOMAIN_REVIEW`  | Product reviews, ratings content                     |
| 4   | `log`     | `DOMAIN_LOG`     | Security events, threat intel, IP addresses          |
| 5   | `payment` | `DOMAIN_PAYMENT` | Stripe IDs, payment references                       |
| 6   | `faq`     | `DOMAIN_FAQ`     | FAQ content, help text                               |
| 7   | `product` | `DOMAIN_PRODUCT` | Product names, descriptions, categories, stores      |
| 8   | `support` | `DOMAIN_SUPPORT` | Support ticket content, terms, legal text, blacklist |
| 9   | `pii`     | `DOMAIN_PII`     | Customer PII: names, emails, phones, addresses       |

### Static Key Encryption (store-generated data)
- Key file: `/var/www/shared/static.key` (outside public root)
- Permissions: `600` (root only)
- Format: 64-character hex string (256-bit key)
- Used for: Product titles, category names, FAQ content, testimonials
- NOT used for: Status codes, boolean flags, numeric IDs, timestamps

---

## 8. "maid" - Context-Aware Support Response Service

### Overview
"maid" is the disguised codename for the Engine's Ollama Mistral 7B inference service.
It generates draft support ticket responses based on full customer context.

> **ZERO references** to "ollama", "mistral", "AI", or "LLM" exist in any PHP code.
> The word "maid" is the only identifier.

### Flow
```
Admin clicks "Generate Draft Response" in Support tab
 |
 v
maidAssembleContext() - gathers NON-BILLING context:
 |-- Ticket subject + full thread (decrypted)
 |-- Customer: status_code, member_since, last_login, prior ticket count
 |-- Orders: dates + statuses ONLY (no amounts, no payment methods)
 |-- Products linked to orders (names only)
 +-- Shipments: status if applicable
 |
 v
HMAC-signed POST to http://10.69.69.6:8301/a/maid
 (via lantern or thread tunnel)
 |
 v
Engine: Ollama processes context -> returns response + confidence
 |
 v
Draft stored in support_replies:
 sender_type = 'system'
 is_maid_draft = 1
 message = Encryption::encrypt(response, 'support')
 |
 v
Admin reviews draft -> Approve (is_maid_draft=0) or Reject (DELETE)
 RBAC: admin_level <= 9011 (ADMIN) required
```

### Files
| File                                 | Purpose                                                        |
| ------------------------------------ | -------------------------------------------------------------- |
| `modules/support/maid_response.php`  | Core: context assembly, Engine dispatch, draft storage         |
| `modules/support/thread_replies.php` | UI: Generate/Approve/Reject buttons, draft rendering           |
| `tabs/support.php`                   | AJAX action handlers: maid_generate, maid_approve, maid_reject |
| `system/maid_migration.sql`          | Schema: is_maid_draft, maid_confidence, maid_approved_by       |

### Status Codes
| Code | Constant      | Badge   | Meaning                     |
| ---- | ------------- | ------- | --------------------------- |
| 8020 | MAID_DRAFT    | warning | Draft awaiting admin review |
| 8021 | MAID_APPROVED | success | Draft approved and sent     |
| 8022 | MAID_REJECTED | danger  | Draft rejected by admin     |

### Security
- **NEVER** accesses billing data (amounts, payment_method, stripe_payment_id)
- All maid responses encrypted with `support` domain (8) at rest
- Requires active admin session with RBAC >= ADMIN (9011)
- Engine endpoint authenticated via HMAC-SHA256 (same pattern as se_client.sh)

---

## 9. EVENT BUS - File Operations Agent

PHP CLI scripts at `/opt/noname/scripts/` - described as "S3RV1C3agent PHP replacement":

| Script                  | Usage       | Purpose                                 |
| ----------------------- | ----------- | --------------------------------------- |
| `event_bus <src> <dst>` | Move file   | Moves files between allowed directories |
| `event_bus_del <path>`  | Delete file | Deletes files in allowed directories    |

**Allowed prefixes**: `/opt/noname/data/`, `/var/www/public/uploads/`, `/tmp/`
**Path traversal protection**: `realpath()` validation prevents escape

---

## 9. KEY FILE PATHS REFERENCE

### Agent & Tools
| Component                           | Path / Reference                          |
| ----------------------------------- | ----------------------------------------- |
| Service daemon (TOOL_ID: a1f0)      | Per-node (see TOOLING_MAP.md)             |
| Service inspector (TOOL_ID: a1f1)   | Per-node (see TOOLING_MAP.md)             |
| Systemd unit                        | Per-node (see TOOLING_MAP.md)             |
| Fleet relay (origin)                | `/opt/cleanpharma/scripts/fleet-relay.sh` |
| Session manager (TOOL_ID: b2e0)     | Per-node (see TOOLING_MAP.md)             |
| Session teardown (TOOL_ID: b2e1)    | Per-node (see TOOLING_MAP.md)             |
| SE client library (TOOL_ID: c3f0)   | Per-node (see TOOLING_MAP.md)             |
| Heartbeat sender (TOOL_ID: a1f0-hb) | Per-node (see TOOLING_MAP.md)             |
| Mesh heartbeat (TOOL_ID: a1f0-mh)   | Per-node (see TOOLING_MAP.md)             |
| Common library                      | `/opt/fleet/scripts/common.sh`            |
| DB logger                           | `/opt/fleet/scripts/log_to_db.sh`         |
| Runtime health                      | `/opt/fleet/scripts/runtime-health.sh`    |
| Event bus (move)                    | `/opt/noname/scripts/event_bus`           |
| Event bus (delete)                  | `/opt/noname/scripts/event_bus_del`       |

### Engine
| Component          | Path                                                |
| ------------------ | --------------------------------------------------- |
| Engine binary      | `/usr/local/sbin/sysd-engine`                       |
| Engine config      | `/opt/sysc/engine/config/engine.json`               |
| Engine service     | Per-node (see TOOLING_MAP.md)                       |
| Engine source      | `/opt/apache/engine/build-v2/compliant/engine/src/` |
| Engine master key  | `/var/lib/sysc-engine/master.key`                   |
| Vault key (memory) | `/dev/shm/vault.key`                                |

### PHP Application
| Component          | Path                                                |
| ------------------ | --------------------------------------------------- |
| Fleet orchestrator | `/var/www/dashboard/core/fleet_orchestrator.php`    |
| Shield relay       | `/var/www/dashboard/core/shield_relay.php`          |
| Mesh API           | `/var/www/dashboard/api/mesh-api.php`               |
| Gateway sync       | `/var/www/dashboard/system/gateway_sync.php`        |
| Status codes class | `/var/www/shared/status_codes.php`                  |
| Payment bridge     | `/var/www/public/system/payment_bridge.php`         |
| Checkout handler   | `/var/www/public/api/handlers/checkout.handler.php` |

### Security Keys
| Component                | Path                                      |
| ------------------------ | ----------------------------------------- |
| Fleet SSH key            | `/var/www/private/keys/fleet`             |
| Relay SSH key            | `/var/lib/gdns/.k/reader.key`             |
| Vault key (memory)       | `/dev/shm/vault.key`                      |
| HMAC master key          | `/var/lib/sysc-engine/master.key`         |
| Static encryption key    | `/var/www/shared/static.key`              |
| Engine vault files       | `/opt/apache/engine/config/.k7x9m2/`      |
| Engine vault decrypt key | `/opt/apache/engine/config/.k7x9m2/v.dat` |

### IPC & Session
| Component             | Path                               |
| --------------------- | ---------------------------------- |
| Relay command queue   | `/var/lib/php/sessions/.pending/`  |
| Hello session log     | `/var/lib/php/sessions/.gc_state`  |
| Hello active sessions | `/opt/binary/session/active/`      |
| Hello session history | `/opt/binary/session/history/`     |
| Lockdown flag         | `/var/run/s3rv1c3agent.lockdown`   |
| Autonomous flag       | `/var/run/s3rv1c3agent.autonomous` |

### Payment Node (714820)
| Component        | Path                                  |
| ---------------- | ------------------------------------- |
| Payment endpoint | `/opt/payment/api/endpoint.php`       |
| Stripe API key   | `/opt/payment/config/.stripe_key`     |
| Webhook secret   | `/opt/payment/config/.webhook_secret` |
| Payment logs     | `/opt/payment/logs/`                  |
| Payment service  | `payment-api.service` (systemd)       |

---

## 10. PAYMENT NODE - Stripe Payment Processor (714820)

### Overview
The Payment node (714820) is a **PCI-isolated Stripe payment processor**. It hosts the Stripe API key and processes all charge, refund, and webhook operations. No other node in the fleet holds Stripe credentials. The Engine node (669427) is the ONLY node that can reach Payment, via the dedicated `rain` WireGuard tunnel.

### Architecture

```
Engine Node (669427)
 |
 | rain tunnel (172.23.83.0/30, port 52891)
 | HMAC-SHA256 signed requests
 |
 v
Payment Node (714820)
 |
 |-- [payment-api.service] (systemd)
 | Local Stripe API: 127.0.0.1:8443
 | Accepts connections ONLY from rain tunnel IP (172.23.83.1)
 | HMAC-SHA256 validation on ALL inbound requests
 |
 |-- ROUTES:
 | |-- /pay/charge - Create Stripe PaymentIntent / charge
 | |-- /pay/refund - Process refund via Stripe
 | |-- /pay/webhook - Stripe webhook receiver (signature verified)
 | |-- /pay/status - Check payment status by transaction ID
 | +-- /pay/verify - Verify payment integrity (HMAC + amount match)
 |
 |-- CREDENTIALS (never leave this node):
 | |-- /opt/payment/config/.stripe_key - Stripe secret key (IP-locked to Payment public IP)
 | +-- /opt/payment/config/.webhook_secret - Stripe webhook signing secret
 |
 |-- LOGGING:
 | +-- /opt/payment/logs/YYYY-MM-DD.jsonl - Daily transaction logs (append-only)
 | Fields: timestamp, action, stripe_id, amount, currency, status, hmac_valid, source_ip
 |
 +-- SECURITY:
 |-- Stripe API key IP-LOCKED to Payment node's public IP via Stripe Dashboard
 |-- 127.0.0.1:8443 binds localhost ONLY - not reachable from public interface
 |-- Firewall: only rain tunnel IP (172.23.83.1) can reach port 8443
 |-- HMAC-SHA256 required on every API call
 |-- All payment data encrypted at rest (AES-256)
 |-- All payment data encrypted in transit (WireGuard ChaCha20-Poly1305)
 +-- NO Stripe credentials stored on ANY other node
```

### Payment Flow (Engine -> Payment)

```
Origin (558201)
 | PaymentBridge::submit() - HMAC-signed POST
 | via lantern (172.28.41.0/30)
 v
Engine (669427)
 | Validates HMAC from Origin
 | Forwards payment request to Payment node
 | via rain tunnel (172.23.83.0/30, port 52891)
 v
Payment (714820)
 | Validates HMAC from Engine
 | Calls Stripe API (key is IP-locked to this node)
 | Logs transaction to /opt/payment/logs/YYYY-MM-DD.jsonl
 | Returns result to Engine
 v
Engine (669427)
 | Records in engine_db.payment_transactions
 | Returns result to Origin via lantern
 v
Origin (558201)
 | Updates order status in noname DB
 | Dashboard reads payment status from Engine DB
 | Dashboard NEVER contacts Payment node directly
```

### Key Security Properties
- Stripe API key is **IP-LOCKED** - only works from Payment node's public IP
- Payment endpoint (127.0.0.1:8443) accepts requests **ONLY** from rain tunnel IP
- **NO** Stripe credentials exist on any other node (not Engine, not Origin, not Shield)
- Engine (669427) is the **ONLY** node that can reach Payment via rain tunnel
- Dashboard (Origin) reads payment status from Engine DB - **never contacts Payment directly**
- All payment data encrypted in transit (WireGuard) and at rest (AES-256)
- HMAC-SHA256 required on ALL payment API calls (both Engine->Payment and Origin->Engine)
- Transaction logs are append-only JSONL files, rotated daily

---

## 9. SECURITY ANALYZER PIPELINE (9d00)

> **Added**: 2026-06-20 System Audit
> **TOOL_ID**: 9d00
> **Node**: Engine (669427) only
> **AI Model**: Ollama Mistral 7B (local inference)

### Flow: Node Telemetry -> Engine -> Mistral AI -> Alerts/Suggestions

```
Node Telemetry (ALL 8 nodes)
 |
 v
[1] sec_tool_audit INSERT (Engine DB)
 | All tool execution logs via log_to_db.sh (TOOL_ID: f6a9)
 | AUDIT FIX: Previously targeted Shield (10.69.69.1 / s3rv1c3_admin)
 | Shield has NO MySQL — all logging was silently failing
 | Now targets: Engine (10.69.69.6 / engine / sec_tool_audit)
 v
[2] C++ Engine Daemon (Engine, 669427)
 | Aggregates telemetry from sec_tool_audit
 | Runs pattern analysis on tool execution data
 | Queries: unusual exit codes, execution spikes, timing anomalies
 v
[3] Ollama Mistral 7B (Engine, local inference)
 | Analyzes aggregated data for:
 | - Anomalous tool execution patterns
 | - Failed authentication spikes (ss-rej action codes)
 | - Configuration drift detection (ph-drft)
 | - Unusual network activity (fw-* codes)
 | - Credential rotation failures (cr-* codes)
 v
[4] engine_alerts INSERT (Engine DB)
 | Creates alert with:
 |   severity (info/low/medium/high/critical)
 |   evidence_json (supporting data)
 |   confidence score (0.000 to 1.000)
 v
[5] engine_suggestions INSERT (Engine DB)
 | Proposes remediation action:
 |   category (firewall, rotation, lockdown, config, deploy)
 |   action_type (block_ip, rotate_key, patch_config, etc.)
 |   risk_level assessment
 |   Encrypted command payload for execution
 |   Auto-expires after configurable TTL
 v
[6] Dashboard Review (Origin, via lantern tunnel)
 | Admin reviews pending suggestions
 | Approve/reject workflow with audit trail
 | Approved actions executed by Engine on target node
```

### Key Tables (Engine DB — 10.69.69.6)

| Table                | Purpose                                                    |
| -------------------- | ---------------------------------------------------------- |
| `sec_tool_audit`     | Obfuscated tool execution log (replaces broken `tool_executions`) |
| `engine_alerts`      | Mistral AI-generated security alerts                       |
| `engine_suggestions` | Proposed remediation with approval workflow                 |
| `wg_config`          | WireGuard mesh configuration (13 existing rows)            |
| `wg_port_allowlist`  | Port rotation pool for WireGuard tunnels                   |
| `wg_subnet_pool`     | Subnet allocation for per-store traffic isolation          |

### AUDIT FIX: log_to_db.sh Redirect (2026-06-20)

| Parameter | Before (BROKEN)             | After (FIXED)               |
| --------- | --------------------------- | --------------------------- |
| DB Host   | 10.69.69.1 (Shield)        | 10.69.69.6 (Engine)         |
| DB Name   | s3rv1c3_admin               | engine                      |
| DB Table  | tool_executions             | sec_tool_audit              |
| DB User   | tooling                     | engine_user                 |
| Columns   | Human-readable              | Obfuscated (per AUDIT_CODES.md) |

> **CAUTION**: Shield (414342) has NO MariaDB/MySQL installed. The previous configuration caused ALL tool execution logging to silently fail across all 8 nodes. Every `tool_wrap()` and `log_to_db()` call in the tooling arsenal was hitting a dead endpoint.

### sec_tool_audit Column Mapping

| Column | Type        | Description                                    |
| ------ | ----------- | ---------------------------------------------- |
| id     | BIGINT      | Auto-increment primary key                     |
| ts     | DATETIME(3) | Timestamp with milliseconds                    |
| nid    | VARCHAR(10) | NODE_ID (6-digit)                              |
| tid    | VARCHAR(8)  | TOOL_ID from TOOLING_MAP.md                    |
| act    | VARCHAR(32) | Action code from AUDIT_CODES.md                |
| src    | VARCHAR(64) | Source NODE_ID or SHA-256 of external IP        |
| res    | TINYINT(1)  | Result: 1=success, 0=fail                      |
| phash  | VARCHAR(64) | SHA-256 of request payload                     |
| sid    | VARCHAR(32) | Session ID from b2e0                           |
| dur    | INT         | Duration in milliseconds                       |
| xc     | SMALLINT    | Exit code                                      |
| out    | TEXT        | Truncated stdout (500 chars max)               |
| err    | TEXT        | Truncated stderr (500 chars max)               |

### Action Code Reference

See [AUDIT_CODES.md](file:///y:/documents/AUDIT_CODES.md) for the complete action code registry.
Key code families: `hb-*` (heartbeat), `ss-*` (session), `wg-*` (WireGuard),
`fw-*` (firewall), `cr-*` (credential), `ai-*` (Mistral), `ph-*` (Phoenix).

