# Helix Engage — Operations Runbook Day-to-day operations guide for deploying, debugging, and maintaining Helix Engage. --- ## Environments | | **VPS (Global)** | **EC2 (Ramaiah)** | |---|---|---| | **Host** | `148.230.67.184` | `13.234.31.194` | | **Domain** | `engage-api.srv1477139.hstgr.cloud` | `*.engage.healix360.net` | | **Docker path** | `/opt/fortytwo` | `/opt/fortytwo` | | **Topology** | Single-tenant | Multi-tenant (2 sidecars + telephony) | --- ## SSH Access ### VPS (Global) ```bash sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184 ``` ### EC2 (Ramaiah) The SSH key is at `~/Downloads/fortytwoai_hostinger` (passphrase-protected). A decrypted copy must exist at `/tmp/ramaiah-ec2-key`. **First-time setup (one of these):** ```bash # Option A: Decrypt key file (non-interactive, passphrase: SasiSuman@2007) openssl pkey -in ~/Downloads/fortytwoai_hostinger -out /tmp/ramaiah-ec2-key chmod 600 /tmp/ramaiah-ec2-key # Option B: Add to ssh-agent (interactive — prompts for passphrase) ssh-add ~/Downloads/fortytwoai_hostinger ``` **After setup:** ```bash ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194 ``` **Quick alias for repeated use:** ```bash alias ec2="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194" alias vps="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184" ``` --- ## Accounts ### Ramaiah (EC2) | Role | Email | Password | Notes | |------|-------|----------|-------| | Marketing Executive | `marketing@ramaiahcare.com` | `AdRamaiah@2026` | Landing: Lead Workspace | | Marketing Executive | `supervisor@ramaiahcare.com` | `MrRamaiah@2026` | Landing: Lead Workspace | | CC Agent | `ccagent@ramaiahcare.com` | `CcRamaiah@2026` | Ozonetel agent: `ramaiahadmin` | | Platform Admin | `dev@fortytwo.dev` | `tim@apple.dev` | Break-glass admin. **NEVER delete.** | ### Ozonetel | Field | Value | |-------|-------| | API Key | `KK8110e6c3de02527f7243ffaa924fa93e` | | Username | `global_healthx` | | Ramaiah Campaign | `Inbound_918041763400` | | Global Campaign | `Inbound_918041763265` | | Ramaiah Agent | `ramaiahadmin` / ext `524435` | --- ## EC2 Containers ```bash ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker ps --format 'table {{.Names}}\t{{.Status}}'" ``` | Container | Purpose | Port | |-----------|---------|------| | `ramaiah-prod-caddy-1` | Reverse proxy + TLS | 80, 443 | | `ramaiah-prod-server-1` | Platform API | 4000 | | `ramaiah-prod-worker-1` | BullMQ worker | — | | `ramaiah-prod-sidecar-ramaiah-1` | Ramaiah sidecar | 4100 | | `ramaiah-prod-sidecar-global-1` | Global sidecar | 4100 | | `ramaiah-prod-telephony-1` | Event dispatcher | 4200 | | `ramaiah-prod-redis-ramaiah-1` | Ramaiah Redis | 6379 | | `ramaiah-prod-redis-global-1` | Global Redis | 6379 | | `ramaiah-prod-redis-telephony-1` | Telephony Redis | 6379 | | `ramaiah-prod-redis-1` | Platform Redis | 6379 | | `ramaiah-prod-db-1` | PostgreSQL | 5432 | --- ## Checking Logs ```bash # EC2 — Ramaiah sidecar ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 30 2>&1" # EC2 — Telephony dispatcher ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1" # EC2 — Platform server ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-server-1 --tail 30 2>&1" # EC2 — Caddy ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-caddy-1 --tail 20 2>&1" # EC2 — Filter errors only ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 100 2>&1" | grep -i "error\|fail\|crash" # VPS — Sidecar sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 \ "docker logs fortytwo-staging-sidecar-1 --tail 30 2>&1" ``` **Healthy sidecar output:** - `Nest application successfully started` - `Helix Engage Server running on port 4100` - `SessionService Redis connected` --- ## Deploying ### Pre-flight checks ```bash # Frontend type check cd helix-engage && npx tsc --noEmit # Sidecar build check cd helix-engage-server && npm run build ``` ### Frontend (EC2) ```bash cd helix-engage && npm run build rsync -avz -e "ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no" \ dist/ ubuntu@13.234.31.194:/opt/fortytwo/helix-engage-frontend/ ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "cd /opt/fortytwo && sudo docker compose restart caddy" ``` ### Sidecar (EC2 — via ECR) ```bash cd helix-engage-server # ECR login + build + push aws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com docker buildx build --platform linux/amd64 \ -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \ --push . # Pull + restart on EC2 ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "cd /opt/fortytwo && sudo docker compose pull sidecar-ramaiah sidecar-global && sudo docker compose up -d sidecar-ramaiah sidecar-global" ``` ### VPS (Global) ```bash cd /Users/satyasumansaridae/Downloads/fortytwo-eap bash deploy.sh frontend # Frontend only bash deploy.sh sidecar # Sidecar only bash deploy.sh all # Both ``` --- ## Post-Deploy: E2E Smoke Tests ```bash cd helix-engage # Run against EC2 (default) npx playwright test # Run against VPS E2E_BASE_URL=https://engage-api.srv1477139.hstgr.cloud npx playwright test ``` 27 tests covering login (invalid creds, CC Agent, Supervisor), every page for both roles, and sign-out. The last test completes sign-out so the agent session is released for the next run. --- ## Redis Operations ### EC2 (Ramaiah sidecar Redis) ```bash SSH="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194" REDIS="docker exec ramaiah-prod-redis-ramaiah-1 redis-cli" # Clear agent session lock (fixes "already logged in from another device") $SSH "$REDIS DEL agent:session:ramaiahadmin" # List all keys $SSH "$REDIS KEYS '*'" # Clear caller cache (stale patient names) $SSH "$REDIS --scan --pattern 'caller:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL" # Clear masterdata cache $SSH "$REDIS --scan --pattern 'masterdata:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL" # Clear agent name cache $SSH "$REDIS --scan --pattern 'agent:name:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL" # Nuclear: flush all $SSH "$REDIS FLUSHDB" ``` ### VPS (Global sidecar Redis) ```bash SSH="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184" REDIS="docker exec fortytwo-staging-redis-1 redis-cli" $SSH "$REDIS DEL agent:session:" $SSH "$REDIS FLUSHDB" ``` --- ## Troubleshooting ### "Already logged in from another device" The sidecar enforces single-session per Ozonetel agent. Clear the lock: ```bash ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker exec ramaiah-prod-redis-ramaiah-1 redis-cli DEL agent:session:ramaiahadmin" ``` ### Agent stuck in ACW / Wrapping Up Three protection layers exist (beforeunload → sendBeacon → server 30s timer). If all fail, force-ready: ```bash curl -X POST https://ramaiah.engage.healix360.net/api/maint/force-ready \ -H "Content-Type: application/json" \ -d '{"agentId": "ramaiahadmin"}' ``` ### Container restart loop ```bash ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 50 2>&1" | grep -i "error\|fail\|crash" ``` Common causes: - `Cannot find module` → need ECR rebuild (new dependencies) - `UndefinedModuleException` → circular dependency in code - `ECONNREFUSED` to Redis → Redis container down, `docker compose up -d redis-ramaiah` ### Theme/branding reset after sidecar restart Config is in Redis. If flushed, re-apply: ```bash curl -X PUT https://ramaiah.engage.healix360.net/api/config/theme \ -H "Content-Type: application/json" \ -d '{"defaults": {"brandName": "Helix Engage", "hospitalName": "Ramaiah Hospitals"}}' ``` ### Telephony events not routing Check dispatcher logs and verify sidecar registration: ```bash # Dispatcher logs ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1" # Check service discovery registry ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ "docker exec ramaiah-prod-redis-telephony-1 redis-cli KEYS '*'" ``` ### Full DB Reset (nuclear — destroys all data) Only when field metadata is missing (0 rows in `core.fieldMetadata`): ```bash ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 << 'EOF' cd /opt/fortytwo sudo docker compose stop server worker sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "DELETE FROM core.workspace;" # Find and drop orphaned workspace schemas sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';" # DROP SCHEMA ... CASCADE for each sudo docker exec ramaiah-prod-redis-1 redis-cli FLUSHALL sudo docker compose up -d server worker EOF ```