helix-engage/docs/runbook.md

Helix Engage — Operations Runbook

Day-to-day operations guide for deploying, debugging, and maintaining Helix Engage.

---

Environments

	VPS (Global)	EC2 (Ramaiah)
Host	148.230.67.184	13.234.31.194
Domain	engage-api.srv1477139.hstgr.cloud	*.engage.healix360.net
Docker path	/opt/fortytwo	/opt/fortytwo
Topology	Single-tenant	Multi-tenant (2 sidecars + telephony)

---

SSH Access

VPS (Global)

sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184

EC2 (Ramaiah)

The SSH key is at ~/Downloads/fortytwoai_hostinger (passphrase-protected). A decrypted copy must exist at /tmp/ramaiah-ec2-key.

First-time setup (one of these):

# Option A: Decrypt key file (non-interactive, passphrase: SasiSuman@2007)
openssl pkey -in ~/Downloads/fortytwoai_hostinger -out /tmp/ramaiah-ec2-key
chmod 600 /tmp/ramaiah-ec2-key

# Option B: Add to ssh-agent (interactive — prompts for passphrase)
ssh-add ~/Downloads/fortytwoai_hostinger

After setup:

ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194

Quick alias for repeated use:

alias ec2="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
alias vps="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184"

---

Accounts

Ramaiah (EC2)

Role	Email	Password	Notes
Marketing Executive	marketing@ramaiahcare.com	AdRamaiah@2026	Landing: Lead Workspace
Marketing Executive	supervisor@ramaiahcare.com	MrRamaiah@2026	Landing: Lead Workspace
CC Agent	ccagent@ramaiahcare.com	CcRamaiah@2026	Ozonetel agent: ramaiahadmin
Platform Admin	dev@fortytwo.dev	tim@apple.dev	Break-glass admin. NEVER delete.

Ozonetel

Field	Value
API Key	KK8110e6c3de02527f7243ffaa924fa93e
Username	global_healthx
Ramaiah Campaign	Inbound_918041763400
Global Campaign	Inbound_918041763265
Ramaiah Agent	ramaiahadmin / ext 524435

---

EC2 Containers

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker ps --format 'table {{.Names}}\t{{.Status}}'"

Container	Purpose	Port
ramaiah-prod-caddy-1	Reverse proxy + TLS	80, 443
ramaiah-prod-server-1	Platform API	4000
ramaiah-prod-worker-1	BullMQ worker	—
ramaiah-prod-sidecar-ramaiah-1	Ramaiah sidecar	4100
ramaiah-prod-sidecar-global-1	Global sidecar	4100
ramaiah-prod-telephony-1	Event dispatcher	4200
ramaiah-prod-redis-ramaiah-1	Ramaiah Redis	6379
ramaiah-prod-redis-global-1	Global Redis	6379
ramaiah-prod-redis-telephony-1	Telephony Redis	6379
ramaiah-prod-redis-1	Platform Redis	6379
ramaiah-prod-db-1	PostgreSQL	5432

---

Checking Logs

# EC2 — Ramaiah sidecar
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 30 2>&1"

# EC2 — Telephony dispatcher
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"

# EC2 — Platform server
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-server-1 --tail 30 2>&1"

# EC2 — Caddy
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-caddy-1 --tail 20 2>&1"

# EC2 — Filter errors only
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 100 2>&1" | grep -i "error\|fail\|crash"

# VPS — Sidecar
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 \
  "docker logs fortytwo-staging-sidecar-1 --tail 30 2>&1"

Healthy sidecar output:

• Nest application successfully started
• Helix Engage Server running on port 4100
• SessionService Redis connected

---

Deploying

Pre-flight checks

# Frontend type check
cd helix-engage && npx tsc --noEmit

# Sidecar build check
cd helix-engage-server && npm run build

Frontend (EC2)

cd helix-engage && npm run build

rsync -avz -e "ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no" \
  dist/ ubuntu@13.234.31.194:/opt/fortytwo/helix-engage-frontend/

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "cd /opt/fortytwo && sudo docker compose restart caddy"

Sidecar (EC2 — via ECR)

cd helix-engage-server

# ECR login + build + push
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com

docker buildx build --platform linux/amd64 \
  -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \
  --push .

# Pull + restart on EC2
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "cd /opt/fortytwo && sudo docker compose pull sidecar-ramaiah sidecar-global && sudo docker compose up -d sidecar-ramaiah sidecar-global"

VPS (Global)

cd /Users/satyasumansaridae/Downloads/fortytwo-eap

bash deploy.sh frontend   # Frontend only
bash deploy.sh sidecar    # Sidecar only
bash deploy.sh all        # Both

---

Post-Deploy: E2E Smoke Tests

cd helix-engage

# Run against EC2 (default)
npx playwright test

# Run against VPS
E2E_BASE_URL=https://engage-api.srv1477139.hstgr.cloud npx playwright test

27 tests covering login (invalid creds, CC Agent, Supervisor), every page for both roles, and sign-out. The last test completes sign-out so the agent session is released for the next run.

---

Redis Operations

EC2 (Ramaiah sidecar Redis)

SSH="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
REDIS="docker exec ramaiah-prod-redis-ramaiah-1 redis-cli"

# Clear agent session lock (fixes "already logged in from another device")
$SSH "$REDIS DEL agent:session:ramaiahadmin"

# List all keys
$SSH "$REDIS KEYS '*'"

# Clear caller cache (stale patient names)
$SSH "$REDIS --scan --pattern 'caller:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Clear masterdata cache
$SSH "$REDIS --scan --pattern 'masterdata:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Clear agent name cache
$SSH "$REDIS --scan --pattern 'agent:name:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Nuclear: flush all
$SSH "$REDIS FLUSHDB"

VPS (Global sidecar Redis)

SSH="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184"
REDIS="docker exec fortytwo-staging-redis-1 redis-cli"

$SSH "$REDIS DEL agent:session:<agentId>"
$SSH "$REDIS FLUSHDB"

---

Troubleshooting

"Already logged in from another device"

The sidecar enforces single-session per Ozonetel agent. Clear the lock:

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker exec ramaiah-prod-redis-ramaiah-1 redis-cli DEL agent:session:ramaiahadmin"

Agent stuck in ACW / Wrapping Up

Three protection layers exist (beforeunload → sendBeacon → server 30s timer). If all fail, force-ready:

curl -X POST https://ramaiah.engage.healix360.net/api/maint/force-ready \
  -H "Content-Type: application/json" \
  -d '{"agentId": "ramaiahadmin"}'

Container restart loop

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 50 2>&1" | grep -i "error\|fail\|crash"

Common causes:

• Cannot find module → need ECR rebuild (new dependencies)
• UndefinedModuleException → circular dependency in code
• ECONNREFUSED to Redis → Redis container down, docker compose up -d redis-ramaiah

Theme/branding reset after sidecar restart

Config is in Redis. If flushed, re-apply:

curl -X PUT https://ramaiah.engage.healix360.net/api/config/theme \
  -H "Content-Type: application/json" \
  -d '{"defaults": {"brandName": "Helix Engage", "hospitalName": "Ramaiah Hospitals"}}'

Telephony events not routing

Check dispatcher logs and verify sidecar registration:

# Dispatcher logs
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"

# Check service discovery registry
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker exec ramaiah-prod-redis-telephony-1 redis-cli KEYS '*'"

Full DB Reset (nuclear — destroys all data)

Only when field metadata is missing (0 rows in core.fieldMetadata):

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 << 'EOF'
cd /opt/fortytwo
sudo docker compose stop server worker
sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "DELETE FROM core.workspace;"
# Find and drop orphaned workspace schemas
sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';"
# DROP SCHEMA ... CASCADE for each
sudo docker exec ramaiah-prod-redis-1 redis-cli FLUSHALL
sudo docker compose up -d server worker
EOF