mirror of
https://dev.azure.com/globalhealthx/EMR/_git/helix-engage
synced 2026-04-11 18:28:15 +00:00
feat: add E2E smoke tests, architecture docs, and operations runbook
- 27 Playwright E2E tests covering login (3 roles), CC Agent pages (call desk, call history, patients, appointments, my performance, sidebar, sign-out), and Supervisor pages (all 11 pages + sidebar) - Tests run against live EC2 at ramaiah.engage.healix360.net - Last test completes sign-out to release agent session for next run - Architecture doc with updated Mermaid diagram including telephony dispatcher, service discovery, and multi-tenant topology - Operations runbook with SSH access (VPS + EC2), accounts, container reference, deploy steps, Redis ops, and troubleshooting guide Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
322
docs/runbook.md
Normal file
322
docs/runbook.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Helix Engage — Operations Runbook
|
||||
|
||||
Day-to-day operations guide for deploying, debugging, and maintaining Helix Engage.
|
||||
|
||||
---
|
||||
|
||||
## Environments
|
||||
|
||||
| | **VPS (Global)** | **EC2 (Ramaiah)** |
|
||||
|---|---|---|
|
||||
| **Host** | `148.230.67.184` | `13.234.31.194` |
|
||||
| **Domain** | `engage-api.srv1477139.hstgr.cloud` | `*.engage.healix360.net` |
|
||||
| **Docker path** | `/opt/fortytwo` | `/opt/fortytwo` |
|
||||
| **Topology** | Single-tenant | Multi-tenant (2 sidecars + telephony) |
|
||||
|
||||
---
|
||||
|
||||
## SSH Access
|
||||
|
||||
### VPS (Global)
|
||||
|
||||
```bash
|
||||
sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184
|
||||
```
|
||||
|
||||
### EC2 (Ramaiah)
|
||||
|
||||
The SSH key is at `~/Downloads/fortytwoai_hostinger` (passphrase-protected).
|
||||
A decrypted copy must exist at `/tmp/ramaiah-ec2-key`.
|
||||
|
||||
**First-time setup (one of these):**
|
||||
|
||||
```bash
|
||||
# Option A: Decrypt key file (non-interactive, passphrase: SasiSuman@2007)
|
||||
openssl pkey -in ~/Downloads/fortytwoai_hostinger -out /tmp/ramaiah-ec2-key
|
||||
chmod 600 /tmp/ramaiah-ec2-key
|
||||
|
||||
# Option B: Add to ssh-agent (interactive — prompts for passphrase)
|
||||
ssh-add ~/Downloads/fortytwoai_hostinger
|
||||
```
|
||||
|
||||
**After setup:**
|
||||
|
||||
```bash
|
||||
ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194
|
||||
```
|
||||
|
||||
**Quick alias for repeated use:**
|
||||
|
||||
```bash
|
||||
alias ec2="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
|
||||
alias vps="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Accounts
|
||||
|
||||
### Ramaiah (EC2)
|
||||
|
||||
| Role | Email | Password | Notes |
|
||||
|------|-------|----------|-------|
|
||||
| Marketing Executive | `marketing@ramaiahcare.com` | `AdRamaiah@2026` | Landing: Lead Workspace |
|
||||
| Marketing Executive | `supervisor@ramaiahcare.com` | `MrRamaiah@2026` | Landing: Lead Workspace |
|
||||
| CC Agent | `ccagent@ramaiahcare.com` | `CcRamaiah@2026` | Ozonetel agent: `ramaiahadmin` |
|
||||
| Platform Admin | `dev@fortytwo.dev` | `tim@apple.dev` | Break-glass admin. **NEVER delete.** |
|
||||
|
||||
### Ozonetel
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| API Key | `KK8110e6c3de02527f7243ffaa924fa93e` |
|
||||
| Username | `global_healthx` |
|
||||
| Ramaiah Campaign | `Inbound_918041763400` |
|
||||
| Global Campaign | `Inbound_918041763265` |
|
||||
| Ramaiah Agent | `ramaiahadmin` / ext `524435` |
|
||||
|
||||
---
|
||||
|
||||
## EC2 Containers
|
||||
|
||||
```bash
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker ps --format 'table {{.Names}}\t{{.Status}}'"
|
||||
```
|
||||
|
||||
| Container | Purpose | Port |
|
||||
|-----------|---------|------|
|
||||
| `ramaiah-prod-caddy-1` | Reverse proxy + TLS | 80, 443 |
|
||||
| `ramaiah-prod-server-1` | Platform API | 4000 |
|
||||
| `ramaiah-prod-worker-1` | BullMQ worker | — |
|
||||
| `ramaiah-prod-sidecar-ramaiah-1` | Ramaiah sidecar | 4100 |
|
||||
| `ramaiah-prod-sidecar-global-1` | Global sidecar | 4100 |
|
||||
| `ramaiah-prod-telephony-1` | Event dispatcher | 4200 |
|
||||
| `ramaiah-prod-redis-ramaiah-1` | Ramaiah Redis | 6379 |
|
||||
| `ramaiah-prod-redis-global-1` | Global Redis | 6379 |
|
||||
| `ramaiah-prod-redis-telephony-1` | Telephony Redis | 6379 |
|
||||
| `ramaiah-prod-redis-1` | Platform Redis | 6379 |
|
||||
| `ramaiah-prod-db-1` | PostgreSQL | 5432 |
|
||||
|
||||
---
|
||||
|
||||
## Checking Logs
|
||||
|
||||
```bash
|
||||
# EC2 — Ramaiah sidecar
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 30 2>&1"
|
||||
|
||||
# EC2 — Telephony dispatcher
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"
|
||||
|
||||
# EC2 — Platform server
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-server-1 --tail 30 2>&1"
|
||||
|
||||
# EC2 — Caddy
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-caddy-1 --tail 20 2>&1"
|
||||
|
||||
# EC2 — Filter errors only
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 100 2>&1" | grep -i "error\|fail\|crash"
|
||||
|
||||
# VPS — Sidecar
|
||||
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 \
|
||||
"docker logs fortytwo-staging-sidecar-1 --tail 30 2>&1"
|
||||
```
|
||||
|
||||
**Healthy sidecar output:**
|
||||
- `Nest application successfully started`
|
||||
- `Helix Engage Server running on port 4100`
|
||||
- `SessionService Redis connected`
|
||||
|
||||
---
|
||||
|
||||
## Deploying
|
||||
|
||||
### Pre-flight checks
|
||||
|
||||
```bash
|
||||
# Frontend type check
|
||||
cd helix-engage && npx tsc --noEmit
|
||||
|
||||
# Sidecar build check
|
||||
cd helix-engage-server && npm run build
|
||||
```
|
||||
|
||||
### Frontend (EC2)
|
||||
|
||||
```bash
|
||||
cd helix-engage && npm run build
|
||||
|
||||
rsync -avz -e "ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no" \
|
||||
dist/ ubuntu@13.234.31.194:/opt/fortytwo/helix-engage-frontend/
|
||||
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"cd /opt/fortytwo && sudo docker compose restart caddy"
|
||||
```
|
||||
|
||||
### Sidecar (EC2 — via ECR)
|
||||
|
||||
```bash
|
||||
cd helix-engage-server
|
||||
|
||||
# ECR login + build + push
|
||||
aws ecr get-login-password --region ap-south-1 | \
|
||||
docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com
|
||||
|
||||
docker buildx build --platform linux/amd64 \
|
||||
-t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \
|
||||
--push .
|
||||
|
||||
# Pull + restart on EC2
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"cd /opt/fortytwo && sudo docker compose pull sidecar-ramaiah sidecar-global && sudo docker compose up -d sidecar-ramaiah sidecar-global"
|
||||
```
|
||||
|
||||
### VPS (Global)
|
||||
|
||||
```bash
|
||||
cd /Users/satyasumansaridae/Downloads/fortytwo-eap
|
||||
|
||||
bash deploy.sh frontend # Frontend only
|
||||
bash deploy.sh sidecar # Sidecar only
|
||||
bash deploy.sh all # Both
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Post-Deploy: E2E Smoke Tests
|
||||
|
||||
```bash
|
||||
cd helix-engage
|
||||
|
||||
# Run against EC2 (default)
|
||||
npx playwright test
|
||||
|
||||
# Run against VPS
|
||||
E2E_BASE_URL=https://engage-api.srv1477139.hstgr.cloud npx playwright test
|
||||
```
|
||||
|
||||
27 tests covering login (invalid creds, CC Agent, Supervisor), every page
|
||||
for both roles, and sign-out. The last test completes sign-out so the agent
|
||||
session is released for the next run.
|
||||
|
||||
---
|
||||
|
||||
## Redis Operations
|
||||
|
||||
### EC2 (Ramaiah sidecar Redis)
|
||||
|
||||
```bash
|
||||
SSH="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
|
||||
REDIS="docker exec ramaiah-prod-redis-ramaiah-1 redis-cli"
|
||||
|
||||
# Clear agent session lock (fixes "already logged in from another device")
|
||||
$SSH "$REDIS DEL agent:session:ramaiahadmin"
|
||||
|
||||
# List all keys
|
||||
$SSH "$REDIS KEYS '*'"
|
||||
|
||||
# Clear caller cache (stale patient names)
|
||||
$SSH "$REDIS --scan --pattern 'caller:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"
|
||||
|
||||
# Clear masterdata cache
|
||||
$SSH "$REDIS --scan --pattern 'masterdata:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"
|
||||
|
||||
# Clear agent name cache
|
||||
$SSH "$REDIS --scan --pattern 'agent:name:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"
|
||||
|
||||
# Nuclear: flush all
|
||||
$SSH "$REDIS FLUSHDB"
|
||||
```
|
||||
|
||||
### VPS (Global sidecar Redis)
|
||||
|
||||
```bash
|
||||
SSH="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184"
|
||||
REDIS="docker exec fortytwo-staging-redis-1 redis-cli"
|
||||
|
||||
$SSH "$REDIS DEL agent:session:<agentId>"
|
||||
$SSH "$REDIS FLUSHDB"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Already logged in from another device"
|
||||
|
||||
The sidecar enforces single-session per Ozonetel agent. Clear the lock:
|
||||
|
||||
```bash
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker exec ramaiah-prod-redis-ramaiah-1 redis-cli DEL agent:session:ramaiahadmin"
|
||||
```
|
||||
|
||||
### Agent stuck in ACW / Wrapping Up
|
||||
|
||||
Three protection layers exist (beforeunload → sendBeacon → server 30s timer).
|
||||
If all fail, force-ready:
|
||||
|
||||
```bash
|
||||
curl -X POST https://ramaiah.engage.healix360.net/api/maint/force-ready \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"agentId": "ramaiahadmin"}'
|
||||
```
|
||||
|
||||
### Container restart loop
|
||||
|
||||
```bash
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 50 2>&1" | grep -i "error\|fail\|crash"
|
||||
```
|
||||
|
||||
Common causes:
|
||||
- `Cannot find module` → need ECR rebuild (new dependencies)
|
||||
- `UndefinedModuleException` → circular dependency in code
|
||||
- `ECONNREFUSED` to Redis → Redis container down, `docker compose up -d redis-ramaiah`
|
||||
|
||||
### Theme/branding reset after sidecar restart
|
||||
|
||||
Config is in Redis. If flushed, re-apply:
|
||||
|
||||
```bash
|
||||
curl -X PUT https://ramaiah.engage.healix360.net/api/config/theme \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"defaults": {"brandName": "Helix Engage", "hospitalName": "Ramaiah Hospitals"}}'
|
||||
```
|
||||
|
||||
### Telephony events not routing
|
||||
|
||||
Check dispatcher logs and verify sidecar registration:
|
||||
|
||||
```bash
|
||||
# Dispatcher logs
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"
|
||||
|
||||
# Check service discovery registry
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||
"docker exec ramaiah-prod-redis-telephony-1 redis-cli KEYS '*'"
|
||||
```
|
||||
|
||||
### Full DB Reset (nuclear — destroys all data)
|
||||
|
||||
Only when field metadata is missing (0 rows in `core.fieldMetadata`):
|
||||
|
||||
```bash
|
||||
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 << 'EOF'
|
||||
cd /opt/fortytwo
|
||||
sudo docker compose stop server worker
|
||||
sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "DELETE FROM core.workspace;"
|
||||
# Find and drop orphaned workspace schemas
|
||||
sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';"
|
||||
# DROP SCHEMA ... CASCADE for each
|
||||
sudo docker exec ramaiah-prod-redis-1 redis-cli FLUSHALL
|
||||
sudo docker compose up -d server worker
|
||||
EOF
|
||||
```
|
||||
Reference in New Issue
Block a user