# Helix Engage — Developer Operations Runbook

## Architecture

See [architecture.md](./architecture.md) for the full multi-tenant topology diagram.

```
Browser (India)
    ↓ HTTPS
Caddy (reverse proxy, TLS, host-routed)
    ├── ramaiah.engage.healix360.net  → sidecar-ramaiah:4100
    ├── global.engage.healix360.net   → sidecar-global:4100
    ├── telephony.engage.healix360.net → telephony:4200
    ├── *.app.healix360.net           → server:4000 (platform)
    ├── monitoring.healix360.net      → grafana:3000
    ├── operations.healix360.net      → woodpecker-server:8000
    ├── git.healix360.net             → gitea:3000
    └── engage.healix360.net          → 404 (no catchall)

Docker Compose stack (EC2 — 13.234.31.194):
    ├── caddy              — Reverse proxy + TLS (Let's Encrypt)
    ├── server             — FortyTwo platform (NestJS, port 4000)
    ├── worker             — BullMQ background jobs
    ├── sidecar-ramaiah    — Ramaiah sidecar (NestJS, port 4100)
    ├── sidecar-global     — Global sidecar (NestJS, port 4100)
    ├── telephony          — Event dispatcher (NestJS, port 4200)
    ├── redis-ramaiah      — Ramaiah sidecar Redis
    ├── redis-global       — Global sidecar Redis
    ├── redis-telephony    — Telephony dispatcher Redis
    ├── redis              — Platform Redis
    ├── db                 — PostgreSQL 16 (workspace-per-schema)
    ├── clickhouse         — Analytics
    ├── minio              — S3-compatible object storage
    ├── redpanda           — Event bus (Kafka-compatible)
    ├── loki               — Log aggregation (receives from Docker logging driver)
    └── grafana            — Monitoring dashboards (Loki + ClickHouse data sources)
```

---

## EC2 Access

```bash
# SSH into EC2 (key passphrase handled by sshpass)
SSHPASS='SasiSuman@2007' sshpass -P "Enter passphrase" -e \
  ssh -i ~/Downloads/fortytwoai_hostinger -o StrictHostKeyChecking=no ubuntu@13.234.31.194
```

| Detail | Value |
|---|---|
| Host | `13.234.31.194` |
| User | `ubuntu` |
| SSH key | `~/Downloads/fortytwoai_hostinger` (passphrase-protected) |
| Passphrase | `SasiSuman@2007` |
| Docker compose dir | `/opt/fortytwo` |
| Frontend static files | `/opt/fortytwo/helix-engage-frontend` |
| Caddyfile | `/opt/fortytwo/Caddyfile` |

### SSH Helper

The key is passphrase-protected. Use `sshpass` to supply the passphrase non-interactively.
No need to decrypt or copy the key — use the original file directly.

```bash
# SSH shorthand
EC2_SSH="SSHPASS='SasiSuman@2007' sshpass -P 'Enter passphrase' -e ssh -i ~/Downloads/fortytwoai_hostinger -o StrictHostKeyChecking=no ubuntu@13.234.31.194"

# Verify
eval $EC2_SSH hostname
```

> **Note:** VPN may block port 22 to AWS. Disconnect VPN before SSH.

---

## URLs

| Service | URL |
|---|---|
| Ramaiah Engage (Frontend + API) | `https://ramaiah.engage.healix360.net` |
| Global Engage (Frontend + API) | `https://global.engage.healix360.net` |
| Ramaiah Platform | `https://ramaiah.app.healix360.net` |
| Global Platform | `https://global.app.healix360.net` |
| Telephony Dispatcher | `https://telephony.engage.healix360.net` |
| Monitoring (Grafana) | `https://monitoring.healix360.net` |
| CI/CD (Woodpecker) | `https://operations.healix360.net` |
| Git (Gitea) | `https://git.healix360.net` |

---

## Login Credentials

### Ramaiah Workspace

| Role | Email | Password |
|---|---|---|
| Marketing Executive | `marketing@ramaiahcare.com` | `AdRamaiah@2026` |
| Marketing Executive | `supervisor@ramaiahcare.com` | `MrRamaiah@2026` |
| CC Agent | `ccagent@ramaiahcare.com` | `CcRamaiah@2026` |
| Platform Admin | `dev@fortytwo.dev` | `tim@apple.dev` |

### Ozonetel

| Field | Value |
|---|---|
| API Key | `KK8110e6c3de02527f7243ffaa924fa93e` |
| Username | `global_healthx` |
| Ramaiah Campaign | `Inbound_918041763400` |
| Ramaiah Agent | `ramaiahadmin` / ext `524435` |

---

## Local Development

### Frontend (Vite dev server)

```bash
cd helix-engage
npm run dev          # http://localhost:5173
npx tsc --noEmit     # Type check
npm run build        # Production build
```

The `.env.local` controls which sidecar the frontend talks to:
```bash
# Remote (default — uses EC2 backend)
VITE_API_URL=https://ramaiah.engage.healix360.net

# Local sidecar
# VITE_API_URL=http://localhost:4100
```

### Sidecar (NestJS dev server)

```bash
cd helix-engage-server
npm run start:dev    # http://localhost:4100 (watch mode)
npm run build        # Build only
```

Sidecar `.env` must have:
```bash
PLATFORM_GRAPHQL_URL=https://ramaiah.app.healix360.net/graphql
PLATFORM_API_KEY=<Ramaiah workspace API key>
PLATFORM_WORKSPACE_SUBDOMAIN=ramaiah
REDIS_URL=redis://localhost:6379
```

### Pre-deploy checklist

1. `npx tsc --noEmit` — passes (frontend)
2. `npm run build` — succeeds (sidecar)
3. Test the changed feature locally
4. Check `package.json` for new dependencies → decides quick vs full deploy

---

## Deployment

### Frontend

```bash
# Helper — reuse in all commands below
EC2="SSHPASS='SasiSuman@2007' sshpass -P 'Enter passphrase' -e ssh -i ~/Downloads/fortytwoai_hostinger -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
EC2_RSYNC="SSHPASS='SasiSuman@2007' sshpass -P 'Enter passphrase' -e ssh -i ~/Downloads/fortytwoai_hostinger -o StrictHostKeyChecking=no"

cd helix-engage && npm run build

rsync -avz -e "$EC2_RSYNC" \
  dist/ ubuntu@13.234.31.194:/opt/fortytwo/helix-engage-frontend/

eval $EC2 "cd /opt/fortytwo && sudo docker compose restart caddy"
```

### Sidecar

```bash
cd helix-engage-server

# 1. Login to ECR
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com

# 2. Build and push Docker image
docker buildx build --platform linux/amd64 \
  -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \
  --push .

# 3. Pull and restart on EC2
eval $EC2 "cd /opt/fortytwo && sudo docker compose pull sidecar-ramaiah sidecar-global && sudo docker compose up -d sidecar-ramaiah sidecar-global"
```

### How to decide

```
Did package.json change?
  ├── YES → ECR build + push + pull (above)
  └── NO  → Same steps (ECR is the only deploy path for EC2)
```

---

## Post-Deploy: E2E Smoke Tests

```bash
cd helix-engage
npx playwright test
```

27 tests covering login, all CC Agent pages, all Supervisor pages, and sign-out.
The last test completes sign-out so the agent session is released for the next run.

---

## Checking Logs

```bash
# Ramaiah sidecar
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 30 2>&1"

# Follow live
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-sidecar-ramaiah-1 -f --tail 10 2>&1"

# Filter errors
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 100 2>&1" | grep -i "error\|fail"

# Telephony dispatcher
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"

# Caddy
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-caddy-1 --tail 20 2>&1"

# Platform server
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-server-1 --tail 30 2>&1"

# All container status
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker ps --format 'table {{.Names}}\t{{.Status}}'"
```

### Healthy startup

Look for these in sidecar logs:
```
[NestApplication] Nest application successfully started
Helix Engage Server running on port 4100
[SessionService] Redis connected
```

### Common failure patterns

| Log pattern | Meaning | Fix |
|---|---|---|
| `Cannot find module 'xxx'` | Missing npm dependency | Rebuild ECR image |
| `UndefinedModuleException` | Circular dependency | Fix code, redeploy |
| `ECONNREFUSED redis:6379` | Redis not ready | `docker compose up -d redis-ramaiah` |
| `Forbidden resource` | Platform permission issue | Check user roles |
| `429 Too Many Requests` | Ozonetel rate limit | Wait, reduce polling |

---

## Redis Operations

```bash
SSH="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
REDIS="docker exec ramaiah-prod-redis-ramaiah-1 redis-cli"

# Clear agent session lock (fixes "already logged in from another device")
$SSH "$REDIS DEL agent:session:ramaiahadmin"

# List all keys
$SSH "$REDIS KEYS '*'"

# Clear caller cache (stale patient names)
$SSH "$REDIS --scan --pattern 'caller:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Clear masterdata cache (departments/doctors/clinics/slots)
$SSH "$REDIS --scan --pattern 'masterdata:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Clear recording analysis cache
$SSH "$REDIS --scan --pattern 'call:analysis:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Clear agent name cache
$SSH "$REDIS --scan --pattern 'agent:name:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"

# Nuclear: flush all sidecar Redis
$SSH "$REDIS FLUSHDB"
```

---

## Database Access

```bash
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker exec -it ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap"
```

### Useful queries

```sql
-- List workspace schemas
SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';

-- List custom entities
SELECT "nameSingular", "isCustom" FROM core."objectMetadata" ORDER BY "nameSingular";

-- List users
SELECT u.email, u."firstName", u."lastName", uw.id as workspace_id
FROM core."user" u
JOIN core."userWorkspace" uw ON uw."userId" = u.id;

-- List roles
SELECT r.label, rt."userWorkspaceId"
FROM core."roleTarget" rt
JOIN core."role" r ON r.id = rt."roleId";
```

---

## Troubleshooting

### "Already logged in from another device"

Single-session enforcement per Ozonetel agent. Clear the lock:
```bash
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker exec ramaiah-prod-redis-ramaiah-1 redis-cli DEL agent:session:ramaiahadmin"
```

### Agent stuck in ACW / Wrapping Up

```bash
curl -X POST https://ramaiah.engage.healix360.net/api/maint/force-ready \
  -H "Content-Type: application/json" \
  -d '{"agentId": "ramaiahadmin"}'
```

### Telephony events not routing

```bash
# Check dispatcher logs
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"

# Check service discovery registry
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker exec ramaiah-prod-redis-telephony-1 redis-cli KEYS '*'"
```

### Theme/branding reset after Redis flush

```bash
curl -X PUT https://ramaiah.engage.healix360.net/api/config/theme \
  -H "Content-Type: application/json" \
  -d '{"defaults": {"brandName": "Helix Engage", "hospitalName": "Ramaiah Hospitals"}}'
```

---

## Rollback

### Frontend

Checkout previous commit → `npm run build` → rsync to EC2.

### Sidecar

Checkout previous commit → ECR build + push → pull on EC2.

For immediate rollback, re-tag a known-good ECR image as `:alpha` and pull.

---

## Git Repositories

| Repo | Azure DevOps | Branch |
|---|---|---|
| Frontend | `helix-engage` in Patient Engagement Platform | `feature/omnichannel-widget` |
| Sidecar | `helix-engage-server` in Patient Engagement Platform | `master` |
| SDK App | `FortyTwoApps/helix-engage/` (monorepo) | `dev` |
| Telephony | `helix-engage-telephony` in Patient Engagement Platform | `master` |

---

## ECR Details

| Detail | Value |
|---|---|
| Registry | `043728036361.dkr.ecr.ap-south-1.amazonaws.com` |
| Sidecar repo | `fortytwo-eap/helix-engage-sidecar` |
| Tag | `alpha` |
| Region | `ap-south-1` (Mumbai) |
