feat: add E2E smoke tests, architecture docs, and operations runbook

- 27 Playwright E2E tests covering login (3 roles), CC Agent pages (call desk, call history, patients, appointments, my performance, sidebar, sign-out), and Supervisor pages (all 11 pages + sidebar) - Tests run against live EC2 at ramaiah.engage.healix360.net - Last test completes sign-out to release agent session for next run - Architecture doc with updated Mermaid diagram including telephony dispatcher, service discovery, and multi-tenant topology - Operations runbook with SSH access (VPS + EC2), accounts, container reference, deploy steps, Redis ops, and troubleshooting guide Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 10:23:27 +00:00 · 2026-04-10 20:54:20 +05:30
parent a1598716ee
commit 1cdb7fe9e7
11 changed files with 1056 additions and 0 deletions
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -0,0 +1,248 @@
+# Helix Engage — Architecture
+
+Single EC2 instance (Mumbai `ap-south-1`) hosting two isolated Helix Engage
+workspaces on top of a shared FortyTwo platform. Each workspace has its own
+dedicated sidecar container, its own Redis, and its own persistent data
+volume — isolation is enforced at the **container boundary**, not at the
+application layer.
+
+**Host:** `13.234.31.194` (m6i.xlarge, Ubuntu 22.04)
+**DNS:** Cloudflare zone `healix360.net`
+**TLS:** Caddy + Let's Encrypt, HTTP-01 challenge per hostname
+
+---
+
+## Principles
+
+1. **Platform is multi-tenant by design.** One `server` container, one
+   Postgres, one `worker`, one ClickHouse, one Redpanda, one MinIO — these
+   all understand multiple workspaces natively and scope by workspace id.
+
+2. **Sidecar is single-tenant by design.** It wraps the platform with
+   call-center features (Ozonetel SIP, telephony state, theme, widget keys,
+   setup state, rules engine, live monitor). Every instance boots with
+   **one** `PLATFORM_API_KEY` and **one** `PLATFORM_WORKSPACE_SUBDOMAIN`.
+   We run one instance per workspace.
+
+3. **Caddy is strictly host-routed.** No default or catchall tenant.
+   A request lands on a host block or it 404s. The apex
+   `engage.healix360.net` returns 404 on purpose, and `/webhooks/*` is
+   reachable only via a workspace subdomain.
+
+4. **Redis is per-sidecar.** Sidecars share Redis key names without a
+   workspace dimension. Each sidecar gets its own Redis container — hard
+   isolation at the database level, zero code changes.
+
+5. **Telephony dispatcher routes events by agent.** Ozonetel event
+   subscriptions are account-level (not per-campaign). A single dispatcher
+   receives all agent/call events and routes them to the correct sidecar
+   using Redis-backed service discovery.
+
+---
+
+## URL Layout
+
+| Who | URL | Routes to |
+|---|---|---|
+| Ramaiah platform UI | `https://ramaiah.app.healix360.net` | `server:4000` |
+| Ramaiah Helix Engage | `https://ramaiah.engage.healix360.net` | `sidecar-ramaiah:4100` |
+| Global platform UI | `https://global.app.healix360.net` | `server:4000` |
+| Global Helix Engage | `https://global.engage.healix360.net` | `sidecar-global:4100` |
+| Telephony dispatcher | `https://telephony.engage.healix360.net` | `telephony:4200` |
+| Apex (dead-end) | `https://engage.healix360.net` | `404` |
+
+Ozonetel campaign webhook URLs — per tenant:
+
+| Campaign | DID | Webhook URL |
+|---|---|---|
+| `Inbound_918041763400` | Ramaiah | `https://ramaiah.engage.healix360.net/webhooks/ozonetel/missed-call` |
+| `Inbound_918041763265` | Global (on VPS until cutover) | `https://global.engage.healix360.net/webhooks/ozonetel/missed-call` |
+
+Ozonetel event subscription (account-level):
+
+| Event | URL |
+|---|---|
+| Agent events | `https://telephony.engage.healix360.net/api/supervisor/agent-event` |
+| Call events | `https://telephony.engage.healix360.net/api/supervisor/call-event` |
+
+---
+
+## Diagram
+
+```mermaid
+flowchart TB
+    subgraph Internet
+        OZO[Ozonetel<br/>CCaaS]
+        USR_R[Ramaiah users]
+        USR_G[Global users]
+    end
+
+    subgraph EC2 ["EC2 — 13.234.31.194 (ap-south-1)"]
+        CADDY{{"caddy<br/>host-routed<br/>Let's Encrypt"}}
+
+        subgraph TEL ["Telephony Dispatcher"]
+            DISP["telephony<br/>NestJS:4200<br/>routes by agentId"]
+            RD_T[("redis-telephony")]
+        end
+
+        subgraph PLATFORM ["Platform (shared, multi-tenant)"]
+            SRV["server<br/>NestJS:4000<br/>platform API + SPA"]
+            WKR["worker<br/>BullMQ"]
+            DB[("db<br/>postgres:16<br/>workspace-per-schema")]
+            CH[("clickhouse<br/>analytics")]
+            RP[("redpanda<br/>event bus")]
+            MINIO[("minio<br/>S3 storage")]
+        end
+
+        subgraph RAMAIAH ["Ramaiah tenant (isolated)"]
+            SC_R["sidecar-ramaiah<br/>NestJS:4100<br/>API_KEY=ramaiah admin"]
+            RD_R[("redis-ramaiah")]
+            VOL_R[/"data-ramaiah volume<br/>theme, telephony,<br/>widget, rules,<br/>setup-state"/]
+        end
+
+        subgraph GLOBAL ["Global tenant (isolated)"]
+            SC_G["sidecar-global<br/>NestJS:4100<br/>API_KEY=global admin"]
+            RD_G[("redis-global")]
+            VOL_G[/"data-global volume<br/>theme, telephony,<br/>widget, rules,<br/>setup-state"/]
+        end
+    end
+
+    USR_R -->|"ramaiah.app.healix360.net"| CADDY
+    USR_R -->|"ramaiah.engage.healix360.net"| CADDY
+    USR_G -->|"global.app.healix360.net"| CADDY
+    USR_G -->|"global.engage.healix360.net"| CADDY
+
+    OZO -->|"webhooks/ozonetel/missed-call<br/>(Ramaiah DID 918041763400)"| CADDY
+    OZO -.->|"webhooks/ozonetel/missed-call<br/>(Global DID 918041763265<br/>— still on VPS today)"| CADDY
+    OZO -->|"agent + call events<br/>(account-level subscription)"| CADDY
+    CADDY -->|"telephony.engage.*"| DISP
+
+    CADDY -->|"*.app.healix360.net<br/>/graphql, /auth/*, SPA"| SRV
+    CADDY -->|"ramaiah.engage.*<br/>/api/*, /webhooks/*, SPA"| SC_R
+    CADDY -->|"global.engage.*<br/>/api/*, /webhooks/*, SPA"| SC_G
+
+    DISP -->|"agentId lookup<br/>→ forward to sidecar"| SC_R
+    DISP -->|"agentId lookup<br/>→ forward to sidecar"| SC_G
+    DISP --- RD_T
+
+    SC_R -->|"self-register on boot<br/>heartbeat 30s"| DISP
+    SC_G -->|"self-register on boot<br/>heartbeat 30s"| DISP
+
+    SC_R -->|"GraphQL<br/>Origin: ramaiah.app.*"| SRV
+    SC_G -->|"GraphQL<br/>Origin: global.app.*"| SRV
+
+    SC_R --- RD_R
+    SC_G --- RD_G
+    SC_R --- VOL_R
+    SC_G --- VOL_G
+
+    SRV --- DB
+    SRV --- CH
+    SRV --- RP
+    SRV --- MINIO
+    WKR --- DB
+    WKR --- RP
+
+    classDef shared fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#000
+    classDef ramaiah fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000
+    classDef global fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#000
+    classDef external fill:#f5f5f5,stroke:#757575,stroke-width:1px,color:#000
+    classDef edge fill:#fce4ec,stroke:#c2185b,stroke-width:2px,color:#000
+    classDef telephony fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
+
+    class SRV,WKR,DB,CH,RP,MINIO shared
+    class SC_R,RD_R,VOL_R ramaiah
+    class SC_G,RD_G,VOL_G global
+    class OZO,USR_R,USR_G external
+    class CADDY edge
+    class DISP,RD_T telephony
+```
+
+---
+
+## Components
+
+| Component | Scope | Container count | Purpose |
+|---|---|---|---|
+| Caddy | Shared | 1 | Host-routed reverse proxy, TLS terminator |
+| Platform server | Shared | 1 | Natively multi-tenant by Origin/subdomain |
+| Platform worker | Shared | 1 | BullMQ jobs carry workspace context per-job |
+| Postgres | Shared | 1 | Multi-tenant via per-workspace schemas |
+| ClickHouse | Shared | 1 | Analytics — workspace dimension per event |
+| Redpanda | Shared | 1 | Event bus — workspace dimension per message |
+| MinIO | Shared | 1 | S3-compatible storage |
+| **Telephony dispatcher** | **Shared** | **1** | Routes Ozonetel events to correct sidecar by agentId |
+| **Redis (telephony)** | **Shared** | **1** | Service discovery registry for dispatcher |
+| **Sidecar** | **Per-tenant** | **2** | Call center layer (Ramaiah + Global) |
+| **Redis (sidecar)** | **Per-tenant** | **2** | Session, agent state, theme, rules cache |
+| **Data volume** | **Per-tenant** | **2** | File-based config in `/app/data/` |
+
+---
+
+## Telephony Event Flow
+
+Ozonetel event subscriptions are **account-level** — one subscription per Ozonetel account, not per campaign. All agent login/logout/state events and call events are POSTed to a single URL.
+
+```
+Ozonetel → POST telephony.engage.healix360.net/api/supervisor/agent-event
+  → Dispatcher receives { agentId: "ramaiahadmin", action: "incall", ... }
+  → Redis lookup: agentId "ramaiahadmin" → sidecar-ramaiah:4100
+  → Forward event to sidecar-ramaiah
+  → sidecar-ramaiah updates SupervisorService state, emits SSE
+```
+
+**Service discovery:** Each sidecar self-registers on boot via `POST /api/supervisor/register` with its agent list. Heartbeat every 30s, TTL 90s. If a sidecar goes down, its entries expire and the dispatcher stops routing to it.
+
+---
+
+## Request Flow
+
+### Agent opens Ramaiah Helix Engage
+```
+Browser → https://ramaiah.engage.healix360.net/
+  → Caddy (TLS, Host=ramaiah.engage.healix360.net)
+  → static SPA from /srv/engage
+
+Browser → POST /api/auth/login { email, password }
+  → Caddy → sidecar-ramaiah:4100
+  → sidecar calls platform with:
+      Origin: https://ramaiah.app.healix360.net
+      Authorization: Bearer <Ramaiah API key>
+  → platform resolves workspace by Origin → Ramaiah
+  → JWT returned
+```
+
+### Ozonetel POSTs a missed-call webhook
+```
+Ozonetel → POST https://ramaiah.engage.healix360.net/webhooks/ozonetel/missed-call
+  → Caddy (Host=ramaiah.engage.healix360.net)
+  → sidecar-ramaiah:4100 ONLY
+  → writes call row into Ramaiah workspace via platform
+```
+
+Cross-tenant leakage is physically impossible — Caddy's host-routing guarantees a Ramaiah webhook can never reach sidecar-global.
+
+---
+
+## Failure Modes
+
+| Failure | Blast radius |
+|---|---|
+| `sidecar-ramaiah` crashes | Ramaiah Engage 502s. Global + platform unaffected. |
+| `sidecar-global` crashes | Global Engage 502s. Ramaiah + platform unaffected. |
+| `redis-ramaiah` crashes | Ramaiah agents kicked from SIP. Global unaffected. |
+| `telephony` crashes | Agent/call state events stop routing. Sidecars still serve UI. |
+| `server` (platform) crashes | **Both workspaces** down for data. |
+| `db` crashes | Same as above. |
+| Caddy crashes | Nothing reachable until restart. |
+
+---
+
+## Adding a New Hospital
+
+1. Add sidecar container + Redis + data volume in `docker-compose.yml`
+2. Add Caddy host block for `newhospital.engage.healix360.net`
+3. Create workspace on platform, generate API key
+4. Set sidecar env: `PLATFORM_API_KEY`, `PLATFORM_WORKSPACE_SUBDOMAIN`
+5. Configure Ozonetel campaign webhook to `newhospital.engage.healix360.net/webhooks/ozonetel/missed-call`
+6. Sidecar self-registers with telephony dispatcher on boot — no dispatcher config needed
--- a/docs/runbook.md
+++ b/docs/runbook.md
@@ -0,0 +1,322 @@
+# Helix Engage — Operations Runbook
+
+Day-to-day operations guide for deploying, debugging, and maintaining Helix Engage.
+
+---
+
+## Environments
+
+| | **VPS (Global)** | **EC2 (Ramaiah)** |
+|---|---|---|
+| **Host** | `148.230.67.184` | `13.234.31.194` |
+| **Domain** | `engage-api.srv1477139.hstgr.cloud` | `*.engage.healix360.net` |
+| **Docker path** | `/opt/fortytwo` | `/opt/fortytwo` |
+| **Topology** | Single-tenant | Multi-tenant (2 sidecars + telephony) |
+
+---
+
+## SSH Access
+
+### VPS (Global)
+
+```bash
+sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184
+```
+
+### EC2 (Ramaiah)
+
+The SSH key is at `~/Downloads/fortytwoai_hostinger` (passphrase-protected).
+A decrypted copy must exist at `/tmp/ramaiah-ec2-key`.
+
+**First-time setup (one of these):**
+
+```bash
+# Option A: Decrypt key file (non-interactive, passphrase: SasiSuman@2007)
+openssl pkey -in ~/Downloads/fortytwoai_hostinger -out /tmp/ramaiah-ec2-key
+chmod 600 /tmp/ramaiah-ec2-key
+
+# Option B: Add to ssh-agent (interactive — prompts for passphrase)
+ssh-add ~/Downloads/fortytwoai_hostinger
+```
+
+**After setup:**
+
+```bash
+ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194
+```
+
+**Quick alias for repeated use:**
+
+```bash
+alias ec2="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
+alias vps="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184"
+```
+
+---
+
+## Accounts
+
+### Ramaiah (EC2)
+
+| Role | Email | Password | Notes |
+|------|-------|----------|-------|
+| Marketing Executive | `marketing@ramaiahcare.com` | `AdRamaiah@2026` | Landing: Lead Workspace |
+| Marketing Executive | `supervisor@ramaiahcare.com` | `MrRamaiah@2026` | Landing: Lead Workspace |
+| CC Agent | `ccagent@ramaiahcare.com` | `CcRamaiah@2026` | Ozonetel agent: `ramaiahadmin` |
+| Platform Admin | `dev@fortytwo.dev` | `tim@apple.dev` | Break-glass admin. **NEVER delete.** |
+
+### Ozonetel
+
+| Field | Value |
+|-------|-------|
+| API Key | `KK8110e6c3de02527f7243ffaa924fa93e` |
+| Username | `global_healthx` |
+| Ramaiah Campaign | `Inbound_918041763400` |
+| Global Campaign | `Inbound_918041763265` |
+| Ramaiah Agent | `ramaiahadmin` / ext `524435` |
+
+---
+
+## EC2 Containers
+
+```bash
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker ps --format 'table {{.Names}}\t{{.Status}}'"
+```
+
+| Container | Purpose | Port |
+|-----------|---------|------|
+| `ramaiah-prod-caddy-1` | Reverse proxy + TLS | 80, 443 |
+| `ramaiah-prod-server-1` | Platform API | 4000 |
+| `ramaiah-prod-worker-1` | BullMQ worker | — |
+| `ramaiah-prod-sidecar-ramaiah-1` | Ramaiah sidecar | 4100 |
+| `ramaiah-prod-sidecar-global-1` | Global sidecar | 4100 |
+| `ramaiah-prod-telephony-1` | Event dispatcher | 4200 |
+| `ramaiah-prod-redis-ramaiah-1` | Ramaiah Redis | 6379 |
+| `ramaiah-prod-redis-global-1` | Global Redis | 6379 |
+| `ramaiah-prod-redis-telephony-1` | Telephony Redis | 6379 |
+| `ramaiah-prod-redis-1` | Platform Redis | 6379 |
+| `ramaiah-prod-db-1` | PostgreSQL | 5432 |
+
+---
+
+## Checking Logs
+
+```bash
+# EC2 — Ramaiah sidecar
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 30 2>&1"
+
+# EC2 — Telephony dispatcher
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"
+
+# EC2 — Platform server
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-server-1 --tail 30 2>&1"
+
+# EC2 — Caddy
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-caddy-1 --tail 20 2>&1"
+
+# EC2 — Filter errors only
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 100 2>&1" | grep -i "error\|fail\|crash"
+
+# VPS — Sidecar
+sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 \
+  "docker logs fortytwo-staging-sidecar-1 --tail 30 2>&1"
+```
+
+**Healthy sidecar output:**
+- `Nest application successfully started`
+- `Helix Engage Server running on port 4100`
+- `SessionService Redis connected`
+
+---
+
+## Deploying
+
+### Pre-flight checks
+
+```bash
+# Frontend type check
+cd helix-engage && npx tsc --noEmit
+
+# Sidecar build check
+cd helix-engage-server && npm run build
+```
+
+### Frontend (EC2)
+
+```bash
+cd helix-engage && npm run build
+
+rsync -avz -e "ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no" \
+  dist/ ubuntu@13.234.31.194:/opt/fortytwo/helix-engage-frontend/
+
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "cd /opt/fortytwo && sudo docker compose restart caddy"
+```
+
+### Sidecar (EC2 — via ECR)
+
+```bash
+cd helix-engage-server
+
+# ECR login + build + push
+aws ecr get-login-password --region ap-south-1 | \
+  docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com
+
+docker buildx build --platform linux/amd64 \
+  -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \
+  --push .
+
+# Pull + restart on EC2
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "cd /opt/fortytwo && sudo docker compose pull sidecar-ramaiah sidecar-global && sudo docker compose up -d sidecar-ramaiah sidecar-global"
+```
+
+### VPS (Global)
+
+```bash
+cd /Users/satyasumansaridae/Downloads/fortytwo-eap
+
+bash deploy.sh frontend   # Frontend only
+bash deploy.sh sidecar    # Sidecar only
+bash deploy.sh all        # Both
+```
+
+---
+
+## Post-Deploy: E2E Smoke Tests
+
+```bash
+cd helix-engage
+
+# Run against EC2 (default)
+npx playwright test
+
+# Run against VPS
+E2E_BASE_URL=https://engage-api.srv1477139.hstgr.cloud npx playwright test
+```
+
+27 tests covering login (invalid creds, CC Agent, Supervisor), every page
+for both roles, and sign-out. The last test completes sign-out so the agent
+session is released for the next run.
+
+---
+
+## Redis Operations
+
+### EC2 (Ramaiah sidecar Redis)
+
+```bash
+SSH="ssh -i /tmp/ramaiah-ec2-key -o StrictHostKeyChecking=no ubuntu@13.234.31.194"
+REDIS="docker exec ramaiah-prod-redis-ramaiah-1 redis-cli"
+
+# Clear agent session lock (fixes "already logged in from another device")
+$SSH "$REDIS DEL agent:session:ramaiahadmin"
+
+# List all keys
+$SSH "$REDIS KEYS '*'"
+
+# Clear caller cache (stale patient names)
+$SSH "$REDIS --scan --pattern 'caller:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"
+
+# Clear masterdata cache
+$SSH "$REDIS --scan --pattern 'masterdata:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"
+
+# Clear agent name cache
+$SSH "$REDIS --scan --pattern 'agent:name:*' | xargs -r docker exec -i ramaiah-prod-redis-ramaiah-1 redis-cli DEL"
+
+# Nuclear: flush all
+$SSH "$REDIS FLUSHDB"
+```
+
+### VPS (Global sidecar Redis)
+
+```bash
+SSH="sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184"
+REDIS="docker exec fortytwo-staging-redis-1 redis-cli"
+
+$SSH "$REDIS DEL agent:session:<agentId>"
+$SSH "$REDIS FLUSHDB"
+```
+
+---
+
+## Troubleshooting
+
+### "Already logged in from another device"
+
+The sidecar enforces single-session per Ozonetel agent. Clear the lock:
+
+```bash
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker exec ramaiah-prod-redis-ramaiah-1 redis-cli DEL agent:session:ramaiahadmin"
+```
+
+### Agent stuck in ACW / Wrapping Up
+
+Three protection layers exist (beforeunload → sendBeacon → server 30s timer).
+If all fail, force-ready:
+
+```bash
+curl -X POST https://ramaiah.engage.healix360.net/api/maint/force-ready \
+  -H "Content-Type: application/json" \
+  -d '{"agentId": "ramaiahadmin"}'
+```
+
+### Container restart loop
+
+```bash
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-sidecar-ramaiah-1 --tail 50 2>&1" | grep -i "error\|fail\|crash"
+```
+
+Common causes:
+- `Cannot find module` → need ECR rebuild (new dependencies)
+- `UndefinedModuleException` → circular dependency in code
+- `ECONNREFUSED` to Redis → Redis container down, `docker compose up -d redis-ramaiah`
+
+### Theme/branding reset after sidecar restart
+
+Config is in Redis. If flushed, re-apply:
+
+```bash
+curl -X PUT https://ramaiah.engage.healix360.net/api/config/theme \
+  -H "Content-Type: application/json" \
+  -d '{"defaults": {"brandName": "Helix Engage", "hospitalName": "Ramaiah Hospitals"}}'
+```
+
+### Telephony events not routing
+
+Check dispatcher logs and verify sidecar registration:
+
+```bash
+# Dispatcher logs
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker logs ramaiah-prod-telephony-1 --tail 30 2>&1"
+
+# Check service discovery registry
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
+  "docker exec ramaiah-prod-redis-telephony-1 redis-cli KEYS '*'"
+```
+
+### Full DB Reset (nuclear — destroys all data)
+
+Only when field metadata is missing (0 rows in `core.fieldMetadata`):
+
+```bash
+ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 << 'EOF'
+cd /opt/fortytwo
+sudo docker compose stop server worker
+sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "DELETE FROM core.workspace;"
+# Find and drop orphaned workspace schemas
+sudo docker exec ramaiah-prod-db-1 psql -U fortytwo -d fortytwo_eap -c "SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';"
+# DROP SCHEMA ... CASCADE for each
+sudo docker exec ramaiah-prod-redis-1 redis-cli FLUSHALL
+sudo docker compose up -d server worker
+EOF
+```
--- a/e2e/.auth/.gitignore
+++ b/e2e/.auth/.gitignore
@@ -0,0 +1 @@
+*.json
--- a/e2e/agent-login.spec.ts
+++ b/e2e/agent-login.spec.ts
@@ -0,0 +1,56 @@
+/**
+ * Login feature tests — covers multiple roles.
+ *
+ * These run WITHOUT saved auth state (fresh browser).
+ */
+import { test, expect } from '@playwright/test';
+import { waitForApp } from './helpers';
+
+const SUPERVISOR = { email: 'supervisor@ramaiahcare.com', password: 'MrRamaiah@2026' };
+
+test.describe('Login', () => {
+
+    test('login page renders with branding', async ({ page }) => {
+        await page.goto('/login');
+
+        await expect(page.locator('img[alt]').first()).toBeVisible();
+        await expect(page.locator('h1').first()).toBeVisible();
+        await expect(page.locator('form')).toBeVisible();
+        await expect(page.locator('input[type="email"], input[placeholder*="@"]').first()).toBeVisible();
+        await expect(page.locator('input[type="password"]').first()).toBeVisible();
+        await expect(page.getByRole('button', { name: 'Sign in', exact: true })).toBeVisible();
+    });
+
+    test('invalid credentials show error', async ({ page }) => {
+        await page.goto('/login');
+
+        await page.locator('input[type="email"], input[placeholder*="@"]').first().fill('bad@bad.com');
+        await page.locator('input[type="password"]').first().fill('wrongpassword');
+        await page.getByRole('button', { name: 'Sign in', exact: true }).click();
+
+        await expect(
+            page.locator('text=/not found|invalid|incorrect|failed|error|unauthorized/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+        await expect(page).toHaveURL(/\/login/);
+    });
+
+    test('Supervisor login → lands on app', async ({ page }) => {
+        await page.goto('/login');
+
+        await page.locator('input[type="email"], input[placeholder*="@"]').first().fill(SUPERVISOR.email);
+        await page.locator('input[type="password"]').first().fill(SUPERVISOR.password);
+        await page.getByRole('button', { name: 'Sign in', exact: true }).click();
+
+        await expect(page).not.toHaveURL(/\/login/, { timeout: 20_000 });
+        await waitForApp(page);
+
+        // Sidebar should be visible
+        await expect(page.locator('aside').first()).toBeVisible();
+    });
+
+    test('unauthenticated user redirected to login', async ({ page }) => {
+        await page.context().clearCookies();
+        await page.goto('/patients');
+        await expect(page).toHaveURL(/\/login/, { timeout: 10_000 });
+    });
+});
--- a/e2e/agent-smoke.spec.ts
+++ b/e2e/agent-smoke.spec.ts
@@ -0,0 +1,155 @@
+/**
+ * CC Agent — happy-path smoke tests.
+ *
+ * Role: cc-agent (ccagent@ramaiahcare.com)
+ * Landing: / → Call Desk
+ * Pages: Call Desk, Call History, Patients, Appointments, My Performance
+ */
+import { test, expect } from '@playwright/test';
+import { waitForApp } from './helpers';
+
+test.describe('CC Agent Smoke', () => {
+
+    test('lands on Call Desk after login', async ({ page }) => {
+        await page.goto('/');
+        await waitForApp(page);
+
+        await expect(page.locator('aside').first()).toContainText(/Call Center/i);
+    });
+
+    test('Call Desk page loads', async ({ page }) => {
+        await page.goto('/');
+        await waitForApp(page);
+
+        // Call Desk is the landing — just verify we're not on an error page
+        await expect(page.locator('aside').first()).toContainText(/Call Desk/i);
+    });
+
+    test('Call History page loads', async ({ page }) => {
+        await page.goto('/call-history');
+        await waitForApp(page);
+
+        // Should show "Call History" title whether or not there are calls
+        await expect(page.locator('text="Call History"').first()).toBeVisible();
+
+        // Filter dropdown present
+        await expect(page.locator('text=/All Calls/i').first()).toBeVisible();
+    });
+
+    test('Patients page loads with search', async ({ page }) => {
+        await page.goto('/patients');
+        await waitForApp(page);
+
+        const search = page.getByPlaceholder(/search/i).or(page.getByLabel(/search/i));
+        await expect(search.first()).toBeVisible();
+    });
+
+    test('Patients search filters results', async ({ page }) => {
+        await page.goto('/patients');
+        await waitForApp(page);
+
+        const search = page.getByPlaceholder(/search/i).or(page.getByLabel(/search/i));
+        await search.first().fill('zzz-nonexistent-patient');
+        await waitForApp(page);
+
+        // Should show empty state
+        const noResults = page.locator('text=/no patient|not found|no results/i');
+        const isEmpty = await noResults.isVisible({ timeout: 5_000 }).catch(() => false);
+        expect(isEmpty).toBe(true);
+    });
+
+    test('Appointments page loads', async ({ page }) => {
+        await page.goto('/appointments');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Appointment|Schedule|No appointment/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('My Performance loads with date controls', async ({ page }) => {
+        // Intercept API to verify agentId is sent
+        const apiHit = page.waitForRequest(
+            (r) => r.url().includes('/api/ozonetel/performance') && r.url().includes('agentId='),
+            { timeout: 15_000 },
+        );
+
+        await page.goto('/my-performance');
+        const req = await apiHit;
+
+        const url = new URL(req.url());
+        expect(url.searchParams.get('agentId')?.length).toBeGreaterThan(0);
+
+        await waitForApp(page);
+
+        await expect(page.getByRole('button', { name: 'Today' })).toBeVisible();
+        await expect(page.getByRole('button', { name: 'Yesterday' })).toBeVisible();
+
+        // Either KPI data or empty state
+        await expect(
+            page.locator('text=/Total Calls|No performance data/i').first(),
+        ).toBeVisible();
+    });
+
+    test('sidebar has all CC Agent nav items', async ({ page }) => {
+        await page.goto('/');
+        await waitForApp(page);
+
+        const sidebar = page.locator('aside').first();
+        for (const item of ['Call Desk', 'Call History', 'Patients', 'Appointments', 'My Performance']) {
+            await expect(sidebar.locator(`text="${item}"`)).toBeVisible();
+        }
+    });
+
+    test('sign-out shows confirmation modal and cancel keeps session', async ({ page }) => {
+        await page.goto('/');
+        await waitForApp(page);
+
+        const sidebar = page.locator('aside').first();
+        const accountArea = sidebar.locator('[class*="account"], [class*="avatar"]').last();
+        if (await accountArea.isVisible()) {
+            await accountArea.click();
+        }
+
+        const signOutBtn = page.locator('button, [role="menuitem"]').filter({ hasText: /sign out/i }).first();
+        if (await signOutBtn.isVisible({ timeout: 5_000 }).catch(() => false)) {
+            await signOutBtn.click();
+
+            const modal = page.locator('[role="dialog"]');
+            await expect(modal).toBeVisible({ timeout: 5_000 });
+            await expect(modal).toContainText(/sign out/i);
+
+            // Cancel — should stay logged in
+            await modal.getByRole('button', { name: /cancel/i }).click();
+            await expect(modal).not.toBeVisible();
+            await expect(page).not.toHaveURL(/\/login/);
+        }
+    });
+
+    // MUST be the last test — completes sign-out so the agent session is
+    // released and the next test run won't hit "already logged in".
+    test('sign-out completes and redirects to login', async ({ page }) => {
+        await page.goto('/');
+        await waitForApp(page);
+
+        const sidebar = page.locator('aside').first();
+        const accountArea = sidebar.locator('[class*="account"], [class*="avatar"]').last();
+        if (await accountArea.isVisible()) {
+            await accountArea.click();
+        }
+
+        const signOutBtn = page.locator('button, [role="menuitem"]').filter({ hasText: /sign out/i }).first();
+        if (await signOutBtn.isVisible({ timeout: 5_000 }).catch(() => false)) {
+            await signOutBtn.click();
+
+            const modal = page.locator('[role="dialog"]');
+            await expect(modal).toBeVisible({ timeout: 5_000 });
+
+            // Confirm sign out
+            await modal.getByRole('button', { name: /sign out/i }).click();
+
+            // Should redirect to login
+            await expect(page).toHaveURL(/\/login/, { timeout: 15_000 });
+        }
+    });
+});
--- a/e2e/auth.setup.ts
+++ b/e2e/auth.setup.ts
@@ -0,0 +1,22 @@
+import { test as setup, expect } from '@playwright/test';
+import path from 'path';
+import { fileURLToPath } from 'url';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const authFile = path.join(__dirname, '.auth/agent.json');
+
+setup('login as CC Agent', async ({ page }) => {
+    await page.goto('/login');
+
+    await page.locator('input[type="email"], input[name="email"], input[placeholder*="@"]').first().fill('ccagent@ramaiahcare.com');
+    await page.locator('input[type="password"]').first().fill('CcRamaiah@2026');
+    await page.getByRole('button', { name: 'Sign in', exact: true }).click();
+
+    // Should land on Call Desk (/ for cc-agent role)
+    await expect(page).not.toHaveURL(/\/login/, { timeout: 20_000 });
+
+    // Sidebar should be visible
+    await expect(page.locator('aside').first()).toBeVisible({ timeout: 10_000 });
+
+    await page.context().storageState({ path: authFile });
+});
--- a/e2e/helpers.ts
+++ b/e2e/helpers.ts
@@ -0,0 +1,15 @@
+import { Page, expect } from '@playwright/test';
+
+export async function waitForApp(page: Page) {
+    await page.waitForLoadState('networkidle', { timeout: 15_000 }).catch(() => {});
+    await page.waitForTimeout(300);
+}
+
+export async function loginAs(page: Page, email: string, password: string) {
+    await page.goto('/login');
+    await page.locator('input[type="email"], input[name="email"], input[placeholder*="@"]').first().fill(email);
+    await page.locator('input[type="password"]').first().fill(password);
+    await page.getByRole('button', { name: 'Sign in', exact: true }).click();
+    await expect(page).not.toHaveURL(/\/login/, { timeout: 20_000 });
+    await waitForApp(page);
+}
--- a/e2e/supervisor-smoke.spec.ts
+++ b/e2e/supervisor-smoke.spec.ts
@@ -0,0 +1,121 @@
+/**
+ * Supervisor / Admin — happy-path smoke tests.
+ *
+ * Role: admin (supervisor@ramaiahcare.com)
+ * Landing: / → Dashboard
+ * Pages: Dashboard, Team Performance, Live Monitor,
+ *        Leads, Patients, Appointments, Call Log,
+ *        Call Recordings, Missed Calls, Campaigns, Settings
+ */
+import { test, expect } from '@playwright/test';
+import { loginAs, waitForApp } from './helpers';
+
+const EMAIL = 'supervisor@ramaiahcare.com';
+const PASSWORD = 'MrRamaiah@2026';
+
+test.describe('Supervisor Smoke', () => {
+    test.beforeEach(async ({ page }) => {
+        await loginAs(page, EMAIL, PASSWORD);
+    });
+
+    test('lands on Dashboard after login', async ({ page }) => {
+        await expect(page.locator('aside').first()).toBeVisible();
+        // Verify we're authenticated and on the app
+        await expect(page).not.toHaveURL(/\/login/);
+    });
+
+    test('Team Performance loads', async ({ page }) => {
+        await page.goto('/team-performance');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Team|Performance|Agent|No data/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Live Call Monitor loads', async ({ page }) => {
+        await page.goto('/live-monitor');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Live|Monitor|Active|No active/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Leads page loads', async ({ page }) => {
+        await page.goto('/leads');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Lead|No leads/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Patients page loads', async ({ page }) => {
+        await page.goto('/patients');
+        await waitForApp(page);
+
+        const search = page.getByPlaceholder(/search/i).or(page.getByLabel(/search/i));
+        await expect(search.first()).toBeVisible();
+    });
+
+    test('Appointments page loads', async ({ page }) => {
+        await page.goto('/appointments');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Appointment|Schedule|No appointment/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Call Log page loads', async ({ page }) => {
+        await page.goto('/call-history');
+        await waitForApp(page);
+
+        await expect(page.locator('text="Call History"').first()).toBeVisible();
+    });
+
+    test('Call Recordings page loads', async ({ page }) => {
+        await page.goto('/call-recordings');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Recording|No recording/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Missed Calls page loads', async ({ page }) => {
+        await page.goto('/missed-calls');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Missed|No missed/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Campaigns page loads', async ({ page }) => {
+        await page.goto('/campaigns');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Campaign|No campaign/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('Settings page loads', async ({ page }) => {
+        await page.goto('/settings');
+        await waitForApp(page);
+
+        await expect(
+            page.locator('text=/Settings|Configuration/i').first(),
+        ).toBeVisible({ timeout: 10_000 });
+    });
+
+    test('sidebar has expected nav items', async ({ page }) => {
+        const sidebar = page.locator('aside').first();
+        // Check key items — exact labels depend on the role the sidecar assigns
+        for (const item of ['Patients', 'Appointments', 'Campaigns']) {
+            await expect(sidebar.locator(`text="${item}"`)).toBeVisible();
+        }
+    });
+});
--- a/package-lock.json
+++ b/package-lock.json
@@ -42,6 +42,7 @@
                "tailwindcss-react-aria-components": "^2.0.1"
            },
            "devDependencies": {
+                "@playwright/test": "^1.59.1",
                "@tailwindcss/postcss": "^4.1.18",
                "@trivago/prettier-plugin-sort-imports": "^5.2.2",
                "@types/jssip": "^3.5.3",
@@ -1077,6 +1078,22 @@
                "@jridgewell/sourcemap-codec": "^1.4.14"
            }
        },
+        "node_modules/@playwright/test": {
+            "version": "1.59.1",
+            "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.59.1.tgz",
+            "integrity": "sha512-PG6q63nQg5c9rIi4/Z5lR5IVF7yU5MqmKaPOe0HSc0O2cX1fPi96sUQu5j7eo4gKCkB2AnNGoWt7y4/Xx3Kcqg==",
+            "dev": true,
+            "license": "Apache-2.0",
+            "dependencies": {
+                "playwright": "1.59.1"
+            },
+            "bin": {
+                "playwright": "cli.js"
+            },
+            "engines": {
+                "node": ">=18"
+            }
+        },
        "node_modules/@react-aria/autocomplete": {
            "version": "3.0.0-rc.6",
            "resolved": "http://localhost:4873/@react-aria/autocomplete/-/autocomplete-3.0.0-rc.6.tgz",
@@ -5471,6 +5488,53 @@
                "url": "https://github.com/sponsors/jonschlinkert"
            }
        },
+        "node_modules/playwright": {
+            "version": "1.59.1",
+            "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.59.1.tgz",
+            "integrity": "sha512-C8oWjPR3F81yljW9o5OxcWzfh6avkVwDD2VYdwIGqTkl+OGFISgypqzfu7dOe4QNLL2aqcWBmI3PMtLIK233lw==",
+            "dev": true,
+            "license": "Apache-2.0",
+            "dependencies": {
+                "playwright-core": "1.59.1"
+            },
+            "bin": {
+                "playwright": "cli.js"
+            },
+            "engines": {
+                "node": ">=18"
+            },
+            "optionalDependencies": {
+                "fsevents": "2.3.2"
+            }
+        },
+        "node_modules/playwright-core": {
+            "version": "1.59.1",
+            "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.59.1.tgz",
+            "integrity": "sha512-HBV/RJg81z5BiiZ9yPzIiClYV/QMsDCKUyogwH9p3MCP6IYjUFu/MActgYAvK0oWyV9NlwM3GLBjADyWgydVyg==",
+            "dev": true,
+            "license": "Apache-2.0",
+            "bin": {
+                "playwright-core": "cli.js"
+            },
+            "engines": {
+                "node": ">=18"
+            }
+        },
+        "node_modules/playwright/node_modules/fsevents": {
+            "version": "2.3.2",
+            "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
+            "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
+            "dev": true,
+            "hasInstallScript": true,
+            "license": "MIT",
+            "optional": true,
+            "os": [
+                "darwin"
+            ],
+            "engines": {
+                "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
+            }
+        },
        "node_modules/postcss": {
            "version": "8.5.8",
            "resolved": "http://localhost:4873/postcss/-/postcss-8.5.8.tgz",
--- a/package.json
+++ b/package.json
@@ -43,6 +43,7 @@
        "tailwindcss-react-aria-components": "^2.0.1"
    },
    "devDependencies": {
+        "@playwright/test": "^1.59.1",
        "@tailwindcss/postcss": "^4.1.18",
        "@trivago/prettier-plugin-sort-imports": "^5.2.2",
        "@types/jssip": "^3.5.3",
--- a/playwright.config.ts
+++ b/playwright.config.ts
@@ -0,0 +1,51 @@
+import { defineConfig } from '@playwright/test';
+import path from 'path';
+import { fileURLToPath } from 'url';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+export default defineConfig({
+    testDir: './e2e',
+    timeout: 60_000,
+    expect: { timeout: 10_000 },
+    retries: 1,
+    workers: 1,
+    reporter: [['html', { open: 'never' }], ['list']],
+    use: {
+        baseURL: process.env.E2E_BASE_URL ?? 'https://ramaiah.engage.healix360.net',
+        headless: true,
+        screenshot: 'only-on-failure',
+        trace: 'on-first-retry',
+        actionTimeout: 8_000,
+        navigationTimeout: 15_000,
+    },
+    projects: [
+        // Login tests run first — fresh browser, no saved auth
+        {
+            name: 'login',
+            testMatch: /agent-login\.spec\.ts/,
+            use: { browserName: 'chromium' },
+        },
+        // Auth setup — saves CC agent session for reuse
+        {
+            name: 'agent-setup',
+            testMatch: /auth\.setup\.ts/,
+        },
+        // CC Agent feature tests — reuse saved auth
+        {
+            name: 'cc-agent',
+            dependencies: ['agent-setup'],
+            use: {
+                storageState: path.join(__dirname, 'e2e/.auth/agent.json'),
+                browserName: 'chromium',
+            },
+            testMatch: /agent-smoke\.spec\.ts/,
+        },
+        // Supervisor tests — logs in fresh each run
+        {
+            name: 'supervisor',
+            testMatch: /supervisor-smoke\.spec\.ts/,
+            use: { browserName: 'chromium' },
+        },
+    ],
+});