helix-engage/docs/architecture.md

# Helix Engage — Architecture

Single EC2 instance (Mumbai `ap-south-1`) hosting two isolated Helix Engage
workspaces on top of a shared FortyTwo platform. Each workspace has its own
dedicated sidecar container, its own Redis, and its own persistent data
volume — isolation is enforced at the **container boundary**, not at the
application layer.

**Host:** `13.234.31.194` (m6i.xlarge, Ubuntu 22.04)
**DNS:** Cloudflare zone `healix360.net`
**TLS:** Caddy + Let's Encrypt, HTTP-01 challenge per hostname

---

## Principles

1. **Platform is multi-tenant by design.** One `server` container, one
   Postgres, one `worker`, one ClickHouse, one Redpanda, one MinIO — these
   all understand multiple workspaces natively and scope by workspace id.

2. **Sidecar is single-tenant by design.** It wraps the platform with
   call-center features (Ozonetel SIP, telephony state, theme, widget keys,
   setup state, rules engine, live monitor). Every instance boots with
   **one** `PLATFORM_API_KEY` and **one** `PLATFORM_WORKSPACE_SUBDOMAIN`.
   We run one instance per workspace.

3. **Caddy is strictly host-routed.** No default or catchall tenant.
   A request lands on a host block or it 404s. The apex
   `engage.healix360.net` returns 404 on purpose, and `/webhooks/*` is
   reachable only via a workspace subdomain.

4. **Redis is per-sidecar.** Sidecars share Redis key names without a
   workspace dimension. Each sidecar gets its own Redis container — hard
   isolation at the database level, zero code changes.

5. **Telephony dispatcher routes events by agent.** Ozonetel event
   subscriptions are account-level (not per-campaign). A single dispatcher
   receives all agent/call events and routes them to the correct sidecar
   using Redis-backed service discovery.

---

## URL Layout

| Who | URL | Routes to |
|---|---|---|
| Ramaiah platform UI | `https://ramaiah.app.healix360.net` | `server:4000` |
| Ramaiah Helix Engage | `https://ramaiah.engage.healix360.net` | `sidecar-ramaiah:4100` |
| Global platform UI | `https://global.app.healix360.net` | `server:4000` |
| Global Helix Engage | `https://global.engage.healix360.net` | `sidecar-global:4100` |
| Telephony dispatcher | `https://telephony.engage.healix360.net` | `telephony:4200` |
| Apex (dead-end) | `https://engage.healix360.net` | `404` |

Ozonetel campaign webhook URLs — per tenant:

| Campaign | DID | Webhook URL |
|---|---|---|
| `Inbound_918041763400` | Ramaiah | `https://ramaiah.engage.healix360.net/webhooks/ozonetel/missed-call` |
| `Inbound_918041763265` | Global (on VPS until cutover) | `https://global.engage.healix360.net/webhooks/ozonetel/missed-call` |

Ozonetel event subscription (account-level):

| Event | URL |
|---|---|
| Agent events | `https://telephony.engage.healix360.net/api/supervisor/agent-event` |
| Call events | `https://telephony.engage.healix360.net/api/supervisor/call-event` |

---

## Diagram

```mermaid
flowchart TB
    subgraph Internet
        OZO[Ozonetel<br/>CCaaS]
        USR_R[Ramaiah users]
        USR_G[Global users]
    end

    subgraph EC2 ["EC2 — 13.234.31.194 (ap-south-1)"]
        CADDY{{"caddy<br/>host-routed<br/>Let's Encrypt"}}

        subgraph TEL ["Telephony Dispatcher"]
            DISP["telephony<br/>NestJS:4200<br/>routes by agentId"]
            RD_T[("redis-telephony")]
        end

        subgraph PLATFORM ["Platform (shared, multi-tenant)"]
            SRV["server<br/>NestJS:4000<br/>platform API + SPA"]
            WKR["worker<br/>BullMQ"]
            DB[("db<br/>postgres:16<br/>workspace-per-schema")]
            CH[("clickhouse<br/>analytics")]
            RP[("redpanda<br/>event bus")]
            MINIO[("minio<br/>S3 storage")]
        end

        subgraph RAMAIAH ["Ramaiah tenant (isolated)"]
            SC_R["sidecar-ramaiah<br/>NestJS:4100<br/>API_KEY=ramaiah admin"]
            RD_R[("redis-ramaiah")]
            VOL_R[/"data-ramaiah volume<br/>theme, telephony,<br/>widget, rules,<br/>setup-state"/]
        end

        subgraph GLOBAL ["Global tenant (isolated)"]
            SC_G["sidecar-global<br/>NestJS:4100<br/>API_KEY=global admin"]
            RD_G[("redis-global")]
            VOL_G[/"data-global volume<br/>theme, telephony,<br/>widget, rules,<br/>setup-state"/]
        end
    end

    USR_R -->|"ramaiah.app.healix360.net"| CADDY
    USR_R -->|"ramaiah.engage.healix360.net"| CADDY
    USR_G -->|"global.app.healix360.net"| CADDY
    USR_G -->|"global.engage.healix360.net"| CADDY

    OZO -->|"webhooks/ozonetel/missed-call<br/>(Ramaiah DID 918041763400)"| CADDY
    OZO -.->|"webhooks/ozonetel/missed-call<br/>(Global DID 918041763265<br/>— still on VPS today)"| CADDY
    OZO -->|"agent + call events<br/>(account-level subscription)"| CADDY
    CADDY -->|"telephony.engage.*"| DISP

    CADDY -->|"*.app.healix360.net<br/>/graphql, /auth/*, SPA"| SRV
    CADDY -->|"ramaiah.engage.*<br/>/api/*, /webhooks/*, SPA"| SC_R
    CADDY -->|"global.engage.*<br/>/api/*, /webhooks/*, SPA"| SC_G

    DISP -->|"agentId lookup<br/>→ forward to sidecar"| SC_R
    DISP -->|"agentId lookup<br/>→ forward to sidecar"| SC_G
    DISP --- RD_T

    SC_R -->|"self-register on boot<br/>heartbeat 30s"| DISP
    SC_G -->|"self-register on boot<br/>heartbeat 30s"| DISP

    SC_R -->|"GraphQL<br/>Origin: ramaiah.app.*"| SRV
    SC_G -->|"GraphQL<br/>Origin: global.app.*"| SRV

    SC_R --- RD_R
    SC_G --- RD_G
    SC_R --- VOL_R
    SC_G --- VOL_G

    SRV --- DB
    SRV --- CH
    SRV --- RP
    SRV --- MINIO
    WKR --- DB
    WKR --- RP

    classDef shared fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#000
    classDef ramaiah fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000
    classDef global fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#000
    classDef external fill:#f5f5f5,stroke:#757575,stroke-width:1px,color:#000
    classDef edge fill:#fce4ec,stroke:#c2185b,stroke-width:2px,color:#000
    classDef telephony fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000

    class SRV,WKR,DB,CH,RP,MINIO shared
    class SC_R,RD_R,VOL_R ramaiah
    class SC_G,RD_G,VOL_G global
    class OZO,USR_R,USR_G external
    class CADDY edge
    class DISP,RD_T telephony
```

---

## Components

| Component | Scope | Container count | Purpose |
|---|---|---|---|
| Caddy | Shared | 1 | Host-routed reverse proxy, TLS terminator |
| Platform server | Shared | 1 | Natively multi-tenant by Origin/subdomain |
| Platform worker | Shared | 1 | BullMQ jobs carry workspace context per-job |
| Postgres | Shared | 1 | Multi-tenant via per-workspace schemas |
| ClickHouse | Shared | 1 | Analytics — workspace dimension per event |
| Redpanda | Shared | 1 | Event bus — workspace dimension per message |
| MinIO | Shared | 1 | S3-compatible storage |
| **Telephony dispatcher** | **Shared** | **1** | Routes Ozonetel events to correct sidecar by agentId |
| **Redis (telephony)** | **Shared** | **1** | Service discovery registry for dispatcher |
| **Sidecar** | **Per-tenant** | **2** | Call center layer (Ramaiah + Global) |
| **Redis (sidecar)** | **Per-tenant** | **2** | Session, agent state, theme, rules cache |
| **Data volume** | **Per-tenant** | **2** | File-based config in `/app/data/` |

---

## Telephony Event Flow

Ozonetel event subscriptions are **account-level** — one subscription per Ozonetel account, not per campaign. All agent login/logout/state events and call events are POSTed to a single URL.

```
Ozonetel → POST telephony.engage.healix360.net/api/supervisor/agent-event
  → Dispatcher receives { agentId: "ramaiahadmin", action: "incall", ... }
  → Redis lookup: agentId "ramaiahadmin" → sidecar-ramaiah:4100
  → Forward event to sidecar-ramaiah
  → sidecar-ramaiah updates SupervisorService state, emits SSE
```

**Service discovery:** Each sidecar self-registers on boot via `POST /api/supervisor/register` with its agent list. Heartbeat every 30s, TTL 90s. If a sidecar goes down, its entries expire and the dispatcher stops routing to it.

---

## Request Flow

### Agent opens Ramaiah Helix Engage
```
Browser → https://ramaiah.engage.healix360.net/
  → Caddy (TLS, Host=ramaiah.engage.healix360.net)
  → static SPA from /srv/engage

Browser → POST /api/auth/login { email, password }
  → Caddy → sidecar-ramaiah:4100
  → sidecar calls platform with:
      Origin: https://ramaiah.app.healix360.net
      Authorization: Bearer <Ramaiah API key>
  → platform resolves workspace by Origin → Ramaiah
  → JWT returned
```

### Ozonetel POSTs a missed-call webhook
```
Ozonetel → POST https://ramaiah.engage.healix360.net/webhooks/ozonetel/missed-call
  → Caddy (Host=ramaiah.engage.healix360.net)
  → sidecar-ramaiah:4100 ONLY
  → writes call row into Ramaiah workspace via platform
```

Cross-tenant leakage is physically impossible — Caddy's host-routing guarantees a Ramaiah webhook can never reach sidecar-global.

---

## Failure Modes

| Failure | Blast radius |
|---|---|
| `sidecar-ramaiah` crashes | Ramaiah Engage 502s. Global + platform unaffected. |
| `sidecar-global` crashes | Global Engage 502s. Ramaiah + platform unaffected. |
| `redis-ramaiah` crashes | Ramaiah agents kicked from SIP. Global unaffected. |
| `telephony` crashes | Agent/call state events stop routing. Sidecars still serve UI. |
| `server` (platform) crashes | **Both workspaces** down for data. |
| `db` crashes | Same as above. |
| Caddy crashes | Nothing reachable until restart. |

---

## Adding a New Hospital

1. Add sidecar container + Redis + data volume in `docker-compose.yml`
2. Add Caddy host block for `newhospital.engage.healix360.net`
3. Create workspace on platform, generate API key
4. Set sidecar env: `PLATFORM_API_KEY`, `PLATFORM_WORKSPACE_SUBDOMAIN`
5. Configure Ozonetel campaign webhook to `newhospital.engage.healix360.net/webhooks/ozonetel/missed-call`
6. Sidecar self-registers with telephony dispatcher on boot — no dispatcher config needed