helix-engage/docs/developer-operations-runbook.md

# Helix Engage — Developer Operations Runbook

## Architecture

```
Browser (India)
    ↓ HTTPS
Caddy (reverse proxy, TLS, static files)
    ├── engage.srv1477139.hstgr.cloud → /srv/engage (static frontend)
    ├── engage-api.srv1477139.hstgr.cloud → sidecar:4100
    └── *.srv1477139.hstgr.cloud → server:4000 (platform)
    
Docker Compose stack:
    ├── caddy          — Reverse proxy + TLS
    ├── server         — FortyTwo platform (ECR image)
    ├── worker         — Background jobs
    ├── sidecar        — Helix Engage NestJS API (ECR image)
    ├── db             — PostgreSQL 16
    ├── redis          — Session + cache
    ├── clickhouse     — Analytics
    ├── minio          — Object storage
    └── redpanda       — Event bus (Kafka)
```

## VPS Access

```bash
# SSH into the VPS
sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184

# Or with SSH key (if configured)
ssh -i ~/Downloads/fortytwoai_hostinger root@148.230.67.184
```

| Detail | Value |
|---|---|
| Host | 148.230.67.184 |
| User | root |
| Password | SasiSuman@2007 |
| Docker compose dir | /opt/fortytwo |
| Frontend static files | /opt/fortytwo/helix-engage-frontend |
| Caddyfile | /opt/fortytwo/Caddyfile |

## URLs

| Service | URL |
|---|---|
| Frontend | https://engage.srv1477139.hstgr.cloud |
| Sidecar API | https://engage-api.srv1477139.hstgr.cloud |
| Platform | https://fortytwo-dev.srv1477139.hstgr.cloud |

## Login Credentials

| Role | Email | Password |
|---|---|---|
| CC Agent | rekha.cc@globalhospital.com | Global@123 |
| CC Agent | ganesh.cc@globalhospital.com | Global@123 |
| Marketing | sanjay.marketing@globalhospital.com | Global@123 |
| Admin/Supervisor | dr.ramesh@globalhospital.com | Global@123 |

---

## Local Testing

Always test locally before deploying to staging.

### Frontend (Vite dev server)

```bash
cd helix-engage

# Start dev server (hot reload)
npm run dev
# → http://localhost:5173

# Type check (catches production build errors)
npx tsc --noEmit

# Production build (same as deploy)
npm run build
```

The `.env.local` controls which sidecar the frontend talks to:
```bash
# Remote sidecar (default — uses deployed backend)
VITE_API_URL=https://engage-api.srv1477139.hstgr.cloud
VITE_SIDECAR_URL=https://engage-api.srv1477139.hstgr.cloud

# Local sidecar (for testing sidecar changes)
# VITE_API_URL=http://localhost:4100
# VITE_SIDECAR_URL=http://localhost:4100

# Split — theme endpoint local, everything else remote
# VITE_THEME_API_URL=http://localhost:4100
```

**Important:** When `VITE_API_URL` points to `localhost:4100`, login and GraphQL only work if the local sidecar can reach the platform. The local sidecar's `.env` must have valid `PLATFORM_GRAPHQL_URL` and `PLATFORM_API_KEY`.

### Sidecar (NestJS dev server)

```bash
cd helix-engage-server

# Start with watch mode (auto-restart on changes)
npm run start:dev
# → http://localhost:4100

# Build only (no run)
npm run build

# Production start
npm run start:prod
```

The sidecar `.env` must have:
```bash
PLATFORM_GRAPHQL_URL=...     # Platform GraphQL endpoint
PLATFORM_API_KEY=...         # Platform API key for server-to-server calls
PLATFORM_WORKSPACE_SUBDOMAIN=fortytwo-dev
REDIS_URL=redis://localhost:6379   # Local Redis required
```

### Local Docker stack (full environment)

For testing with a local platform + database + Redis:

```bash
cd helix-engage-local

# First time — pull images + start
./deploy-local.sh up

# Deploy frontend to local stack
./deploy-local.sh frontend

# Deploy sidecar to local stack
./deploy-local.sh sidecar

# Both
./deploy-local.sh all

# Logs
./deploy-local.sh logs

# Stop
./deploy-local.sh down
```

Local stack URLs:
- Platform: `http://localhost:5001`
- Sidecar: `http://localhost:5100`
- Frontend: `http://localhost:5080`

### Pre-deploy checklist

Before running `deploy.sh`:

1. `npx tsc --noEmit` — passes with no errors (frontend)
2. `npm run build` — succeeds (sidecar)
3. Test the changed feature locally (dev server or local stack)
4. Check `package.json` for new dependencies → decides quick vs full deploy

---

## Deployment

### Prerequisites (local machine)

```bash
# Required tools
brew install sshpass         # SSH with password
aws configure                # AWS CLI (for ECR)
docker desktop               # Docker with buildx

# Verify AWS access
aws sts get-caller-identity  # Should show account 043728036361
```

### Path 1: Quick Deploy (no new dependencies)

Use when only code changes — no new npm packages.

```bash
cd /path/to/fortytwo-eap

# Deploy frontend only
bash deploy.sh frontend

# Deploy sidecar only
bash deploy.sh sidecar

# Deploy both
bash deploy.sh all
```

**What it does:**
- Frontend: `npm run build` → tar `dist/` → SCP to VPS → extract to `/opt/fortytwo/helix-engage-frontend`
- Sidecar: `nest build` → tar `dist/` + `src/` → docker cp into running container → `docker compose restart sidecar`

### Path 2: Full Deploy (new dependencies)

Use when `package.json` changed (new npm packages added).

```bash
cd /path/to/fortytwo-eap/helix-engage-server

# 1. Login to ECR
aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com

# 2. Build cross-platform image and push
docker buildx build --platform linux/amd64 \
  -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \
  --push .

# 3. Pull and restart on VPS
ECR_TOKEN=$(aws ecr get-login-password --region ap-south-1)
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "
  echo '$ECR_TOKEN' | docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com
  cd /opt/fortytwo
  docker compose pull sidecar
  docker compose up -d sidecar
"
```

### How to decide which path

```
Did package.json change?
  ├── YES → Path 2 (ECR build + push + pull)
  └── NO → Path 1 (deploy.sh)
```

---

## Checking Logs

### Sidecar logs

```bash
# SSH into VPS first, or run remotely:
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 --tail 30"

# Follow live
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 -f --tail 10"

# Filter for errors
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 --tail 100 2>&1 | grep -i error"

# Via deploy.sh
bash deploy.sh logs
```

### Caddy logs

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-caddy-1 --tail 30"
```

### Platform server logs

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-server-1 --tail 30"
```

### All container status

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
```

---

## Health Checks

### Sidecar healthy startup

Look for these lines in sidecar logs:
```
[NestApplication] Nest application successfully started
Helix Engage Server running on port 4100
[SessionService] Redis connected
[ThemeService] Theme loaded from file  (or "Using default theme")
[RulesStorageService] Initialized empty rules config
```

### Common failure patterns

| Log pattern | Meaning | Fix |
|---|---|---|
| `Cannot find module 'xxx'` | Missing npm dependency | Path 2 deploy (rebuild ECR image) |
| `UndefinedModuleException` | Circular dependency or missing import | Fix code, redeploy |
| `ECONNREFUSED redis:6379` | Redis not ready | `docker compose restart redis sidecar` |
| `Forbidden resource` | Platform permission issue | Check user roles |
| `429 Too Many Requests` | Ozonetel rate limit | Wait, reduce polling frequency |

---

## Redis Cache Operations

### Clear caller resolution cache

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli KEYS 'caller:*'"

# Clear all caller cache
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'caller:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL"
```

### Clear recording analysis cache

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'call:analysis:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL"
```

### Clear agent name cache

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'agent:name:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL"
```

### Clear all session/cache keys

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli FLUSHDB"
```

---

## Database Access

```bash
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-db-1 psql -U fortytwo -d fortytwo_staging"
```

### Useful queries

```sql
-- List workspace schemas
SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';

-- List custom entities
SELECT "nameSingular", "isCustom" FROM core."objectMetadata" ORDER BY "nameSingular";

-- List users
SELECT u.email, u."firstName", u."lastName", uw.id as workspace_id
FROM core."user" u
JOIN core."userWorkspace" uw ON uw."userId" = u.id;

-- List roles
SELECT r.label, rt."userWorkspaceId"
FROM core."roleTarget" rt
JOIN core."role" r ON r.id = rt."roleId";
```

---

## Rollback

### Frontend rollback

The previous frontend build is overwritten. To rollback:
1. Checkout the previous git commit
2. `npm run build`
3. `bash deploy.sh frontend`

### Sidecar rollback (quick deploy)

Same as frontend — checkout previous commit, rebuild, redeploy.

### Sidecar rollback (ECR)

```bash
# Tag the current image as rollback
# Then re-tag the previous image as :alpha
# Or use a specific tag/digest

# On VPS:
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "
  cd /opt/fortytwo
  docker compose restart sidecar
"
```

---

## Theme Management

### View current theme
```bash
curl -s https://engage-api.srv1477139.hstgr.cloud/api/config/theme | python3 -m json.tool
```

### Reset theme to defaults
```bash
curl -s -X POST https://engage-api.srv1477139.hstgr.cloud/api/config/theme/reset | python3 -m json.tool
```

### Theme backups
Stored on the sidecar container at `/app/data/theme-backups/`. Each save creates a timestamped backup.

---

## Git Repositories

| Repo | Azure DevOps URL | Branch |
|---|---|---|
| Frontend | `https://dev.azure.com/globalhealthx/EMR/_git/helix-engage` | `dev` |
| Sidecar | `https://dev.azure.com/globalhealthx/EMR/_git/helix-engage-server` | `dev` |
| SDK App | `FortyTwoApps/helix-engage/` (in fortytwo-eap monorepo) | `dev` |

### Commit and push pattern
```bash
# Frontend
cd helix-engage
git add -A && git commit -m "feat: description" && git push origin dev

# Sidecar
cd helix-engage-server
git add -A && git commit -m "feat: description" && git push origin dev
```

---

## ECR Details

| Detail | Value |
|---|---|
| Registry | 043728036361.dkr.ecr.ap-south-1.amazonaws.com |
| Repository | fortytwo-eap/helix-engage-sidecar |
| Tag | alpha |
| Region | ap-south-1 (Mumbai) |