diff --git a/docs/developer-operations-runbook.md b/docs/developer-operations-runbook.md new file mode 100644 index 0000000..98ff151 --- /dev/null +++ b/docs/developer-operations-runbook.md @@ -0,0 +1,431 @@ +# Helix Engage — Developer Operations Runbook + +## Architecture + +``` +Browser (India) + ↓ HTTPS +Caddy (reverse proxy, TLS, static files) + ├── engage.srv1477139.hstgr.cloud → /srv/engage (static frontend) + ├── engage-api.srv1477139.hstgr.cloud → sidecar:4100 + └── *.srv1477139.hstgr.cloud → server:4000 (platform) + +Docker Compose stack: + ├── caddy — Reverse proxy + TLS + ├── server — FortyTwo platform (ECR image) + ├── worker — Background jobs + ├── sidecar — Helix Engage NestJS API (ECR image) + ├── db — PostgreSQL 16 + ├── redis — Session + cache + ├── clickhouse — Analytics + ├── minio — Object storage + └── redpanda — Event bus (Kafka) +``` + +## VPS Access + +```bash +# SSH into the VPS +sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184 + +# Or with SSH key (if configured) +ssh -i ~/Downloads/fortytwoai_hostinger root@148.230.67.184 +``` + +| Detail | Value | +|---|---| +| Host | 148.230.67.184 | +| User | root | +| Password | SasiSuman@2007 | +| Docker compose dir | /opt/fortytwo | +| Frontend static files | /opt/fortytwo/helix-engage-frontend | +| Caddyfile | /opt/fortytwo/Caddyfile | + +## URLs + +| Service | URL | +|---|---| +| Frontend | https://engage.srv1477139.hstgr.cloud | +| Sidecar API | https://engage-api.srv1477139.hstgr.cloud | +| Platform | https://fortytwo-dev.srv1477139.hstgr.cloud | + +## Login Credentials + +| Role | Email | Password | +|---|---|---| +| CC Agent | rekha.cc@globalhospital.com | Global@123 | +| CC Agent | ganesh.cc@globalhospital.com | Global@123 | +| Marketing | sanjay.marketing@globalhospital.com | Global@123 | +| Admin/Supervisor | dr.ramesh@globalhospital.com | Global@123 | + +--- + +## Local Testing + +Always test locally before deploying to staging. + +### Frontend (Vite dev server) + +```bash +cd helix-engage + +# Start dev server (hot reload) +npm run dev +# → http://localhost:5173 + +# Type check (catches production build errors) +npx tsc --noEmit + +# Production build (same as deploy) +npm run build +``` + +The `.env.local` controls which sidecar the frontend talks to: +```bash +# Remote sidecar (default — uses deployed backend) +VITE_API_URL=https://engage-api.srv1477139.hstgr.cloud +VITE_SIDECAR_URL=https://engage-api.srv1477139.hstgr.cloud + +# Local sidecar (for testing sidecar changes) +# VITE_API_URL=http://localhost:4100 +# VITE_SIDECAR_URL=http://localhost:4100 + +# Split — theme endpoint local, everything else remote +# VITE_THEME_API_URL=http://localhost:4100 +``` + +**Important:** When `VITE_API_URL` points to `localhost:4100`, login and GraphQL only work if the local sidecar can reach the platform. The local sidecar's `.env` must have valid `PLATFORM_GRAPHQL_URL` and `PLATFORM_API_KEY`. + +### Sidecar (NestJS dev server) + +```bash +cd helix-engage-server + +# Start with watch mode (auto-restart on changes) +npm run start:dev +# → http://localhost:4100 + +# Build only (no run) +npm run build + +# Production start +npm run start:prod +``` + +The sidecar `.env` must have: +```bash +PLATFORM_GRAPHQL_URL=... # Platform GraphQL endpoint +PLATFORM_API_KEY=... # Platform API key for server-to-server calls +PLATFORM_WORKSPACE_SUBDOMAIN=fortytwo-dev +REDIS_URL=redis://localhost:6379 # Local Redis required +``` + +### Local Docker stack (full environment) + +For testing with a local platform + database + Redis: + +```bash +cd helix-engage-local + +# First time — pull images + start +./deploy-local.sh up + +# Deploy frontend to local stack +./deploy-local.sh frontend + +# Deploy sidecar to local stack +./deploy-local.sh sidecar + +# Both +./deploy-local.sh all + +# Logs +./deploy-local.sh logs + +# Stop +./deploy-local.sh down +``` + +Local stack URLs: +- Platform: `http://localhost:5001` +- Sidecar: `http://localhost:5100` +- Frontend: `http://localhost:5080` + +### Pre-deploy checklist + +Before running `deploy.sh`: + +1. `npx tsc --noEmit` — passes with no errors (frontend) +2. `npm run build` — succeeds (sidecar) +3. Test the changed feature locally (dev server or local stack) +4. Check `package.json` for new dependencies → decides quick vs full deploy + +--- + +## Deployment + +### Prerequisites (local machine) + +```bash +# Required tools +brew install sshpass # SSH with password +aws configure # AWS CLI (for ECR) +docker desktop # Docker with buildx + +# Verify AWS access +aws sts get-caller-identity # Should show account 043728036361 +``` + +### Path 1: Quick Deploy (no new dependencies) + +Use when only code changes — no new npm packages. + +```bash +cd /path/to/fortytwo-eap + +# Deploy frontend only +bash deploy.sh frontend + +# Deploy sidecar only +bash deploy.sh sidecar + +# Deploy both +bash deploy.sh all +``` + +**What it does:** +- Frontend: `npm run build` → tar `dist/` → SCP to VPS → extract to `/opt/fortytwo/helix-engage-frontend` +- Sidecar: `nest build` → tar `dist/` + `src/` → docker cp into running container → `docker compose restart sidecar` + +### Path 2: Full Deploy (new dependencies) + +Use when `package.json` changed (new npm packages added). + +```bash +cd /path/to/fortytwo-eap/helix-engage-server + +# 1. Login to ECR +aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com + +# 2. Build cross-platform image and push +docker buildx build --platform linux/amd64 \ + -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \ + --push . + +# 3. Pull and restart on VPS +ECR_TOKEN=$(aws ecr get-login-password --region ap-south-1) +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 " + echo '$ECR_TOKEN' | docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com + cd /opt/fortytwo + docker compose pull sidecar + docker compose up -d sidecar +" +``` + +### How to decide which path + +``` +Did package.json change? + ├── YES → Path 2 (ECR build + push + pull) + └── NO → Path 1 (deploy.sh) +``` + +--- + +## Checking Logs + +### Sidecar logs + +```bash +# SSH into VPS first, or run remotely: +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 --tail 30" + +# Follow live +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 -f --tail 10" + +# Filter for errors +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 --tail 100 2>&1 | grep -i error" + +# Via deploy.sh +bash deploy.sh logs +``` + +### Caddy logs + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-caddy-1 --tail 30" +``` + +### Platform server logs + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-server-1 --tail 30" +``` + +### All container status + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'" +``` + +--- + +## Health Checks + +### Sidecar healthy startup + +Look for these lines in sidecar logs: +``` +[NestApplication] Nest application successfully started +Helix Engage Server running on port 4100 +[SessionService] Redis connected +[ThemeService] Theme loaded from file (or "Using default theme") +[RulesStorageService] Initialized empty rules config +``` + +### Common failure patterns + +| Log pattern | Meaning | Fix | +|---|---|---| +| `Cannot find module 'xxx'` | Missing npm dependency | Path 2 deploy (rebuild ECR image) | +| `UndefinedModuleException` | Circular dependency or missing import | Fix code, redeploy | +| `ECONNREFUSED redis:6379` | Redis not ready | `docker compose restart redis sidecar` | +| `Forbidden resource` | Platform permission issue | Check user roles | +| `429 Too Many Requests` | Ozonetel rate limit | Wait, reduce polling frequency | + +--- + +## Redis Cache Operations + +### Clear caller resolution cache + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli KEYS 'caller:*'" + +# Clear all caller cache +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'caller:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL" +``` + +### Clear recording analysis cache + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'call:analysis:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL" +``` + +### Clear agent name cache + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'agent:name:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL" +``` + +### Clear all session/cache keys + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli FLUSHDB" +``` + +--- + +## Database Access + +```bash +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-db-1 psql -U fortytwo -d fortytwo_staging" +``` + +### Useful queries + +```sql +-- List workspace schemas +SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%'; + +-- List custom entities +SELECT "nameSingular", "isCustom" FROM core."objectMetadata" ORDER BY "nameSingular"; + +-- List users +SELECT u.email, u."firstName", u."lastName", uw.id as workspace_id +FROM core."user" u +JOIN core."userWorkspace" uw ON uw."userId" = u.id; + +-- List roles +SELECT r.label, rt."userWorkspaceId" +FROM core."roleTarget" rt +JOIN core."role" r ON r.id = rt."roleId"; +``` + +--- + +## Rollback + +### Frontend rollback + +The previous frontend build is overwritten. To rollback: +1. Checkout the previous git commit +2. `npm run build` +3. `bash deploy.sh frontend` + +### Sidecar rollback (quick deploy) + +Same as frontend — checkout previous commit, rebuild, redeploy. + +### Sidecar rollback (ECR) + +```bash +# Tag the current image as rollback +# Then re-tag the previous image as :alpha +# Or use a specific tag/digest + +# On VPS: +sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 " + cd /opt/fortytwo + docker compose restart sidecar +" +``` + +--- + +## Theme Management + +### View current theme +```bash +curl -s https://engage-api.srv1477139.hstgr.cloud/api/config/theme | python3 -m json.tool +``` + +### Reset theme to defaults +```bash +curl -s -X POST https://engage-api.srv1477139.hstgr.cloud/api/config/theme/reset | python3 -m json.tool +``` + +### Theme backups +Stored on the sidecar container at `/app/data/theme-backups/`. Each save creates a timestamped backup. + +--- + +## Git Repositories + +| Repo | Azure DevOps URL | Branch | +|---|---|---| +| Frontend | `https://dev.azure.com/globalhealthx/EMR/_git/helix-engage` | `dev` | +| Sidecar | `https://dev.azure.com/globalhealthx/EMR/_git/helix-engage-server` | `dev` | +| SDK App | `FortyTwoApps/helix-engage/` (in fortytwo-eap monorepo) | `dev` | + +### Commit and push pattern +```bash +# Frontend +cd helix-engage +git add -A && git commit -m "feat: description" && git push origin dev + +# Sidecar +cd helix-engage-server +git add -A && git commit -m "feat: description" && git push origin dev +``` + +--- + +## ECR Details + +| Detail | Value | +|---|---| +| Registry | 043728036361.dkr.ecr.ap-south-1.amazonaws.com | +| Repository | fortytwo-eap/helix-engage-sidecar | +| Tag | alpha | +| Region | ap-south-1 (Mumbai) |