helix-engage/docs/developer-operations-runbook.md

Helix Engage — Developer Operations Runbook

Architecture

Browser (India)
    ↓ HTTPS
Caddy (reverse proxy, TLS, static files)
    ├── engage.srv1477139.hstgr.cloud → /srv/engage (static frontend)
    ├── engage-api.srv1477139.hstgr.cloud → sidecar:4100
    └── *.srv1477139.hstgr.cloud → server:4000 (platform)
    
Docker Compose stack:
    ├── caddy          — Reverse proxy + TLS
    ├── server         — FortyTwo platform (ECR image)
    ├── worker         — Background jobs
    ├── sidecar        — Helix Engage NestJS API (ECR image)
    ├── db             — PostgreSQL 16
    ├── redis          — Session + cache
    ├── clickhouse     — Analytics
    ├── minio          — Object storage
    └── redpanda       — Event bus (Kafka)

VPS Access

# SSH into the VPS
sshpass -p 'SasiSuman@2007' ssh -o StrictHostKeyChecking=no root@148.230.67.184

# Or with SSH key (if configured)
ssh -i ~/Downloads/fortytwoai_hostinger root@148.230.67.184

Detail	Value
Host	148.230.67.184
User	root
Password	SasiSuman@2007
Docker compose dir	/opt/fortytwo
Frontend static files	/opt/fortytwo/helix-engage-frontend
Caddyfile	/opt/fortytwo/Caddyfile

URLs

Service	URL
Frontend	https://engage.srv1477139.hstgr.cloud
Sidecar API	https://engage-api.srv1477139.hstgr.cloud
Platform	https://fortytwo-dev.srv1477139.hstgr.cloud

Login Credentials

Role	Email	Password
CC Agent	rekha.cc@globalhospital.com	Global@123
CC Agent	ganesh.cc@globalhospital.com	Global@123
Marketing	sanjay.marketing@globalhospital.com	Global@123
Admin/Supervisor	dr.ramesh@globalhospital.com	Global@123

---

Local Testing

Always test locally before deploying to staging.

Frontend (Vite dev server)

cd helix-engage

# Start dev server (hot reload)
npm run dev
# → http://localhost:5173

# Type check (catches production build errors)
npx tsc --noEmit

# Production build (same as deploy)
npm run build

The .env.local controls which sidecar the frontend talks to:

# Remote sidecar (default — uses deployed backend)
VITE_API_URL=https://engage-api.srv1477139.hstgr.cloud
VITE_SIDECAR_URL=https://engage-api.srv1477139.hstgr.cloud

# Local sidecar (for testing sidecar changes)
# VITE_API_URL=http://localhost:4100
# VITE_SIDECAR_URL=http://localhost:4100

# Split — theme endpoint local, everything else remote
# VITE_THEME_API_URL=http://localhost:4100

Important: When VITE_API_URL points to localhost:4100, login and GraphQL only work if the local sidecar can reach the platform. The local sidecar's .env must have valid PLATFORM_GRAPHQL_URL and PLATFORM_API_KEY.

Sidecar (NestJS dev server)

cd helix-engage-server

# Start with watch mode (auto-restart on changes)
npm run start:dev
# → http://localhost:4100

# Build only (no run)
npm run build

# Production start
npm run start:prod

The sidecar .env must have:

PLATFORM_GRAPHQL_URL=...     # Platform GraphQL endpoint
PLATFORM_API_KEY=...         # Platform API key for server-to-server calls
PLATFORM_WORKSPACE_SUBDOMAIN=fortytwo-dev
REDIS_URL=redis://localhost:6379   # Local Redis required

Local Docker stack (full environment)

For testing with a local platform + database + Redis:

cd helix-engage-local

# First time — pull images + start
./deploy-local.sh up

# Deploy frontend to local stack
./deploy-local.sh frontend

# Deploy sidecar to local stack
./deploy-local.sh sidecar

# Both
./deploy-local.sh all

# Logs
./deploy-local.sh logs

# Stop
./deploy-local.sh down

Local stack URLs:

• Platform: http://localhost:5001
• Sidecar: http://localhost:5100
• Frontend: http://localhost:5080

Pre-deploy checklist

Before running deploy.sh:

1. npx tsc --noEmit — passes with no errors (frontend)
2. npm run build — succeeds (sidecar)
3. Test the changed feature locally (dev server or local stack)
4. Check package.json for new dependencies → decides quick vs full deploy

---

Deployment

Prerequisites (local machine)

# Required tools
brew install sshpass         # SSH with password
aws configure                # AWS CLI (for ECR)
docker desktop               # Docker with buildx

# Verify AWS access
aws sts get-caller-identity  # Should show account 043728036361

Path 1: Quick Deploy (no new dependencies)

Use when only code changes — no new npm packages.

cd /path/to/fortytwo-eap

# Deploy frontend only
bash deploy.sh frontend

# Deploy sidecar only
bash deploy.sh sidecar

# Deploy both
bash deploy.sh all

What it does:

• Frontend: npm run build → tar dist/ → SCP to VPS → extract to /opt/fortytwo/helix-engage-frontend
• Sidecar: nest build → tar dist/ + src/ → docker cp into running container → docker compose restart sidecar

Path 2: Full Deploy (new dependencies)

Use when package.json changed (new npm packages added).

cd /path/to/fortytwo-eap/helix-engage-server

# 1. Login to ECR
aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com

# 2. Build cross-platform image and push
docker buildx build --platform linux/amd64 \
  -t 043728036361.dkr.ecr.ap-south-1.amazonaws.com/fortytwo-eap/helix-engage-sidecar:alpha \
  --push .

# 3. Pull and restart on VPS
ECR_TOKEN=$(aws ecr get-login-password --region ap-south-1)
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "
  echo '$ECR_TOKEN' | docker login --username AWS --password-stdin 043728036361.dkr.ecr.ap-south-1.amazonaws.com
  cd /opt/fortytwo
  docker compose pull sidecar
  docker compose up -d sidecar
"

How to decide which path

Did package.json change?
  ├── YES → Path 2 (ECR build + push + pull)
  └── NO → Path 1 (deploy.sh)

---

Checking Logs

Sidecar logs

# SSH into VPS first, or run remotely:
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 --tail 30"

# Follow live
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 -f --tail 10"

# Filter for errors
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-sidecar-1 --tail 100 2>&1 | grep -i error"

# Via deploy.sh
bash deploy.sh logs

Caddy logs

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-caddy-1 --tail 30"

Platform server logs

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker logs fortytwo-staging-server-1 --tail 30"

All container status

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"

---

Health Checks

Sidecar healthy startup

Look for these lines in sidecar logs:

[NestApplication] Nest application successfully started
Helix Engage Server running on port 4100
[SessionService] Redis connected
[ThemeService] Theme loaded from file  (or "Using default theme")
[RulesStorageService] Initialized empty rules config

Common failure patterns

Log pattern	Meaning	Fix
Cannot find module 'xxx'	Missing npm dependency	Path 2 deploy (rebuild ECR image)
UndefinedModuleException	Circular dependency or missing import	Fix code, redeploy
ECONNREFUSED redis:6379	Redis not ready	docker compose restart redis sidecar
Forbidden resource	Platform permission issue	Check user roles
429 Too Many Requests	Ozonetel rate limit	Wait, reduce polling frequency

---

Redis Cache Operations

Clear caller resolution cache

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli KEYS 'caller:*'"

# Clear all caller cache
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'caller:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL"

Clear recording analysis cache

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'call:analysis:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL"

Clear agent name cache

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli --scan --pattern 'agent:name:*' | xargs -r docker exec -i fortytwo-staging-redis-1 redis-cli DEL"

Clear all session/cache keys

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-redis-1 redis-cli FLUSHDB"

---

Database Access

sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "docker exec fortytwo-staging-db-1 psql -U fortytwo -d fortytwo_staging"

Useful queries

-- List workspace schemas
SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'workspace_%';

-- List custom entities
SELECT "nameSingular", "isCustom" FROM core."objectMetadata" ORDER BY "nameSingular";

-- List users
SELECT u.email, u."firstName", u."lastName", uw.id as workspace_id
FROM core."user" u
JOIN core."userWorkspace" uw ON uw."userId" = u.id;

-- List roles
SELECT r.label, rt."userWorkspaceId"
FROM core."roleTarget" rt
JOIN core."role" r ON r.id = rt."roleId";

---

Rollback

Frontend rollback

The previous frontend build is overwritten. To rollback:

1. Checkout the previous git commit
2. npm run build
3. bash deploy.sh frontend

Sidecar rollback (quick deploy)

Same as frontend — checkout previous commit, rebuild, redeploy.

Sidecar rollback (ECR)

# Tag the current image as rollback
# Then re-tag the previous image as :alpha
# Or use a specific tag/digest

# On VPS:
sshpass -p 'SasiSuman@2007' ssh root@148.230.67.184 "
  cd /opt/fortytwo
  docker compose restart sidecar
"

---

Theme Management

View current theme

curl -s https://engage-api.srv1477139.hstgr.cloud/api/config/theme | python3 -m json.tool

Reset theme to defaults

curl -s -X POST https://engage-api.srv1477139.hstgr.cloud/api/config/theme/reset | python3 -m json.tool

Theme backups

Stored on the sidecar container at /app/data/theme-backups/. Each save creates a timestamped backup.

---

Git Repositories

Repo	Azure DevOps URL	Branch
Frontend	https://dev.azure.com/globalhealthx/EMR/_git/helix-engage	dev
Sidecar	https://dev.azure.com/globalhealthx/EMR/_git/helix-engage-server	dev
SDK App	FortyTwoApps/helix-engage/ (in fortytwo-eap monorepo)	dev

Commit and push pattern

# Frontend
cd helix-engage
git add -A && git commit -m "feat: description" && git push origin dev

# Sidecar
cd helix-engage-server
git add -A && git commit -m "feat: description" && git push origin dev

---

ECR Details

Detail	Value
Registry	043728036361.dkr.ecr.ap-south-1.amazonaws.com
Repository	fortytwo-eap/helix-engage-sidecar
Tag	alpha
Region	ap-south-1 (Mumbai)