helix-engage/docs/ci-cd-operations.md

Helix Engage — CI/CD & Operations Dashboard

Overview

Three services on EC2 provide CI/CD and operational visibility:

• Gitea (git.healix360.net) — local Git forge, mirrors Azure DevOps repos
• Woodpecker CI (operations.healix360.net) — build dashboard, runs pipelines
• MinIO (internal) — stores test reports, served via Caddy

URLs

Service	URL	Auth
Build Dashboard	https://operations.healix360.net	Gitea OAuth (helix-admin / Global@2026)
Test Reports	https://operations.healix360.net/reports/{run}/index.html	Basic auth (helix-admin / Global@2026)
Git Forge	https://git.healix360.net	helix-admin / Global@2026

How It Works

Azure DevOps (push)
    ↓ mirror sync (every 15min or manual)
Gitea (git.healix360.net)
    ↓ webhook
Woodpecker CI (operations.healix360.net)
    ↓ runs pipeline steps in Docker containers
    ├── typecheck (node:20, yarn tsc)
    ├── e2e-tests (playwright, 40 smoke tests)
    ├── publish-report (S3 plugin → MinIO)
    └── notify-teams (curl → Power Automate → Teams channel)

Pipelines

helix-engage (frontend)

Triggers on push to any branch or manual run.

Steps:

1. typecheck — yarn install + tsc --noEmit (node:20 image)
2. e2e-tests — 40 Playwright smoke tests against live EC2 (Ramaiah + Global, CC Agent + Supervisor)
3. publish-report — uploads Playwright HTML report to MinIO via S3 plugin
4. notify-teams — sends Adaptive Card to Teams "Deployment updates" channel with pipeline link + report link

Report URL: https://operations.healix360.net/reports/{pipeline-number}/index.html

helix-engage-server (sidecar)

Triggers on push to any branch or manual run.

Steps:

1. unit-tests — npm ci + jest --ci --forceExit (node:20 image)
2. notify-teams — sends Adaptive Card to Teams with pipeline link

Mirrored Repos

Azure DevOps Repo	Gitea Mirror	Branch
globalhealthx/EMR/_git/helix-engage	helix-admin/helix-engage	feature/omnichannel-widget
globalhealthx/EMR/_git/helix-engage-server	helix-admin/helix-engage-server	master

Mirror syncs every 15 minutes automatically. To force sync:

curl -s -X POST "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage/mirror-sync" \
  -u "helix-admin:Global@2026"

curl -s -X POST "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage-server/mirror-sync" \
  -u "helix-admin:Global@2026"

Teams Notifications

Notifications go to the "Deployment updates" channel via Power Automate Workflow webhook.

Each notification includes:

• Project name and build number
• Branch name
• Commit message
• "View Pipeline" button (links to Woodpecker)
• "View Report" button (links to Playwright HTML report, frontend only)

Secrets (Woodpecker)

Configured per-repo in Woodpecker Settings → Secrets:

Secret	Used by	Purpose
s3_access_key	publish-report	MinIO access key (minio)
s3_secret_key	publish-report	MinIO secret key
teams_webhook	notify-teams	Power Automate webhook URL

Docker Containers

Container	Image	Purpose
ramaiah-prod-gitea-1	gitea/gitea:latest	Git forge
ramaiah-prod-woodpecker-server-1	woodpeckerci/woodpecker-server:v3	CI dashboard + pipeline engine
ramaiah-prod-woodpecker-agent-1	woodpeckerci/woodpecker-agent:v3	Executes pipeline steps in Docker

Agent Configuration

The Woodpecker agent is configured to:

• Run pipeline containers on the ramaiah-prod_default Docker network (so they can reach Gitea and MinIO)
• Allow up to 2 concurrent workflows

Troubleshooting

Pipeline fails at git clone

Check that Gitea's REQUIRE_SIGNIN_VIEW is false (public repos must be cloneable without auth):

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "docker exec ramaiah-prod-gitea-1 grep REQUIRE_SIGNIN /data/gitea/conf/app.ini"

npm install crashes with "Exit handler never called"

Known npm bug in CI containers. Use yarn instead of npm for the frontend. The sidecar's lockfile is clean so npm ci works.

Pipeline says "pipeline definition not found"

The .woodpecker.yml file is missing or has invalid YAML. Check:

curl -s "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage/contents/.woodpecker.yml?ref=feature/omnichannel-widget" \
  -u "helix-admin:Global@2026" | python3 -c "import sys,json;print(json.load(sys.stdin).get('name','NOT FOUND'))"

Teams notification not arriving

Verify the webhook secret is set in Woodpecker and the Power Automate workflow is active.

Test reports not loading (403/XML error)

Caddy must strip the Authorization header before proxying to MinIO. Check:

ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
  "grep -A8 'handle_path /reports' /opt/fortytwo/Caddyfile"

Should include header_up -Authorization.

Manually trigger a pipeline

WP_TOKEN="<woodpecker-api-token>"
curl -s -X POST "https://operations.healix360.net/api/repos/1/pipelines" \
  -H "Authorization: Bearer $WP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"branch":"feature/omnichannel-widget"}'

Delete old pipeline runs

WP_TOKEN="<woodpecker-api-token>"
for i in $(seq 1 20); do
  curl -s -X DELETE "https://operations.healix360.net/api/repos/1/pipelines/$i" \
    -H "Authorization: Bearer $WP_TOKEN"
done

E2E Test Coverage

40 tests across 2 hospitals, 3 roles:

Login (4): branding, invalid creds, supervisor login, auth guard

Ramaiah CC Agent (10): landing, call desk, call history, patients (list + search), appointments, my performance (API + KPI), sidebar, sign-out modal, sign-out complete

Ramaiah Supervisor (12): landing, team performance, live monitor, leads, patients, appointments, call log, recordings, missed calls, campaigns, settings, sidebar

Global CC Agent (7): landing, call history, patients, appointments, my performance, sidebar, sign-out

Global Supervisor (5): landing, patients, appointments, campaigns, settings

Auto-cleanup: Last CC Agent test completes sign-out to release agent session. Setup steps call /api/maint/unlock-agent to clear stale locks.