From 8955062b6df0295b7cf61e1113e8aa08f336093d Mon Sep 17 00:00:00 2001 From: saridsa2 Date: Sat, 11 Apr 2026 15:41:26 +0530 Subject: [PATCH] docs: add CI/CD operations guide Covers Gitea + Woodpecker + MinIO pipeline setup, Teams notifications, test report publishing, mirrored repos, secrets config, troubleshooting, and E2E test coverage. Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/ci-cd-operations.md | 181 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 docs/ci-cd-operations.md diff --git a/docs/ci-cd-operations.md b/docs/ci-cd-operations.md new file mode 100644 index 0000000..5620cca --- /dev/null +++ b/docs/ci-cd-operations.md @@ -0,0 +1,181 @@ +# Helix Engage — CI/CD & Operations Dashboard + +## Overview + +Three services on EC2 provide CI/CD and operational visibility: + +- **Gitea** (`git.healix360.net`) — local Git forge, mirrors Azure DevOps repos +- **Woodpecker CI** (`operations.healix360.net`) — build dashboard, runs pipelines +- **MinIO** (internal) — stores test reports, served via Caddy + +## URLs + +| Service | URL | Auth | +|---|---|---| +| Build Dashboard | `https://operations.healix360.net` | Gitea OAuth (helix-admin / Global@2026) | +| Test Reports | `https://operations.healix360.net/reports/{run}/index.html` | Basic auth (helix-admin / Global@2026) | +| Git Forge | `https://git.healix360.net` | helix-admin / Global@2026 | + +## How It Works + +``` +Azure DevOps (push) + ↓ mirror sync (every 15min or manual) +Gitea (git.healix360.net) + ↓ webhook +Woodpecker CI (operations.healix360.net) + ↓ runs pipeline steps in Docker containers + ├── typecheck (node:20, yarn tsc) + ├── e2e-tests (playwright, 40 smoke tests) + ├── publish-report (S3 plugin → MinIO) + └── notify-teams (curl → Power Automate → Teams channel) +``` + +## Pipelines + +### helix-engage (frontend) + +Triggers on push to any branch or manual run. + +**Steps:** +1. **typecheck** — `yarn install` + `tsc --noEmit` (node:20 image) +2. **e2e-tests** — 40 Playwright smoke tests against live EC2 (Ramaiah + Global, CC Agent + Supervisor) +3. **publish-report** — uploads Playwright HTML report to MinIO via S3 plugin +4. **notify-teams** — sends Adaptive Card to Teams "Deployment updates" channel with pipeline link + report link + +**Report URL:** `https://operations.healix360.net/reports/{pipeline-number}/index.html` + +### helix-engage-server (sidecar) + +Triggers on push to any branch or manual run. + +**Steps:** +1. **unit-tests** — `npm ci` + `jest --ci --forceExit` (node:20 image) +2. **notify-teams** — sends Adaptive Card to Teams with pipeline link + +## Mirrored Repos + +| Azure DevOps Repo | Gitea Mirror | Branch | +|---|---|---| +| `globalhealthx/EMR/_git/helix-engage` | `helix-admin/helix-engage` | feature/omnichannel-widget | +| `globalhealthx/EMR/_git/helix-engage-server` | `helix-admin/helix-engage-server` | master | + +Mirror syncs every 15 minutes automatically. To force sync: + +```bash +curl -s -X POST "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage/mirror-sync" \ + -u "helix-admin:Global@2026" + +curl -s -X POST "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage-server/mirror-sync" \ + -u "helix-admin:Global@2026" +``` + +## Teams Notifications + +Notifications go to the "Deployment updates" channel via Power Automate Workflow webhook. + +Each notification includes: +- Project name and build number +- Branch name +- Commit message +- "View Pipeline" button (links to Woodpecker) +- "View Report" button (links to Playwright HTML report, frontend only) + +## Secrets (Woodpecker) + +Configured per-repo in Woodpecker Settings → Secrets: + +| Secret | Used by | Purpose | +|---|---|---| +| `s3_access_key` | publish-report | MinIO access key (`minio`) | +| `s3_secret_key` | publish-report | MinIO secret key | +| `teams_webhook` | notify-teams | Power Automate webhook URL | + +## Docker Containers + +| Container | Image | Purpose | +|---|---|---| +| `ramaiah-prod-gitea-1` | `gitea/gitea:latest` | Git forge | +| `ramaiah-prod-woodpecker-server-1` | `woodpeckerci/woodpecker-server:v3` | CI dashboard + pipeline engine | +| `ramaiah-prod-woodpecker-agent-1` | `woodpeckerci/woodpecker-agent:v3` | Executes pipeline steps in Docker | + +## Agent Configuration + +The Woodpecker agent is configured to: +- Run pipeline containers on the `ramaiah-prod_default` Docker network (so they can reach Gitea and MinIO) +- Allow up to 2 concurrent workflows + +## Troubleshooting + +### Pipeline fails at git clone + +Check that Gitea's `REQUIRE_SIGNIN_VIEW` is `false` (public repos must be cloneable without auth): + +```bash +ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ + "docker exec ramaiah-prod-gitea-1 grep REQUIRE_SIGNIN /data/gitea/conf/app.ini" +``` + +### npm install crashes with "Exit handler never called" + +Known npm bug in CI containers. Use `yarn` instead of `npm` for the frontend. The sidecar's lockfile is clean so `npm ci` works. + +### Pipeline says "pipeline definition not found" + +The `.woodpecker.yml` file is missing or has invalid YAML. Check: + +```bash +curl -s "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage/contents/.woodpecker.yml?ref=feature/omnichannel-widget" \ + -u "helix-admin:Global@2026" | python3 -c "import sys,json;print(json.load(sys.stdin).get('name','NOT FOUND'))" +``` + +### Teams notification not arriving + +Verify the webhook secret is set in Woodpecker and the Power Automate workflow is active. + +### Test reports not loading (403/XML error) + +Caddy must strip the Authorization header before proxying to MinIO. Check: + +```bash +ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \ + "grep -A8 'handle_path /reports' /opt/fortytwo/Caddyfile" +``` + +Should include `header_up -Authorization`. + +### Manually trigger a pipeline + +```bash +WP_TOKEN="" +curl -s -X POST "https://operations.healix360.net/api/repos/1/pipelines" \ + -H "Authorization: Bearer $WP_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"branch":"feature/omnichannel-widget"}' +``` + +### Delete old pipeline runs + +```bash +WP_TOKEN="" +for i in $(seq 1 20); do + curl -s -X DELETE "https://operations.healix360.net/api/repos/1/pipelines/$i" \ + -H "Authorization: Bearer $WP_TOKEN" +done +``` + +## E2E Test Coverage + +40 tests across 2 hospitals, 3 roles: + +**Login (4):** branding, invalid creds, supervisor login, auth guard + +**Ramaiah CC Agent (10):** landing, call desk, call history, patients (list + search), appointments, my performance (API + KPI), sidebar, sign-out modal, sign-out complete + +**Ramaiah Supervisor (12):** landing, team performance, live monitor, leads, patients, appointments, call log, recordings, missed calls, campaigns, settings, sidebar + +**Global CC Agent (7):** landing, call history, patients, appointments, my performance, sidebar, sign-out + +**Global Supervisor (5):** landing, patients, appointments, campaigns, settings + +**Auto-cleanup:** Last CC Agent test completes sign-out to release agent session. Setup steps call `/api/maint/unlock-agent` to clear stale locks.