mirror of
https://dev.azure.com/globalhealthx/EMR/_git/helix-engage
synced 2026-04-11 18:28:15 +00:00
docs: add CI/CD operations guide
Covers Gitea + Woodpecker + MinIO pipeline setup, Teams notifications, test report publishing, mirrored repos, secrets config, troubleshooting, and E2E test coverage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
181
docs/ci-cd-operations.md
Normal file
181
docs/ci-cd-operations.md
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
# Helix Engage — CI/CD & Operations Dashboard
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Three services on EC2 provide CI/CD and operational visibility:
|
||||||
|
|
||||||
|
- **Gitea** (`git.healix360.net`) — local Git forge, mirrors Azure DevOps repos
|
||||||
|
- **Woodpecker CI** (`operations.healix360.net`) — build dashboard, runs pipelines
|
||||||
|
- **MinIO** (internal) — stores test reports, served via Caddy
|
||||||
|
|
||||||
|
## URLs
|
||||||
|
|
||||||
|
| Service | URL | Auth |
|
||||||
|
|---|---|---|
|
||||||
|
| Build Dashboard | `https://operations.healix360.net` | Gitea OAuth (helix-admin / Global@2026) |
|
||||||
|
| Test Reports | `https://operations.healix360.net/reports/{run}/index.html` | Basic auth (helix-admin / Global@2026) |
|
||||||
|
| Git Forge | `https://git.healix360.net` | helix-admin / Global@2026 |
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
```
|
||||||
|
Azure DevOps (push)
|
||||||
|
↓ mirror sync (every 15min or manual)
|
||||||
|
Gitea (git.healix360.net)
|
||||||
|
↓ webhook
|
||||||
|
Woodpecker CI (operations.healix360.net)
|
||||||
|
↓ runs pipeline steps in Docker containers
|
||||||
|
├── typecheck (node:20, yarn tsc)
|
||||||
|
├── e2e-tests (playwright, 40 smoke tests)
|
||||||
|
├── publish-report (S3 plugin → MinIO)
|
||||||
|
└── notify-teams (curl → Power Automate → Teams channel)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pipelines
|
||||||
|
|
||||||
|
### helix-engage (frontend)
|
||||||
|
|
||||||
|
Triggers on push to any branch or manual run.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. **typecheck** — `yarn install` + `tsc --noEmit` (node:20 image)
|
||||||
|
2. **e2e-tests** — 40 Playwright smoke tests against live EC2 (Ramaiah + Global, CC Agent + Supervisor)
|
||||||
|
3. **publish-report** — uploads Playwright HTML report to MinIO via S3 plugin
|
||||||
|
4. **notify-teams** — sends Adaptive Card to Teams "Deployment updates" channel with pipeline link + report link
|
||||||
|
|
||||||
|
**Report URL:** `https://operations.healix360.net/reports/{pipeline-number}/index.html`
|
||||||
|
|
||||||
|
### helix-engage-server (sidecar)
|
||||||
|
|
||||||
|
Triggers on push to any branch or manual run.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. **unit-tests** — `npm ci` + `jest --ci --forceExit` (node:20 image)
|
||||||
|
2. **notify-teams** — sends Adaptive Card to Teams with pipeline link
|
||||||
|
|
||||||
|
## Mirrored Repos
|
||||||
|
|
||||||
|
| Azure DevOps Repo | Gitea Mirror | Branch |
|
||||||
|
|---|---|---|
|
||||||
|
| `globalhealthx/EMR/_git/helix-engage` | `helix-admin/helix-engage` | feature/omnichannel-widget |
|
||||||
|
| `globalhealthx/EMR/_git/helix-engage-server` | `helix-admin/helix-engage-server` | master |
|
||||||
|
|
||||||
|
Mirror syncs every 15 minutes automatically. To force sync:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage/mirror-sync" \
|
||||||
|
-u "helix-admin:Global@2026"
|
||||||
|
|
||||||
|
curl -s -X POST "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage-server/mirror-sync" \
|
||||||
|
-u "helix-admin:Global@2026"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Teams Notifications
|
||||||
|
|
||||||
|
Notifications go to the "Deployment updates" channel via Power Automate Workflow webhook.
|
||||||
|
|
||||||
|
Each notification includes:
|
||||||
|
- Project name and build number
|
||||||
|
- Branch name
|
||||||
|
- Commit message
|
||||||
|
- "View Pipeline" button (links to Woodpecker)
|
||||||
|
- "View Report" button (links to Playwright HTML report, frontend only)
|
||||||
|
|
||||||
|
## Secrets (Woodpecker)
|
||||||
|
|
||||||
|
Configured per-repo in Woodpecker Settings → Secrets:
|
||||||
|
|
||||||
|
| Secret | Used by | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `s3_access_key` | publish-report | MinIO access key (`minio`) |
|
||||||
|
| `s3_secret_key` | publish-report | MinIO secret key |
|
||||||
|
| `teams_webhook` | notify-teams | Power Automate webhook URL |
|
||||||
|
|
||||||
|
## Docker Containers
|
||||||
|
|
||||||
|
| Container | Image | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `ramaiah-prod-gitea-1` | `gitea/gitea:latest` | Git forge |
|
||||||
|
| `ramaiah-prod-woodpecker-server-1` | `woodpeckerci/woodpecker-server:v3` | CI dashboard + pipeline engine |
|
||||||
|
| `ramaiah-prod-woodpecker-agent-1` | `woodpeckerci/woodpecker-agent:v3` | Executes pipeline steps in Docker |
|
||||||
|
|
||||||
|
## Agent Configuration
|
||||||
|
|
||||||
|
The Woodpecker agent is configured to:
|
||||||
|
- Run pipeline containers on the `ramaiah-prod_default` Docker network (so they can reach Gitea and MinIO)
|
||||||
|
- Allow up to 2 concurrent workflows
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Pipeline fails at git clone
|
||||||
|
|
||||||
|
Check that Gitea's `REQUIRE_SIGNIN_VIEW` is `false` (public repos must be cloneable without auth):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||||
|
"docker exec ramaiah-prod-gitea-1 grep REQUIRE_SIGNIN /data/gitea/conf/app.ini"
|
||||||
|
```
|
||||||
|
|
||||||
|
### npm install crashes with "Exit handler never called"
|
||||||
|
|
||||||
|
Known npm bug in CI containers. Use `yarn` instead of `npm` for the frontend. The sidecar's lockfile is clean so `npm ci` works.
|
||||||
|
|
||||||
|
### Pipeline says "pipeline definition not found"
|
||||||
|
|
||||||
|
The `.woodpecker.yml` file is missing or has invalid YAML. Check:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://git.healix360.net/api/v1/repos/helix-admin/helix-engage/contents/.woodpecker.yml?ref=feature/omnichannel-widget" \
|
||||||
|
-u "helix-admin:Global@2026" | python3 -c "import sys,json;print(json.load(sys.stdin).get('name','NOT FOUND'))"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Teams notification not arriving
|
||||||
|
|
||||||
|
Verify the webhook secret is set in Woodpecker and the Power Automate workflow is active.
|
||||||
|
|
||||||
|
### Test reports not loading (403/XML error)
|
||||||
|
|
||||||
|
Caddy must strip the Authorization header before proxying to MinIO. Check:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -i /tmp/ramaiah-ec2-key ubuntu@13.234.31.194 \
|
||||||
|
"grep -A8 'handle_path /reports' /opt/fortytwo/Caddyfile"
|
||||||
|
```
|
||||||
|
|
||||||
|
Should include `header_up -Authorization`.
|
||||||
|
|
||||||
|
### Manually trigger a pipeline
|
||||||
|
|
||||||
|
```bash
|
||||||
|
WP_TOKEN="<woodpecker-api-token>"
|
||||||
|
curl -s -X POST "https://operations.healix360.net/api/repos/1/pipelines" \
|
||||||
|
-H "Authorization: Bearer $WP_TOKEN" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"branch":"feature/omnichannel-widget"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete old pipeline runs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
WP_TOKEN="<woodpecker-api-token>"
|
||||||
|
for i in $(seq 1 20); do
|
||||||
|
curl -s -X DELETE "https://operations.healix360.net/api/repos/1/pipelines/$i" \
|
||||||
|
-H "Authorization: Bearer $WP_TOKEN"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## E2E Test Coverage
|
||||||
|
|
||||||
|
40 tests across 2 hospitals, 3 roles:
|
||||||
|
|
||||||
|
**Login (4):** branding, invalid creds, supervisor login, auth guard
|
||||||
|
|
||||||
|
**Ramaiah CC Agent (10):** landing, call desk, call history, patients (list + search), appointments, my performance (API + KPI), sidebar, sign-out modal, sign-out complete
|
||||||
|
|
||||||
|
**Ramaiah Supervisor (12):** landing, team performance, live monitor, leads, patients, appointments, call log, recordings, missed calls, campaigns, settings, sidebar
|
||||||
|
|
||||||
|
**Global CC Agent (7):** landing, call history, patients, appointments, my performance, sidebar, sign-out
|
||||||
|
|
||||||
|
**Global Supervisor (5):** landing, patients, appointments, campaigns, settings
|
||||||
|
|
||||||
|
**Auto-cleanup:** Last CC Agent test completes sign-out to release agent session. Setup steps call `/api/maint/unlock-agent` to clear stale locks.
|
||||||
Reference in New Issue
Block a user