Skip to content

Runbook — Incident alerting via GitHub Issues

End-to-end documentation for the auto-incident pipeline that turns external monitoring alerts into deduplicated GitHub issues, pushed to maintainers via the official GitHub mobile app.

Beta posture (ADR-039): during the restricted beta, error tracking runs on Sentry SaaS (free tier), not self-hosted GlitchTip. Sentry emits the same Sentry-compatible webhook this pipeline already parses, so the error_alert path is unchanged: read every "GlitchTip" step below as "Sentry" while the beta profile is active. GlitchTip stays the post-beta self-hosted target (ADR-031); uptime has the parallel UptimeRobot-only posture (uptime-monitoring.md).

Architecture

GlitchTip ─────────┐
Uptime Kuma ───────┤
deploy smoke-test ─┤
UptimeRobot ───────┴──► POST /repos/.../dispatches (Bearer INCIDENT_DISPATCH_TOKEN)
UptimeRobot ───────┘                 │
                                     ▼
              GitHub repository_dispatch event
                                     │
                                     ▼
                .github/workflows/incident-create.yml
                                     │
                  ┌──────────────────┴──────────────────┐
                  ▼                                     ▼
        existing open issue                  no existing issue
        with event-fp:<hash> label           with that label
                  │                                     │
                  ▼                                     ▼
        comment with new occurrence          create issue with labels
                                             [incident, severity-pN,
                                              event-fp:<hash>]
                                     │
                                     ▼
                       GitHub mobile push notification

Authentication & secrets

External services call the GitHub REST API POST /repos/vincentchalamon/bike-trip-planner/dispatches with a bearer token.

  • Secret name: INCIDENT_DISPATCH_TOKEN (already provisioned)
  • Type: Fine-grained PAT
  • Permissions: Contents: Read and write on this repository only (what POST /dispatches requires; Metadata: Read is added automatically)
  • Storage: GitHub Actions secret (read by the smoke-test job of deploy.yml, which dispatches an uptime_alert when the post-deploy probe fails) and the webhook configuration of each external monitor
  • Rotation: every 90 days

incident-create.yml itself authenticates with the built-in GITHUB_TOKEN (scope issues: write); it does not read INCIDENT_DISPATCH_TOKEN.

Rotating INCIDENT_DISPATCH_TOKEN

  1. GitHub → Settings → Developer settings → Fine-grained tokens → generate new token, same permissions, 90-day expiry.
  2. Update the secret in Settings → Secrets and variables → Actions.
  3. Update the bearer token in:
    • GlitchTip → project Settings → Alerts → webhook integration
    • Uptime Kuma → Settings → Notifications → Webhook
    • UptimeRobot → My Settings → Integrations & API → Webhook
  4. Trigger a test dispatch (see below) and confirm an issue is opened.
  5. Revoke the old token.

Expected payload schemas

error_alert (GlitchTip)

GlitchTip is Sentry-webhook-compatible; the parser is tolerant of missing fields and falls back to unknown.

{
  "event_type": "error_alert",
  "client_payload": {
    "issue": {
      "title": "RuntimeException: Redis connection refused",
      "culprit": "App\\MessageHandler\\ComputeStageHandler::__invoke",
      "level": "error",
      "count": 12,
      "web_url": "https://errors.bike-trip-planner.com/issues/4242",
      "tags": { "request_id": "0192c0d8-7e3a-7000-9f3a-4f6d5b2c8a91" }
    },
    "environment": "production",
    "release": "abc1234"
  }
}

Fingerprint: sha256("<culprit>::<title>")[:12].

uptime_alert (Uptime Kuma / UptimeRobot)

{
  "event_type": "uptime_alert",
  "client_payload": {
    "monitor_name": "biketrip-healthz",
    "monitor_url": "https://www.bike-trip-planner.com/api/healthz",
    "status": "down",
    "message": "Connection timed out after 10s",
    "heartbeat": { "ping": 10000 }
  }
}

UptimeRobot field aliases (monitorFriendlyName, alertType, alertDetails, responseTime) are accepted as fallbacks.

Fingerprint: <monitor_name>-<status> lowercased, non-alphanumerics replaced with -, truncated to 41 chars (e.g. biketrip-healthz-down).

Manual test

Replace <TOKEN> with INCIDENT_DISPATCH_TOKEN (local copy, never commit).

# Error alert
curl -X POST \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/vincentchalamon/bike-trip-planner/dispatches \
  -d '{
    "event_type": "error_alert",
    "client_payload": {
      "issue": {
        "title": "Manual test error",
        "culprit": "App\\Tests\\ManualDispatch",
        "level": "error",
        "count": 1
      },
      "environment": "staging"
    }
  }'

# Uptime alert
curl -X POST \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/vincentchalamon/bike-trip-planner/dispatches \
  -d '{
    "event_type": "uptime_alert",
    "client_payload": {
      "monitor_name": "manual-test",
      "monitor_url": "https://www.bike-trip-planner.com/api/healthz",
      "status": "down",
      "message": "Manual dispatch test"
    }
  }'

A new incident issue should appear within ~30 seconds. Re-running the same command must comment on the existing issue (no duplicate).

Webhook configuration per service

GlitchTip

  1. Open the GlitchTip project → Settings → Alerts.
  2. Create alert → trigger condition (e.g. "An event is seen", frequency 1 min, environment production).
  3. Action type Webhook.
  4. URL: https://api.github.com/repos/vincentchalamon/bike-trip-planner/dispatches.
  5. Method POST, headers:
    • Accept: application/vnd.github+json
    • Authorization: Bearer <INCIDENT_DISPATCH_TOKEN>
    • Content-Type: application/json

  6. Body template (GlitchTip supports raw JSON; adjust placeholders to project syntax):

    {
  "event_type": "error_alert",
  "client_payload": {
    "issue": {
      "title": "{{ issue.title }}",
      "culprit": "{{ issue.culprit }}",
      "level": "{{ issue.level }}",
      "count": {{ issue.count }},
      "web_url": "{{ issue.web_url }}"
    },
    "environment": "{{ environment }}",
    "release": "{{ release }}"
  }
}

Uptime Kuma

  1. Uptime Kuma UI → Settings → Notifications → Setup Notification.
  2. Type Webhook.
  3. POST URL: https://api.github.com/repos/vincentchalamon/bike-trip-planner/dispatches.

  4. Request body: Custom Body with:

    {
  "event_type": "uptime_alert",
  "client_payload": {
    "monitor_name": "{{ monitorJSON.name }}",
    "monitor_url": "{{ monitorJSON.url }}",
    "status": "{{ status }}",
    "message": "{{ msg }}",
    "heartbeat": { "ping": {{ heartbeatJSON.ping }} }
  }
}

  5. Additional headers: Authorization: Bearer <INCIDENT_DISPATCH_TOKEN>, Accept: application/vnd.github+json.

  6. Attach the notification to every monitor (default behaviour when "Apply on all existing monitors" is checked).

UptimeRobot

  1. My Settings → Integrations & API → Add Integration.
  2. Integration type Web-Hook.
  3. URL to notify: https://api.github.com/repos/vincentchalamon/bike-trip-planner/dispatches.

  4. POST value (JSON):

    {
  "event_type": "uptime_alert",
  "client_payload": {
    "monitor_name": "*monitorFriendlyName*",
    "monitor_url": "*monitorURL*",
    "status": "*alertTypeFriendlyName*",
    "message": "*alertDetails*",
    "responseTime": *responseTime*
  }
}

  5. Send as JSON: Yes. Custom HTTP headers:

    • Authorization: Bearer <INCIDENT_DISPATCH_TOKEN>
    • Accept: application/vnd.github+json
  6. Attach the integration to the /api/healthz external monitor.

Troubleshooting

  • No issue appears: check the Actions tab for a failed Incident — auto-create issue run. The Parse payload, dedupe, create or comment step logs the received event_type; the created issue embeds the raw payload (truncated to 4000 characters). If no run exists at all, the dispatch itself was rejected (see the HTTP codes below).
  • Duplicate issues for the same incident: the fingerprint changed. Confirm the upstream service is sending stable culprit / monitor_name values.
  • HTTP 401 from dispatches: the PAT expired. See "Rotating INCIDENT_DISPATCH_TOKEN".
  • HTTP 422 from dispatches: event_type is not in the workflow's types: list (only error_alert and uptime_alert are accepted).