Incident Response Playbook

Step-by-step guide for handling production incidents across the EyvaX platform. Designed for both human operators and automated L1 DevOps agents.

---|-----|---------| | PagerDuty | bluesemi.pagerduty.com | Incident management | | Sentry | sentry.io/bluesemi-qv | Error tracking | | ArgoCD | argocd.eyva.io | Deployment management | | GCP Console | console.cloud.google.com | GKE cluster management |

Severity Levels

Alert Routing Table

When an alert fires, use this table to determine which runbook section to consult and whether auto-resolution is possible.

Cross-Service Dependency Map

Service Architecture

Mobile Apps / Web
      │
      ▼
┌─────────────────────┐
│  eyvax-backend      │ ── MongoDB (Atlas)
│  (eyvax-backend)    │ ── Redis (sidecar)
│  Port: 8000         │ ── ClickHouse
└────────┬────────────┘
         │ calls for AI insights
         ▼
┌─────────────────────┐
│  eyvax-ai           │ ── MongoDB
│  (eyvax-ai)         │ ── Redis (sidecar)
│  Port: 8045         │
└────────┬────────────┘
         │ called by data-pipeline for vital sign computation
         │
┌────────┴────────────┐      ┌─────────────────────┐
│  eyvax-data-pipeline│      │  eyvax-emd-glucose   │
│  (eyvax-data)       │─────▶│  (eyvax-glucose)     │
│  Ports: 8000, 8080  │      │  Port: 8000          │
└─────────────────────┘      └──────────────────────┘
         │                            │
         ▼                            ▼
    Kafka (RedPanda)            Redis (standalone)
    ClickHouse                  Eyva API, Juno API

Cascade Triage Order

When multiple services alert simultaneously, investigate in dependency order (upstream first):

1. eyvax-ai — shared dependency for data-pipeline AND backend-server. If eyvax-ai is down, both downstream services will show errors.
2. emd-glucose — called by data-pipeline for glucose metric computation. If eyvax-ai is healthy but emd-glucose is down, data-pipeline glucose metrics will fail.
3. data-pipeline — if upstream services are healthy, investigate Kafka/ClickHouse connectivity.
4. backend-server — usually a downstream victim, not root cause. If all other services are healthy, investigate MongoDB/auth.

Decision Tree: Multiple Services Alerting

IF eyvax-ai alerting AND (data-pipeline OR backend-server) also alerting:
  → Root cause is almost certainly eyvax-ai
  → Fix eyvax-ai FIRST — see ai-service.md
  → Other alerts will auto-resolve once eyvax-ai recovers
  → Wait 5 min after eyvax-ai fix, then check if other alerts clear

IF data-pipeline alerting AND emd-glucose alerting BUT eyvax-ai healthy:
  → Root cause is likely emd-glucose (data-pipeline calls it)
  → Fix emd-glucose FIRST — see emd-glucose.md
  → Data-pipeline glucose metric errors will clear after

IF data-pipeline alerting BUT eyvax-ai AND emd-glucose healthy:
  → Isolated data-pipeline issue
  → Check Kafka connectivity and consumer health
  → Check ClickHouse connectivity
  → See data-pipeline.md

IF backend-server alerting BUT all other services healthy:
  → Isolated backend issue
  → Check MongoDB connectivity (Atlas status)
  → Check for recent deployment (rollback candidate?)
  → Check rate limiting (traffic spike?)
  → See backend-server.md

IF all services alerting simultaneously:
  → Likely infrastructure issue (node failure, network partition)
  → Check: kubectl get nodes
  → Check: kubectl get pods --all-namespaces | grep -v Running
  → Escalate P1

Auto-Resolution Safety Boundaries

Safe to Auto-Execute (No Human Approval Needed)

Requires Human Approval

• Rollback to previous image version (may reintroduce bugs)
• Increase resource limits (CPU/memory) in deployment manifest
• Reset Kafka consumer group offsets (may cause data reprocessing)
• Delete PVCs or persistent data
• Force-push to git or merge PRs
• Any direct action on the database (MongoDB, ClickHouse)
• Change environment variables or secrets

Escalation Triggers (Stop and Page P2)

• Same alert fires 3+ times in 1 hour after auto-remediation
• Service remains unhealthy for > 15 minutes after remediation
• Multiple services alerting with no clear root cause after dependency check
• Data loss or corruption suspected (ClickHouse insert failures > 1%)
• All pods in CrashLoopBackOff with restart count > 5
• Error budget burn rate > 10x normal

When You Get Paged

1. Acknowledge (< 5 min)

# Acknowledge via PagerDuty API
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{
    "incidents": [{"id": "<INCIDENT_ID>", "type": "incident_reference", "status": "acknowledged"}]
  }'

2. Assess (< 10 min)

Check all service health endpoints:

# Backend Server
curl -s https://api.eyvax.com/health | python3 -m json.tool
# Expected 200: {"status": "healthy", "checks": {"database": {"status": "healthy"}, "database_write": {"status": "healthy"}}}
# Unhealthy 503: {"status": "unhealthy", "checks": {"database": {"status": "unhealthy", "type": "mongodb"}}}

# AI Service
curl -s https://eyvax-ai.eyva.io/health | python3 -m json.tool
# Expected 200: {"status": "healthy", "dependencies": {...}}

# EMD Glucose
curl -s https://emd-glucose.eyva.io/health | python3 -m json.tool
# Expected 200: {"status": "healthy", "checks": {"configuration": {"healthy": true}, "external_api": {"healthy": true}, "caching": {"healthy": true}}}
# Unhealthy 503: checks will show which dependency is failing

# Data Pipeline (internal only — not exposed via ingress)
# Check via kubectl port-forward or pod exec
kubectl exec deploy/eyvax-sensor-raw-data-processor -n eyvax-data -- curl -s http://localhost:8080/health | python3 -m json.tool
# Expected 200: {"status": "healthy", "checks": [{"name": "kafka", "status": "healthy"}, {"name": "clickhouse", "status": "healthy"}]}
# Degraded 503: {"status": "degraded"|"unhealthy", "checks": [...]} — check which check failed

Check pods across all namespaces:

# Set cluster context
gcloud container clusters get-credentials eyvax-dev-zone --region asia-south1 --project eyva-440707

# Quick status across all services
for ns in eyvax-backend eyvax-ai eyvax-glucose eyvax-data; do
  echo "=== $ns ==="
  kubectl get pods -n $ns -o wide 2>/dev/null
done

# Expected healthy output per pod:
# NAME                                    READY   STATUS    RESTARTS   AGE
# eyvax-backend-server-xxx-yyy            2/2     Running   0          2d
#
# Unhealthy indicators:
# STATUS = OOMKilled → memory limit exceeded, needs restart
# STATUS = CrashLoopBackOff → application crashing, check logs
# STATUS = ImagePullBackOff → image not found in registry, check CI
# READY = 0/2 → container not passing readiness probe
# RESTARTS > 3 → pod is crash-looping, check describe and logs

Check recent deployments (was something just deployed?):

# ArgoCD sync status for all apps
for app in eyvax-backend-server eyvax-ai eyvax-emd-glucose eyvax-data-pipeline; do
  STATUS=$(kubectl get application $app -n argocd -o jsonpath='{.status.sync.status} {.status.health.status}' 2>/dev/null)
  echo "$app: $STATUS"
done

# Expected: "Synced Healthy" for each app
# "OutOfSync" → ArgoCD hasn't applied latest manifest changes
# "Degraded" → pods not healthy after sync
# "Progressing" → rollout in progress

# Check if a deployment happened in the last hour
for ns in eyvax-backend eyvax-ai eyvax-glucose eyvax-data; do
  echo "=== $ns ==="
  kubectl rollout history deployment -n $ns 2>/dev/null | tail -5
done

Check Sentry for errors across all projects:

# Sweep all Sentry projects for unresolved issues
for project in eyvax-backend-server eyvax-ai eyvax-emd-glucose eyvax-data-pipeline; do
  echo "=== $project ==="
  curl -s -H "Authorization: Bearer $SENTRY_API_KEY" \
    "https://sentry.io/api/0/projects/bluesemi-qv/$project/issues/?query=is:unresolved&sort=date&statsPeriod=1h&limit=5" | \
    python3 -c "
import json, sys
try:
  issues = json.load(sys.stdin)
  if isinstance(issues, list):
    if not issues:
      print('  No unresolved issues')
    else:
      for i in issues:
        print(f'  [{i[\"level\"]}] {i[\"count\"]}x | {i[\"title\"][:80]} | Last: {i[\"lastSeen\"][:19]}')
  else:
    print(f'  API error or project not found')
except Exception as e:
  print(f'  Parse error: {e}')
"
done

# NOTE: emd-glucose events may appear in eyvax-backend-server project
# due to shared Sentry DSN. Check server_name tag to identify source pod.

Check PagerDuty for open incidents:

curl -s -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=25" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
incidents = data.get('incidents', [])
if not incidents:
  print('No open incidents')
else:
  for i in incidents:
    print(f'  [{i[\"status\"]}] {i[\"title\"][:80]} | ID: {i[\"id\"]} | Created: {i[\"created_at\"][:19]}')
"

3. Communicate

For P1/P2 incidents:

Post in #incidents:

INCIDENT IN PROGRESS
- Service: [SERVICE_NAME]
- Impact: [DESCRIPTION]
- Status: Investigating
- IC: [YOUR_NAME]
- Next update: [TIME]

Escalate if needed:

• Backend issues → @backend-team
• AI/ML issues → @ai-team
• Infrastructure → @platform-team
• Data pipeline → @data-team

4. Mitigate

Common quick fixes:

Rollback commands (K8s):

# Rollback to previous revision
kubectl rollout undo deployment/eyvax-backend-server -n eyvax-backend

# Check rollback status
kubectl rollout status deployment/eyvax-backend-server -n eyvax-backend
# Expected: "deployment "eyvax-backend-server" successfully rolled out"
# Stuck: "Waiting for deployment ... rollout to finish: X out of Y new replicas have been updated..."

# ArgoCD rollback (if deployed via ArgoCD)
argocd app history eyvax-backend-server
argocd app rollback eyvax-backend-server <revision-number>

5. Resolve

Verify resolution:

# 1. Check health endpoint returns 200
curl -s -o /dev/null -w "%{http_code}" https://api.eyvax.com/health
# Expected: 200

# 2. Bulk-resolve PagerDuty incidents
# First, list open incidents and get their IDs
INCIDENT_IDS=$(curl -s -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=50" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
incidents = data.get('incidents', [])
payload = [{'id': i['id'], 'type': 'incident_reference', 'status': 'resolved'} for i in incidents]
print(json.dumps({'incidents': payload}))
")

# Then resolve them all at once
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d "$INCIDENT_IDS"

# 3. Bulk-resolve stale Sentry issues (optional)
# Resolve all unresolved issues in a project where last event was > 1 hour ago
curl -s -X PUT -H "Authorization: Bearer $SENTRY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved"}' \
  "https://sentry.io/api/0/projects/bluesemi-qv/<project>/issues/?query=is:unresolved+lastSeen:-1h"

Resolution message template:

INCIDENT RESOLVED
- Service: [SERVICE_NAME]
- Duration: [X] minutes
- Root cause: [BRIEF DESCRIPTION]
- Resolution: [WHAT FIXED IT]
- Postmortem: [Yes/No - scheduled for DATE]

Service-Specific Runbooks

Escalation Matrix

Emergency contacts:

• Primary on-call: Check PagerDuty schedule
• Secondary on-call: Check PagerDuty schedule
• Engineering Lead: Contact via Slack

After the Incident

P1/P2 Incidents Require:

1. Postmortem within 48 hours (template)
2. Blameless review meeting within 1 week
3. Action items tracked to completion

Metrics to Track:

• Time to acknowledge (TTA)
• Time to mitigate (TTM)
• Time to resolve (TTR)
• Customer impact duration

SLO Targets Reference