EyvaX AI Service Runbook

ML-powered service for vital signs computation including heart rate, HRV, blood pressure, SpO2, and glucose prediction.

Tech Stack: Python 3.10, PyTorch, FastAPI, MongoDB, Redis, NumPy/SciPy Repo: software-bluesemi/eyvax-ai Namespace: eyvax-ai Deployment: eyvax-ai ArgoCD App: eyvax-ai Sentry Project: eyvax-ai (may not exist — events may go to eyvax-backend-server)

Table of Contents

1. Infrastructure Overview
2. Health Check Endpoints
3. 502 Errors and Cascading Failures
4. Model Inference Issues
5. Memory & Resource Issues
6. Signal Processing Errors
7. Database Issues
8. Performance Issues
9. Deployment & Rollback
10. CI/CD Failure Patterns
11. Sentry Investigation
12. PagerDuty Operations
13. Real Incident Patterns
14. Escalation

Infrastructure Overview

K8s Resource Configuration

Container: eyvax-ai-api (port 8045)

Container: eyvax-ai-dashboard (port 8501)

Container: redis-cache sidecar (port 6379)

Probe Configuration

VPA Configuration

GKE Cluster Commands

# Check pods
kubectl get pods -n eyvax-ai -l app=eyvax-ai -o wide

# Expected healthy:
# NAME                       READY   STATUS    RESTARTS   AGE    IP           NODE
# eyvax-ai-xxx-yyy           3/3     Running   0          2d     10.x.x.x     gke-xxx
# eyvax-ai-xxx-zzz           3/3     Running   0          2d     10.x.x.x     gke-xxx
#
# Note: READY 3/3 because there are 3 containers (api, dashboard, redis)
# READY 2/3 → one container not ready (check which: -c flag)
# OOMKilled → memory exceeded 4Gi limit

# Check deployment
kubectl get deployment eyvax-ai -n eyvax-ai

# Check HPA
kubectl get hpa -n eyvax-ai

# Check resource usage
kubectl top pods -n eyvax-ai

# Expected healthy:
# CPU ~100-200m, Memory ~800Mi-1.5Gi per pod
# Warning if Memory > 3.2Gi (80% of 4Gi limit) → OOM risk

# Check pod events (useful for OOMKilled, CrashLoopBackOff)
kubectl describe pod -n eyvax-ai -l app=eyvax-ai | grep -A10 "Events:"

ArgoCD

# Check sync status
kubectl get application eyvax-ai -n argocd -o jsonpath='{.status.sync.status} {.status.health.status}'

# Expected: "Synced Healthy"

# Force sync
argocd app sync eyvax-ai

Key Environment Variables

Health Check Endpoints

Response Schemas

GET /health (port 8045) — Startup + readiness probe

curl -s https://eyvax-ai.eyva.io/health | python3 -m json.tool

Healthy response (HTTP 200):

{
    "status": "healthy",
    "timestamp": "2026-03-03T10:00:00",
    "version": "1.0.0",
    "dependencies": {
        "mongodb": {"healthy": true},
        "redis": {"healthy": true}
    },
    "models": {
        "glucose_transformer": {"loaded": true, "version": "2.0"},
        "blood_pressure_bilstm": {"loaded": true, "version": "2.0"},
        "hrv_analyzer": {"loaded": true, "version": "1.0"}
    }
}

Unhealthy response (HTTP 503):

{
    "status": "unhealthy",
    "dependencies": {
        "mongodb": {"healthy": false, "error": "connection timeout"},
        "redis": {"healthy": true}
    }
}

GET /live (port 8045) — Liveness probe

Always HTTP 200 if process is alive:

{"status": "alive", "timestamp": "2026-03-03T10:00:00"}

GET / (port 8501) — Dashboard liveness

Returns Streamlit dashboard HTML. HTTP 200 = healthy.

502 Errors and Cascading Failures

This is the most critical section. eyvax-ai is a shared dependency — when it returns 502, both data-pipeline and backend-server are affected.

Downstream Impact When eyvax-ai Returns 502

Decision Tree: eyvax-ai 502 Errors

IF eyvax-ai returning 502:
  → Step 1: Check pod status
    kubectl get pods -n eyvax-ai -l app=eyvax-ai

  IF any pod STATUS = OOMKilled:
    → Check restart count
    IF restart_count < 3 in last 30 minutes:
      → AUTO-RESOLVABLE: Rolling restart
        kubectl rollout restart deployment/eyvax-ai -n eyvax-ai
      → Monitor: kubectl get pods -n eyvax-ai -w
      → Wait up to 195s (startup probe window) for pods to become ready
      → After pods ready, verify: curl -s https://eyvax-ai.eyva.io/health
      IF healthy → auto-resolve PD incident
    IF restart_count >= 3:
      → Escalate P2 — repeated OOMKills indicate memory limit too low
      → Immediate mitigation: set DEVICE=cpu to reduce memory usage

  IF any pod STATUS = CrashLoopBackOff:
    → Check logs: kubectl logs -n eyvax-ai <pod> -c eyvax-ai-api --previous --tail=50
    IF "CUDA out of memory":
      → GPU memory exhaustion
      → Set DEVICE=cpu as env var and restart
    IF "model file not found" or "ModuleNotFoundError":
      → Bad image or model artifact issue
      → Escalate P2 — needs rollback or rebuild
    IF "sqlite3.DatabaseError: database disk image is malformed":
      → Cache corruption
      → Delete cache and restart:
        kubectl exec deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api -- rm -f /tmp/eyvax_cache.db*
        kubectl rollout restart deployment/eyvax-ai -n eyvax-ai
    IF other error:
      → Escalate P2 with error details

  IF all pods STATUS = Running but 502 persists:
    → Step 2: Check resource usage
      kubectl top pods -n eyvax-ai
    IF CPU > 1600m (80% of 2000m limit):
      → CPU throttled. HPA should scale — check:
        kubectl get hpa -n eyvax-ai
      IF replicas < maxReplicas (5) → HPA scaling, wait 2 min
      IF replicas = maxReplicas (5) → at capacity, escalate P2
    IF Memory > 3.2Gi (80% of 4Gi limit):
      → OOM risk imminent
      → Rolling restart: kubectl rollout restart deployment/eyvax-ai -n eyvax-ai
    IF resources look normal:
      → Check downstream dependencies: MongoDB, Redis
      → Check if readiness probe is failing:
        kubectl describe pod -n eyvax-ai -l app=eyvax-ai | grep -A5 "Readiness"

  → Step 3: If eyvax-ai 502 persists > 5 min after restart
    → Escalate P2
    → Check if data-pipeline and backend-server are also alerting (cascade)
    → Downstream services will auto-recover once eyvax-ai is healthy

Model Inference Issues

Model Loading Failures

Symptoms:

• Server startup logs show model_loading_failed
• ML endpoints return 500 errors
• Logs: Model not found at algorithms/models/{model_name}/v{version}/

Diagnosis:

# Check if model files exist
kubectl exec deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api -- ls -la algorithms/models/

# Expected: directories for glucose, blood_pressure, hrv_analyzer
# Missing directories → bad Docker image

# Check model loader logs
kubectl logs deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api --tail=100 | grep -i "model\|loading"

# Expected: "Model loaded successfully: glucose_transformer v2.0"
# Failure: "Model not found" or "Failed to load model"

Decision Tree:

IF model loading failed:
  IF "Model not found at path":
    → Model files missing from container
    → Check if Docker image was built correctly
    → Rollback to previous working image (requires human approval)
  IF "RuntimeError: Invalid model file":
    → Corrupted model artifact
    → Rollback to previous image
  IF "CUDA out of memory" during loading:
    → Set DEVICE=cpu and restart
  → All model loading issues require escalation P2 unless DEVICE=cpu fixes it

CUDA Out-of-Memory (OOM)

Symptoms:

• RuntimeError: CUDA out of memory
• GPU memory spikes during 24h HRV computation

Decision Tree:

IF "CUDA out of memory" in logs:
  → Step 1: Check if DEVICE is set to cuda or cpu
    kubectl exec deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api -- env | grep DEVICE
  IF DEVICE=cuda:
    → Switch to CPU: set DEVICE=cpu in environment
    → This reduces memory usage but increases latency
    → Rolling restart after env change
  IF already DEVICE=cpu:
    → CPU memory OOM, not GPU
    → Escalate P2 — needs resource limit increase

Tensor Shape Mismatch

Symptoms:

• RuntimeError: size mismatch: input has shape [1, 20] but expected [18]
• 500 errors on specific endpoints

Resolution:

• Client sending wrong number of features
• Escalate to AI team — this is a code/model compatibility issue
• Not auto-resolvable

Memory & Resource Issues

Container OOMKilled

Symptoms:

• Pod status shows OOMKilled
• Container restart loop
• kubectl describe pod shows Last State: Terminated, Reason: OOMKilled

Decision Tree:

IF pod OOMKilled:
  → Step 1: Check restart count
    kubectl get pods -n eyvax-ai -l app=eyvax-ai -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount}{"\n"}{end}'

  IF restart_count < 3 in last 30 minutes:
    → AUTO-RESOLVABLE: Rolling restart
      kubectl rollout restart deployment/eyvax-ai -n eyvax-ai
    → Monitor for 5 min
    IF restarts again → go to restart_count >= 3

  IF restart_count >= 3:
    → DO NOT auto-restart — will just OOM again
    → Immediate mitigation:
      → Set DEVICE=cpu (if using GPU) to reduce memory footprint
    → Escalate P2 — needs resource limit increase (currently 4Gi)
    → Possible causes:
      → Large batch inference request
      → Memory leak in signal processing
      → Model loaded twice due to process fork

Cache Database Corruption

Symptoms:

• sqlite3.DatabaseError: database disk image is malformed
• HRV cache lookups fail

Resolution (AUTO-RESOLVABLE):

# Delete corrupted cache
kubectl exec deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api -- rm -f /tmp/eyvax_cache.db*

# Restart to rebuild cache
kubectl rollout restart deployment/eyvax-ai -n eyvax-ai

# Verify after restart (wait for startup probe: 195s max)
kubectl exec deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api -- sqlite3 /tmp/eyvax_cache.db "PRAGMA integrity_check"
# Expected: "ok"

Memory Leak in Signal Processing

Symptoms:

• Memory usage grows linearly over time
• Pod eventually OOMKilled after hours/days of uptime

Decision Tree:

IF memory usage > 3Gi and pod uptime > 24h:
  → Likely memory leak
  → AUTO-RESOLVABLE: Rolling restart
    kubectl rollout restart deployment/eyvax-ai -n eyvax-ai
  → This is a temporary fix — the leak will recur
  → Escalate P3 for code-level investigation

Signal Processing Errors

Insufficient RR Intervals for HRV

Symptoms:

• API returns 422 with PeakDetectionError: peaks_found: 1, required: 2
• Quality score = 0.0

Resolution: This is a CLIENT-SIDE issue (poor signal quality). No server action needed.

• Return graceful error to user
• Not a production incident — no escalation needed
• High volume is normal (depends on user measurement conditions)

Motion Artifact Detection

Symptoms:

• Quality scores artificially low
• Heart rate underestimation

Resolution: Requires AI team investigation. Tuning MOTION_AMPLITUDE_THRESHOLD is not safe for L1.

Database Issues

MongoDB Connection Pool Exhaustion

Symptoms:

• pymongo.errors.OperationFailure: waitQueueTimeoutMS expired
• API returns 503

Decision Tree:

IF "waitQueueTimeoutMS expired" in logs:
  → Step 1: Check current connections
    kubectl exec deploy/eyvax-ai -n eyvax-ai -c eyvax-ai-api -- python3 -c "
import pymongo, os
c = pymongo.MongoClient(os.environ['MONGODB_URI'])
print(c.admin.command('serverStatus')['connections'])
"
  → Step 2: If connections > 80:
    → Pool near exhaustion
    → Rolling restart to reset connections:
      kubectl rollout restart deployment/eyvax-ai -n eyvax-ai
    → If recurring, set MONGODB_MAX_POOL_SIZE=200

Performance Issues

Slow HRV Computation (>5s)

SLO: Model inference P95 < 1000ms, overall P95 < 2000ms

Decision Tree:

IF HRV latency > 5s:
  IF HRV_ENABLE_FREQUENCY_DOMAIN=true:
    → Frequency domain computation is expensive
    → Consider setting HRV_ENABLE_FREQUENCY_DOMAIN=false for 2x speedup
    → WARNING: This reduces measurement precision. Requires human approval.
  IF already false:
    → Check CPU usage — may be throttled
    → Check if caching is enabled (ENABLE_CACHING=true, REDIS_HOST set)
    → Escalate P3 for optimization

Deployment & Rollback

Normal Deployment Flow

Code merged to main → GitHub Actions CI → Docker image built →
Image pushed to Artifact Registry → k8s/eyvax-deployment.yaml updated →
ArgoCD detects change → Rolling update (maxSurge=1, maxUnavailable=0)

Rollback

# Check rollout history
kubectl rollout history deployment/eyvax-ai -n eyvax-ai

# Rollback to previous revision (REQUIRES HUMAN APPROVAL)
kubectl rollout undo deployment/eyvax-ai -n eyvax-ai

# Verify rollback
kubectl rollout status deployment/eyvax-ai -n eyvax-ai
# Expected: "deployment "eyvax-ai" successfully rolled out"

Stuck Rolling Update

IF new pods not becoming ready after 195s (startup probe max):
  → Check new pod logs:
    kubectl logs <new-pod> -n eyvax-ai -c eyvax-ai-api --tail=50
  IF "model file not found" → bad image, rollback
  IF "MONGODB_URI" not set → secret not mounted, check ExternalSecret
  IF startup timeout → resource constraints, check kubectl top
  → If stuck > 10 min → rollback (requires human approval)

CI/CD Failure Patterns

Decision Tree

IF code merged but deployment not updated:
  → Check CI: gh run list --repo software-bluesemi/eyvax-ai --limit 3
  IF CI failing:
    → Check failure: gh run view <id> --log-failed
    → Fix tests/lint on new branch, merge, CI will re-run
  IF CI passing but ArgoCD not synced:
    → Force sync: argocd app sync eyvax-ai

Sentry Investigation

# Check eyvax-ai Sentry issues
# NOTE: eyvax-ai may not have its own Sentry project.
# Check eyvax-backend-server project and filter by server_name tag containing "eyvax-ai"

curl -s -H "Authorization: Bearer $SENTRY_API_KEY" \
  "https://sentry.io/api/0/projects/bluesemi-qv/eyvax-backend-server/issues/?query=is:unresolved&sort=date&statsPeriod=24h&limit=10" | \
  python3 -c "
import json, sys
issues = json.load(sys.stdin)
if isinstance(issues, list):
  for i in issues:
    print(f'[{i[\"level\"]}] {i[\"count\"]}x | {i[\"title\"][:80]} | Last: {i[\"lastSeen\"][:19]}')
"

# To check if an issue is from eyvax-ai, look at event tags:
# curl -s -H "Authorization: Bearer $SENTRY_API_KEY" \
#   "https://sentry.io/api/0/issues/<ISSUE_ID>/events/?limit=1" | python3 -c "
# import json, sys
# events = json.load(sys.stdin)
# for e in events:
#   for t in e.get('tags', []):
#     if t['key'] == 'server_name':
#       print(f'  server_name: {t[\"value\"]}')"

PagerDuty Operations

# List open incidents
curl -s -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=25" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
for i in data.get('incidents', []):
  print(f'[{i[\"status\"]}] {i[\"title\"][:80]} | ID: {i[\"id\"]}')
"

# Acknowledge an incident
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{"incidents": [{"id": "<ID>", "type": "incident_reference", "status": "acknowledged"}]}'

# Resolve an incident
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{"incidents": [{"id": "<ID>", "type": "incident_reference", "status": "resolved"}]}'

Real Incident Patterns

eyvax-ai 502 Cascading Failures — March 2026

What happened: Resource limits were reduced in January 2026 (commit 1bfdd9d) based on average usage analysis (CPU 250m→100m, memory 1Gi→512Mi). Under peak load, the reduced resources caused 502 errors and OOMKilled pods. This cascaded to data-pipeline (all metric computation stopped, consumer lag grew to 10k+) and backend-server (AI chat failures).

Root cause: Resources right-sized for average load but insufficient for peak load. HPA couldn't scale fast enough because startup takes ~195s.

Fix (PR #67):

• Restored resources: CPU request 250m, limit 2000m; Memory request 1Gi, limit 4Gi
• Increased base replicas from 1 to 2 (always have spare capacity)
• Extended startup probe: failureThreshold 10→18, timeout 5s→10s
• Increased HPA max from 3 to 5
• Updated VPA min allowed to match restored values

Detection pattern: If you see eyvax-ai 502s combined with data-pipeline consumer lag and backend AI errors, check if resource limits were recently reduced. Compare current resource requests against the values in this table (Infrastructure Overview section).

Lesson: Never reduce eyvax-ai resources below: CPU request 250m, Memory request 1Gi. Peak load requires headroom.

Escalation

If unable to resolve within 15 minutes:

1. Check if multiple services are affected (cascade — fix eyvax-ai first, see incident-response.md)
2. Check Sentry for error traces
3. Review recent deployments (kubectl rollout history deployment/eyvax-ai -n eyvax-ai)
4. Escalate to AI/ML team via PagerDuty
5. If P1, follow escalation matrix in incident-response.md