EyvaX Backend Server Runbook

FastAPI-based backend service handling authentication, AI chat, health tracking, and user management.

Tech Stack: Python 3.12, FastAPI, MongoDB (Motor async), Redis, ClickHouse, Kafka Repo: software-bluesemi/eyvax-backend-server Namespace: eyvax-backend Deployment: eyvax-backend-server ArgoCD App: eyvax-backend-server Sentry Project: eyvax-backend-server

Table of Contents

1. Infrastructure Overview
2. Health Check Endpoints
3. Database Issues
4. Authentication Issues
5. Circuit Breaker Patterns
6. External Service Failures
7. Rate Limiting Issues
8. Performance Issues
9. Deployment & Rollback
10. CI/CD Failure Patterns
11. Sentry Investigation
12. PagerDuty Operations
13. Real Incident Patterns
14. Escalation

Infrastructure Overview

K8s Resource Configuration

Container: eyvax-backend-server (port 8000)

Container: redis-cache sidecar (port 6379)

Probe Configuration

GKE Cluster Commands

# Set cluster context
gcloud container clusters get-credentials eyvax-dev-zone --region asia-south1 --project eyva-440707

# Check pods
kubectl get pods -n eyvax-backend -l app=eyvax-backend-server -o wide

# Expected healthy output:
# NAME                                    READY   STATUS    RESTARTS   AGE    IP             NODE
# eyvax-backend-server-xxx-yyy            2/2     Running   0          2d     10.x.x.x       gke-xxx
# eyvax-backend-server-xxx-zzz            2/2     Running   0          2d     10.x.x.x       gke-xxx
#
# Unhealthy indicators:
# READY 1/2 → one container (redis sidecar?) not ready
# STATUS OOMKilled → memory limit (2Gi) exceeded
# STATUS CrashLoopBackOff → application crash, check logs
# RESTARTS > 3 → crash-looping

# Check deployment status
kubectl get deployment eyvax-backend-server -n eyvax-backend

# Expected healthy:
# NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
# eyvax-backend-server     2/2     2            2           14d
#
# READY 0/2 → all pods failing
# UP-TO-DATE < desired → stuck rolling update
# AVAILABLE 0 → service completely down

# Check HPA
kubectl get hpa -n eyvax-backend

# Expected:
# NAME                         REFERENCE                       TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
# eyvax-backend-server-hpa     Deployment/eyvax-backend-server 30%/70%, 45%/80% 2         5         2          14d
#
# If TARGETS show <unknown>/70% → metrics-server issue

# Check resource usage
kubectl top pods -n eyvax-backend -l app=eyvax-backend-server

# Expected (healthy):
# NAME                                    CPU(cores)   MEMORY(bytes)
# eyvax-backend-server-xxx-yyy            50m          400Mi
# eyvax-backend-server-xxx-zzz            45m          380Mi
#
# Warning if CPU > 800m (80% of limit) or Memory > 1.6Gi (80% of limit)

ArgoCD

# Check sync status
kubectl get application eyvax-backend-server -n argocd -o jsonpath='{.status.sync.status} {.status.health.status}'

# Expected: "Synced Healthy"
# "OutOfSync Healthy" → manifest changes not yet applied
# "Synced Degraded" → pods not healthy
# "Synced Progressing" → rollout in progress

# Force sync
argocd app sync eyvax-backend-server

Secrets Management

Secrets are managed via GCP Secret Manager → External Secrets Operator → K8s Secrets.

# Check ExternalSecret status
kubectl get externalsecret -n eyvax-backend

# Expected: STATUS=SecretSynced
# If STATUS=SecretSyncError → check GCP SM permissions

# Force secret refresh
kubectl annotate externalsecret <name> -n eyvax-backend force-sync=$(date +%s) --overwrite

Health Check Endpoints

Response Schemas

GET /health — Startup probe + comprehensive check

curl -s https://api.eyvax.com/health | python3 -m json.tool

Healthy response (HTTP 200):

{
    "status": "healthy",
    "timestamp": 1709424000.123,
    "version": "1.0.0",
    "checks": {
        "database": {
            "status": "healthy",
            "type": "mongodb"
        },
        "database_write": {
            "status": "healthy"
        }
    }
}

Unhealthy response (HTTP 503):

{
    "status": "unhealthy",
    "timestamp": 1709424000.123,
    "version": "1.0.0",
    "checks": {
        "database": {
            "status": "unhealthy",
            "type": "mongodb"
        },
        "database_write": {
            "status": "unhealthy"
        }
    }
}

GET /ready — Readiness probe

Healthy (HTTP 200):

{"status": "ready"}

Not ready (HTTP 503):

{"status": "not ready", "reason": "database not connected"}

GET /live — Liveness probe

Always returns HTTP 200 if process is running:

{"status": "alive", "timestamp": 1709424000.123}

Health Check Decision Tree

IF /health returns HTTP 503:
  → Parse response JSON
  IF checks.database.status == "unhealthy":
    → MongoDB issue — see Database Issues section
  IF checks.database_write.status == "unhealthy":
    → MongoDB write issue — possible read replica lag or auth problem
  → In all cases, check pod logs: kubectl logs deploy/eyvax-backend-server -n eyvax-backend --tail=100

IF /ready returns HTTP 503:
  → Pod not ready to serve traffic
  → Kubernetes will stop sending traffic to this pod
  IF reason == "database not connected":
    → MongoDB connectivity issue — see Database Issues
  → Check if pod is still starting up (within 225s startup window)

IF /live returns non-200 or times out:
  → Process is dead or hung
  → Kubernetes will kill and restart the pod automatically
  → If repeated restarts → check CrashLoopBackOff below

Database Issues

MongoDB Connection Failure

Symptoms:

• API returns 503 Service Unavailable
• Health check shows checks.database.status: "unhealthy"
• Logs show ServerSelectionTimeoutError or connectTimeoutMS exceeded
• Circuit breaker database_init in OPEN state

Decision Tree:

IF /health shows database unhealthy:
  → Step 1: Check logs
    kubectl logs deploy/eyvax-backend-server -n eyvax-backend --tail=100 | grep -i "mongo\|ServerSelection"

  IF logs contain "ServerSelectionTimeoutError":
    → Step 2: Check MongoDB Atlas status
      # Visit https://cloud.mongodb.com or check status page
    IF Atlas is up and accessible:
      → Step 3: Check circuit breaker
        curl -s https://api.eyvax.com/admin/circuit-breakers | python3 -m json.tool
      IF database_init circuit breaker is OPEN:
        → Reset it:
          curl -s -X POST https://api.eyvax.com/admin/circuit-breakers/database_init/reset
        → Wait 30s, re-check /health
        IF healthy → auto-resolve PD incident
        ELSE → rolling restart: kubectl rollout restart deploy/eyvax-backend-server -n eyvax-backend
      ELSE (breaker CLOSED but still unhealthy):
        → Rolling restart: kubectl rollout restart deploy/eyvax-backend-server -n eyvax-backend
        → Wait 225s (startup probe window)
        IF still unhealthy → escalate P2

    IF Atlas is down:
      → Escalate P1 immediately (external dependency, cannot auto-fix)

  IF logs contain "connection pool exhausted" or "Too many open connections":
    → Connection pool exhaustion
    → Rolling restart to reset connections:
      kubectl rollout restart deploy/eyvax-backend-server -n eyvax-backend
    → Monitor: mongodb_connections_current metric should drop

  IF logs contain "Authentication failed":
    → Credential mismatch
    → Check ExternalSecret: kubectl get externalsecret -n eyvax-backend
    → Force sync: kubectl annotate externalsecret <name> -n eyvax-backend force-sync=$(date +%s) --overwrite
    → Escalate P2 if sync fails

Diagnosis Commands:

# Check MongoDB connectivity from inside the pod
kubectl exec deploy/eyvax-backend-server -n eyvax-backend -c eyvax-backend-server -- \
  python3 -c "import asyncio; from motor.motor_asyncio import AsyncIOMotorClient; import os; c=AsyncIOMotorClient(os.environ['MONGODB_URL']); print(asyncio.run(c.admin.command('ping')))"

# Expected: {'ok': 1.0}
# Failure: ServerSelectionTimeoutError or ConnectionRefusedError

# Check circuit breaker states
curl -s https://api.eyvax.com/admin/circuit-breakers | python3 -m json.tool

# Expected: each breaker shows "state": "CLOSED", "failure_count": 0
# Problem: "state": "OPEN" → underlying service is failing

ClickHouse Unavailable

Symptoms:

• Analytics endpoints return errors with ERR_EXTERNAL_4006
• Logs show ClickHouse connection failed

Decision Tree:

IF ClickHouse errors in logs:
  → ClickHouse is non-critical — backend server continues without it
  → Analytics features degrade but core API works
  IF error rate is low (<1%) → monitor, no action needed
  IF error rate is high → check ClickHouse cluster health
    kubectl exec deploy/eyvax-backend-server -n eyvax-backend -- \
      python3 -c "import clickhouse_connect; c=clickhouse_connect.get_client(host='$CH_HOST'); print(c.query('SELECT 1').result_rows)"
  → No auto-resolution — escalate to data team if persistent

Authentication Issues

JWT Token Validation Failures

Error Codes: ERR_AUTH_1001 (missing), ERR_AUTH_1002 (invalid), ERR_AUTH_1003 (expired), ERR_AUTH_1010 (refresh token invalid)

Decision Tree:

IF high volume of ERR_AUTH_1001/1002/1003 errors:
  → Check if this is baseline noise or a spike
  → Baseline: 35-1000+ auth errors/day is NORMAL (expired tokens, invalid client requests)

  IF error count is within 2x of daily baseline:
    → Normal — no action needed
    → These are client-side issues (expired tokens, bad requests)

  IF error count is 5x+ above baseline:
    → Possible JWT_SECRET_KEY rotation or mismatch
    → Check: kubectl exec deploy/eyvax-backend-server -n eyvax-backend -- env | grep JWT_SECRET_KEY
    → Verify same key across all pods
    → If key was recently rotated → all existing tokens are invalid, expected spike
    → Escalate P3 if not explained by rotation

  IF ERR_AUTH_1010 (refresh token) spike:
    → Refresh token secret may have changed
    → Escalate P2 — users cannot re-authenticate

Diagnosis:

# Decode a JWT token to check expiry and claims
echo $TOKEN | cut -d'.' -f2 | base64 -d 2>/dev/null | python3 -m json.tool

# Check JWT secret is configured
kubectl exec deploy/eyvax-backend-server -n eyvax-backend -c eyvax-backend-server -- env | grep JWT_SECRET_KEY
# Should return a non-empty value. If empty → escalate immediately

# Check OTP verification issues
kubectl logs deploy/eyvax-backend-server -n eyvax-backend --tail=200 | grep -i "otp\|sms\|2factor\|mandrill"

Circuit Breaker Patterns

How Circuit Breakers Work

The backend server uses circuit breakers to protect against cascading failures. Each breaker has three states:

CLOSED (normal) → OPEN (after N failures) → HALF_OPEN (after recovery_timeout) → CLOSED (after N successes)
                                          → OPEN (if failures resume in HALF_OPEN)

Circuit Breaker Configuration

Error Responses When Breaker is Open

• HTTP 503 with {"error_code": "CIRCUIT_BREAKER_OPEN"} or {"error_code": "ERR_SYSTEM_7004"}
• HTTP 504 with {"error_code": "CIRCUIT_BREAKER_TIMEOUT"}

Admin Endpoints

# View all circuit breaker states
curl -s https://api.eyvax.com/admin/circuit-breakers | python3 -m json.tool

# Expected (healthy): all breakers show state=CLOSED, failure_count=0
# Problem: state=OPEN means the underlying dependency is failing

# Reset a specific breaker
curl -s -X POST https://api.eyvax.com/admin/circuit-breakers/{breaker_name}/reset

# Reset ALL breakers
curl -s -X POST https://api.eyvax.com/admin/circuit-breakers/reset-all

Decision Tree: Circuit Breaker Issues

IF error response contains "CIRCUIT_BREAKER_OPEN" or "ERR_SYSTEM_7004":
  → Step 1: Identify which breaker is open
    curl -s https://api.eyvax.com/admin/circuit-breakers | python3 -c "
import json, sys
breakers = json.load(sys.stdin)
for name, state in breakers.items():
  if state.get('state') != 'CLOSED':
    print(f'  OPEN: {name} (failures: {state.get(\"failure_count\", \"?\")})')
"

  IF only ONE breaker is OPEN:
    → Step 2: Check if the underlying dependency is actually down
      IF breaker is database_init or database_operations:
        → Check MongoDB: see Database Issues section
      IF breaker is ai_provider:
        → Check AI service: curl -s https://eyvax-ai.eyva.io/health
        → If eyvax-ai is healthy, reset breaker:
          curl -X POST https://api.eyvax.com/admin/circuit-breakers/ai_provider/reset
      IF breaker is clickhouse:
        → Non-critical, reset and monitor
      IF breaker is external_email, external_sms, external_weather:
        → Non-critical external services, reset and monitor
      IF breaker is redis_operations:
        → Check Redis sidecar: kubectl exec deploy/eyvax-backend-server -n eyvax-backend -c redis-cache -- redis-cli ping
        → Expected: PONG. If no response → restart pod

    → Step 3: Reset the breaker
      curl -X POST https://api.eyvax.com/admin/circuit-breakers/{name}/reset
    → Step 4: Monitor for 60s (recovery_timeout)
      IF breaker stays CLOSED → auto-resolve PD
      IF breaker re-opens within 60s → underlying dependency is actually down, escalate

  IF MULTIPLE breakers are OPEN:
    → Likely a cascading failure
    → Check eyvax-ai health first (most common root cause)
    → If eyvax-ai is healthy, check MongoDB
    → Reset all breakers: curl -X POST https://api.eyvax.com/admin/circuit-breakers/reset-all
    → If breakers keep re-opening → escalate P2

External Service Failures

AI Provider Failures (OpenAI/Anthropic/Gemini/OpenRouter)

Symptoms:

• AI chat endpoints slow or failing
• Logs show API rate limit, quota exhausted, or 429 errors
• Error code ERR_EXTERNAL_4003

LLM Provider Fallback Chain:

The backend uses generate_answer_with_fallback() which tries providers in priority order:

1. OpenRouter
2. Gemini Flash
3. Claude Haiku
4. GPT Full

If a provider returns a retryable error (5xx, 429, rate_limit, overloaded, insufficient_quota), the system automatically falls back to the next provider.

Decision Tree:

IF AI chat errors in logs:
  IF error is 429 (rate limit) or "insufficient_quota":
    → System should auto-fallback to next provider
    → If fallback is working → no action needed, monitor
    → If ALL providers failing:
      → Check provider status pages
      → Escalate P2 if sustained > 5 min

  IF error is 5xx from provider:
    → Auto-fallback should handle this
    → Monitor fallback success rate in logs

  IF error is "CIRCUIT_BREAKER_OPEN" for ai_provider:
    → All providers failed 5 times consecutively
    → Wait 60s for auto-recovery (HALF_OPEN)
    → Or reset: curl -X POST https://api.eyvax.com/admin/circuit-breakers/ai_provider/reset

External API Timeouts (Email/SMS/Weather)

Non-critical services with circuit breaker protection. Backend continues functioning without them.

IF external_email/sms/weather breaker OPEN:
  → Non-critical, notifications will be delayed
  → Wait for 60s automatic recovery
  → If persistent > 10 min → check external service status pages
  → No escalation needed unless impacting core user flows

Rate Limiting Issues

Configuration

• Global rate limit: 100 requests/minute per IP (via slowapi)
• Daily AI quotas by tier:
  • Internal Team: 1000/day
  • Premium: 100/day
  • Performance: 75/day
  • Standard: 50/day

Error Codes

Decision Tree

IF rate limit errors spiking:
  IF error is ERR_RATE_LIMIT_5001 (per-minute):
    → Normal rate limiting
    → If from many different IPs → traffic spike, may need to scale
    → If from single IP → possible bot/abuse, monitor

  IF error is ERR_RATE_LIMIT_5003 (daily quota):
    → User exhausted their AI quota for the day
    → No auto-resolution — quota resets at midnight UTC
    → If legitimate user needs more → escalate to support for tier upgrade

  → Rate limiting errors are NEVER a reason to escalate or page
  → They are a safety mechanism working as intended

Performance Issues

High API Latency

SLO Thresholds:

• P95 < 500ms (critical alert at > 500ms, warning at > 400ms)
• P99 < 1000ms
• Endpoint-specific: /health < 50ms, /api/v1/auth/login < 300ms, /api/v1/ai/chat < 5000ms

Decision Tree:

IF BackendServerHighLatency alert fires (P95 > 500ms):
  → Step 1: Check which endpoints are slow
    # Check which endpoints have high latency

  IF /api/v1/ai/chat is slow but other endpoints normal:
    → AI provider latency issue
    → Check eyvax-ai health: curl -s https://eyvax-ai.eyva.io/health
    → If eyvax-ai slow → see ai-service.md
    → Not actionable at backend level — AI endpoints have 5s SLO

  IF all endpoints slow:
    → Step 2: Check resource usage
      kubectl top pods -n eyvax-backend -l app=eyvax-backend-server
    IF CPU > 800m (80% of 1000m limit):
      → HPA should be scaling. Check: kubectl get hpa -n eyvax-backend
      → If already at max replicas (5) → escalate P2
    IF Memory > 1.6Gi (80% of 2Gi limit):
      → OOM risk — rolling restart:
        kubectl rollout restart deploy/eyvax-backend-server -n eyvax-backend

  IF MongoDB queries are slow:
    → Check slow query logs
    → Escalate to backend team for index optimization

Deployment & Rollback

Normal Deployment Flow

Code merged to main → GitHub Actions CI → Docker image built →
Image pushed to Artifact Registry → deployment.yaml updated with new tag →
ArgoCD detects change → Rolling update (maxSurge=1, maxUnavailable=0)

Checking Deployment Status

# Check rollout status
kubectl rollout status deployment/eyvax-backend-server -n eyvax-backend

# Expected (success): "deployment "eyvax-backend-server" successfully rolled out"
# Expected (in progress): "Waiting for deployment ... rollout to finish: 1 out of 2 new replicas have been updated..."
# Stuck (> 225s): likely startup probe failing on new pods

# Check current image
kubectl get deployment eyvax-backend-server -n eyvax-backend -o jsonpath='{.spec.template.spec.containers[0].image}'
# Expected: asia-south1-docker.pkg.dev/eyva-440707/apps/eyvax-backend-server:<commit-hash>

Rollback

# Rollback to previous revision
kubectl rollout undo deployment/eyvax-backend-server -n eyvax-backend

# Verify rollback
kubectl rollout status deployment/eyvax-backend-server -n eyvax-backend
# Expected: "deployment "eyvax-backend-server" successfully rolled out"

# NOTE: This requires human approval per auto-resolution safety boundaries

Stuck Rolling Update Decision Tree

IF rollout stuck (new pods not becoming ready):
  → Step 1: Check new pod status
    kubectl get pods -n eyvax-backend -l app=eyvax-backend-server
    # Look for pods in Pending/ContainerCreating/CrashLoopBackOff

  IF new pods CrashLoopBackOff:
    → Check logs: kubectl logs <new-pod> -n eyvax-backend --previous --tail=50
    IF "ImportError" or "ModuleNotFoundError" → bad image, rollback
    IF "MONGODB_URL" missing → secret not mounted, check ExternalSecret
    IF startup crash → rollback to previous image

  IF new pods running but not ready (readiness probe failing):
    → /ready returning 503 → database not connected
    → Check MongoDB connectivity from new pod
    → May resolve once MongoDB connection establishes (wait up to 225s)

  IF pods stuck in Pending:
    → Resource quota or node capacity issue
    → kubectl describe pod <pending-pod> -n eyvax-backend → check Events
    → If "insufficient cpu/memory" → node scaling needed, escalate

  IF stuck > 10 min:
    → Rollback: kubectl rollout undo deployment/eyvax-backend-server -n eyvax-backend
    → Escalate P2 for investigation

CI/CD Failure Patterns

CI Pipeline Structure

The CI pipeline runs on push to main:

1. Secrets Scan — TruffleHog scans for leaked secrets
2. Lint — flake8 + bandit security scan
3. Test — pytest with coverage
4. Build & Push Docker Image — builds and pushes to Artifact Registry, updates deployment.yaml

Decision Tree: CI Failure Blocking Deployment

IF code merged to main but deployment not updated:
  → Step 1: Check CI status
    gh run list --repo software-bluesemi/eyvax-backend-server --limit 3

  IF latest CI run status = "failure":
    → Step 2: Check which step failed
      gh run view <run-id> --log-failed

    IF Test step failed:
      → Pre-existing test issue or new test regression
      → Check if failure is in test code or source code
      → Fix tests on a new branch, create PR, merge
      → CI will re-run on main and auto-deploy

    IF Lint step failed:
      → Code style or security issue
      → Fix lint issues on a new branch

    IF Build step failed:
      → Docker build error (missing dependencies, syntax error)
      → Check Dockerfile and requirements.txt
      → Escalate to backend team

    IF Secrets Scan failed:
      → Leaked secret detected — CRITICAL
      → Rotate the exposed secret immediately
      → Escalate P1

  IF latest CI run status = "success" but deployment not updated:
    → Check ArgoCD sync status
    → Force sync: argocd app sync eyvax-backend-server

Sentry Investigation

Querying Sentry Issues

# List unresolved issues (last 24h)
curl -s -H "Authorization: Bearer $SENTRY_API_KEY" \
  "https://sentry.io/api/0/projects/bluesemi-qv/eyvax-backend-server/issues/?query=is:unresolved&sort=date&statsPeriod=24h&limit=20" | \
  python3 -c "
import json, sys
issues = json.load(sys.stdin)
if isinstance(issues, list):
  for i in issues:
    print(f'[{i[\"level\"]}] {i[\"count\"]}x | {i[\"title\"][:80]} | Last: {i[\"lastSeen\"][:19]} | ID: {i[\"id\"]}')
else:
  print('API error:', issues)
"

# NOTE: emd-glucose events may appear here due to shared Sentry DSN
# Check server_name tag in events to identify actual source service

Common Error Categories

Resolving Sentry Issues

# Resolve a specific issue
curl -s -X PUT -H "Authorization: Bearer $SENTRY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved"}' \
  "https://sentry.io/api/0/issues/<ISSUE_ID>/"

# Bulk resolve all unresolved issues (use with caution)
curl -s -X PUT -H "Authorization: Bearer $SENTRY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved"}' \
  "https://sentry.io/api/0/projects/bluesemi-qv/eyvax-backend-server/issues/?query=is:unresolved"

PagerDuty Operations

Managing Incidents

# List open incidents
curl -s -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=25" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
for i in data.get('incidents', []):
  print(f'[{i[\"status\"]}] {i[\"title\"][:80]} | ID: {i[\"id\"]}')
"

# Acknowledge an incident
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{"incidents": [{"id": "<ID>", "type": "incident_reference", "status": "acknowledged"}]}'

# Resolve an incident
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{"incidents": [{"id": "<ID>", "type": "incident_reference", "status": "resolved"}]}'

Real Incident Patterns

OpenAI 429 Rate Limit — March 2026

What happened: OpenAI returned 429 (rate limit exceeded) errors, but the is_retryable_llm_error() function only checked for 5xx errors. The 429 wasn't caught as retryable, so no fallback to alternative providers occurred. Users saw AI chat failures.

Root cause: generate_insight() in answer.py used a hardcoded two-provider try/except instead of the full fallback chain.

Fix (PR #135):

• Added 429, rate_limit, and insufficient_quota to is_retryable_llm_error() in llm.py
• Updated generate_insight() to use generate_answer_with_fallback() with full provider priority: OpenRouter → Gemini Flash → Claude Haiku → GPT Full

Detection pattern: Sentry errors with openai.RateLimitError or HTTP 429 from OpenAI. If you see these, check if the fallback chain is working.

Sentry DSN Misconfiguration — Ongoing

What happened: emd-glucose service shares the same Sentry DSN as eyvax-backend-server. All emd-glucose errors appear in the backend-server Sentry project.

How to identify: Check the server_name tag in Sentry events. If it shows emd-glucose-api-* pod names, the error is from emd-glucose, not backend-server.

Impact: Sentry issue counts for backend-server are inflated. Always check server_name before investigating.

Environment Variables

Escalation

If unable to resolve within 15 minutes:

1. Check Sentry for detailed error traces
2. Review recent deployments (kubectl rollout history)
3. Check if multiple services are affected (cascade — see incident-response.md)
4. Escalate to backend team via PagerDuty
5. If P1, follow escalation matrix in incident-response.md