EyvaX Data Pipeline Runbook

Real-time sensor data processing pipeline consuming from Kafka, computing metrics via eyvax-ai/emd-glucose, and storing in ClickHouse.

Tech Stack: Python 3.9, Kafka/RedPanda, ClickHouse, FastAPI, NumPy/SciPy Repo: software-bluesemi/eyvax-data-pipeline Namespace: eyvax-data ArgoCD App: eyvax-data-pipeline Sentry Project: eyvax-data-pipeline

Table of Contents

1. Infrastructure Overview
2. Health Check Endpoints
3. Kafka/Messaging Issues
4. Data Quality Issues
5. Database/Storage Issues
6. External Service Issues
7. Performance Issues
8. Batch Job Issues
9. Deployment & Rollback
10. CI/CD Failure Patterns
11. Sentry Investigation
12. PagerDuty Operations
13. Escalation

Infrastructure Overview

This service has 4 deployments in the eyvax-data namespace:

Deployments

Resource Configuration

eyvax-sensor-raw-data-processor

Probes: Startup GET /health:8080 (delay=10s, period=10s, failThreshold=12 → max 130s), Liveness GET /health:8080 (delay=30s, period=30s), Readiness GET /ready:8080 (delay=15s, period=10s)

eyvax-vitals-server

HPA: min=2, max=4 (CPU 70%, Memory 80%)

CronJob: batch-metrics-processor

CronJob: daily-backfill-processor

Namespace ResourceQuota

GKE Cluster Commands

# Check all pods in namespace
kubectl get pods -n eyvax-data -o wide

# Expected healthy:
# NAME                                              READY   STATUS    RESTARTS   AGE
# eyvax-sensor-raw-data-processor-xxx-yyy           2/2     Running   0          2d
# eyvax-sensor-raw-data-processor-xxx-zzz           2/2     Running   0          2d
# eyvax-vitals-server-xxx-yyy                       1/1     Running   0          2d
# eyvax-vitals-server-xxx-zzz                       1/1     Running   0          2d
#
# Note: sensor-raw-data-processor has 2/2 (app + redis sidecar)
# CronJob pods appear only during scheduled runs

# Check CronJobs
kubectl get cronjob -n eyvax-data

# Expected:
# NAME                        SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
# batch-metrics-processor     */5 * * * *   False     0        2m              14d
# daily-backfill-processor    0 0 * * *     False     0        12h             14d

# Check recent jobs
kubectl get jobs -n eyvax-data --sort-by=.metadata.creationTimestamp | tail -5

# Check resource usage
kubectl top pods -n eyvax-data

Getting Kafka Broker Address

# Get KAFKA_BROKER from the pod's environment
kubectl exec deploy/eyvax-sensor-raw-data-processor -n eyvax-data -- env | grep KAFKA

# Expected output includes:
# KAFKA_BOOTSTRAP_SERVERS=<broker-address>
# KAFKA_TOPIC=eyvax-raw-sensor-data
# KAFKA_CONSUMER_GROUP=eyvax-sensor-consumer

Key Dependencies

Health Check Endpoints

The sensor-raw-data-processor runs a health server on port 8080.

Response Schemas

GET /health (port 8080) — Overall health with dependency checks

Healthy response (HTTP 200):

{
    "status": "healthy",
    "timestamp": "2026-03-03T10:00:00+00:00",
    "service": "eyvax-sensor-raw-data-processor",
    "version": "1.0.0",
    "checks": [
        {"name": "kafka", "status": "healthy", "message": "Connected to Kafka broker", "details": {}},
        {"name": "clickhouse", "status": "healthy", "message": "ClickHouse connection OK"},
        {"name": "database_schema", "status": "healthy", "message": "sensor_data table exists"}
    ]
}

Degraded response (HTTP 503):

{
    "status": "degraded",
    "checks": [
        {"name": "kafka", "status": "healthy"},
        {"name": "clickhouse", "status": "unhealthy", "message": "Connection refused"},
        {"name": "database_schema", "status": "unhealthy", "message": "sensor_data table not found"}
    ]
}

Readiness critical checks: Both kafka AND clickhouse must be "healthy" for /ready to return 200.

GET /health/ready (port 8080) — Readiness probe

Ready (HTTP 200):

{"status": "ready", "timestamp": "...", "service": "eyvax-sensor-raw-data-processor"}

Not ready (HTTP 503):

{"status": "not_ready", "timestamp": "...", "service": "..."}

GET /health/live (port 8080) — Liveness (always 200)

{"status": "alive", "timestamp": "...", "service": "eyvax-sensor-raw-data-processor"}

Kafka/Messaging Issues

Kafka Connection Failure

Symptoms:

• Consumer not processing messages
• Logs show KAFKA_CONNECTION_ERROR or KafkaConnectionError
• Health check shows kafka status: unhealthy

Decision Tree:

IF /health shows kafka unhealthy OR DataPipelineConsumerDown alert:
  → Step 1: Check consumer pod status
    kubectl get pods -n eyvax-data -l app=eyvax-sensor-raw-data-processor

  IF pods not running:
    → Rolling restart:
      kubectl rollout restart deployment/eyvax-sensor-raw-data-processor -n eyvax-data
    → Wait 130s (startup probe window)
    → If still failing → escalate P1 (Kafka broker may be down)

  IF pods running but kafka unhealthy:
    → Step 2: Check logs for specific error
      kubectl logs deploy/eyvax-sensor-raw-data-processor -n eyvax-data --tail=100 | grep -i "kafka\|connection\|broker"

    IF "NoBrokersAvailable" or "Connection refused":
      → Kafka broker is unreachable
      → Escalate P1 — infrastructure issue

    IF "SASL authentication failed":
      → Credential issue
      → Check ExternalSecret for Kafka credentials
      → Escalate P2

    IF "Topic 'eyvax-raw-sensor-data' not found":
      → Topic was deleted (rare but critical)
      → Escalate P1 — need to recreate topic
      → NOTE: Consumer group offset reset may be needed after topic recreation

  → Step 3: If Kafka is accessible but consumer stuck
    → Rolling restart consumer:
      kubectl rollout restart deployment/eyvax-sensor-raw-data-processor -n eyvax-data
    → Monitor consumer lag for 5 min
    IF lag decreasing → recovering, auto-resolve after lag < 500
    IF lag still growing → escalate P2

High Consumer Lag

Alert Thresholds:

• Warning: kafka_consumer_lag > 500 messages (5 min)
• Critical: kafka_consumer_lag > 1000 messages (5 min)
• Severe: kafka_consumer_lag > 10000 messages (1 min)

Decision Tree:

IF DataPipelineHighConsumerLag alert (lag > 1000):
  → Step 1: Check current lag trend
    # Is lag increasing, stable, or decreasing?

  IF lag > 10000 AND increasing:
    → Step 2: Determine root cause
      kubectl logs deploy/eyvax-sensor-raw-data-processor -n eyvax-data --tail=50 | grep -i "error\|timeout\|failed"

    IF logs show errors calling eyvax-ai:
      → eyvax-ai is slow or down — check eyvax-ai health first
        curl -s https://eyvax-ai.eyva.io/health | python3 -m json.tool
      IF eyvax-ai unhealthy → fix eyvax-ai first (see ai-service.md)
        → Lag will drain once eyvax-ai recovers
        → Do NOT restart data-pipeline — it's a downstream victim

    IF logs show ClickHouse insert errors:
      → ClickHouse is the bottleneck
      → See ClickHouse Insert Failures section

    IF no specific errors, just slow processing:
      → Rolling restart consumer:
        kubectl rollout restart deployment/eyvax-sensor-raw-data-processor -n eyvax-data
      → If lag continues growing → escalate P2

  IF lag > 1000 but stable or decreasing:
    → Consumer is catching up — monitor for 10 min
    → Auto-resolve PD if lag drops below 500
    → No action needed unless lag growth resumes

  IF lag > 500 (warning) but < 1000:
    → Monitor only, likely a temporary spike
    → Auto-resolve if lag returns to normal within 15 min

HTTP Retry Logic

The data pipeline uses async HTTP with retry for API calls:

• Connection pool: 100 total, 20 per host
• Max retries: 3 per metric call
• 4xx errors: no retry (client error)
• 5xx errors: retry with exponential backoff (0s, 2s, 4s)
• Timeout: retry with same backoff

Kafka Producer Config:

• ACKS=all, RETRIES=3, REQUEST_TIMEOUT_MS=30000
• DELIVERY_TIMEOUT_MS=120000, COMPRESSION_TYPE=gzip

Data Quality Issues

No Valid Peaks Detected

Symptoms:

• Logs: No valid peak intervals detected
• Heart rate = 0 for a device
• Quality score = 0.0

Resolution: Client/hardware issue, not a production incident. Signal too noisy or sensor contact poor. No server action needed.

Buffer Not Filling

Symptoms:

• Device data arriving but buffer not reaching threshold (10s default)
• Config: BUFFER_THRESHOLD_SECONDS=10, MAX_DEVICES=1000, CLEANUP_INTERVAL_SECONDS=60

Resolution: Intermittent device connectivity. Buffers auto-expire. No action needed unless systematic across many devices.

Database/Storage Issues

ClickHouse Connection Failure

Symptoms:

• Health check shows clickhouse status: unhealthy
• Logs show CLICKHOUSE_CONNECTION_ERROR

Decision Tree:

IF ClickHouse connectivity failing:
  → Step 1: Test from inside the pod
    kubectl exec deploy/eyvax-sensor-raw-data-processor -n eyvax-data -- python3 -c "
import clickhouse_connect, os
c = clickhouse_connect.get_client(host=os.environ.get('CLICKHOUSE_HOST','localhost'))
print(c.query('SELECT 1').result_rows)
"
  # Expected: [[1]]

  IF connection refused:
    → ClickHouse server is down
    → Escalate P1 — critical data store

  IF authentication error:
    → Check credentials in ExternalSecret
    → Escalate P2

  IF timeout:
    → ClickHouse overloaded
    → Check ClickHouse cluster health
    → Escalate P2

ClickHouse Insert Failures

Alert: DataPipelineClickHouseInsertFailures fires when insert failure rate > 0.1%

Decision Tree:

IF ClickHouse insert failures:
  → Check logs for specific error:
    kubectl logs deploy/eyvax-sensor-raw-data-processor -n eyvax-data --tail=100 | grep -i "insert\|clickhouse"

  IF "No such column" or schema error:
    → Schema mismatch between code and table
    → Check table schema:
      kubectl exec deploy/eyvax-sensor-raw-data-processor -n eyvax-data -- python3 -c "
import clickhouse_connect, os
c = clickhouse_connect.get_client(host=os.environ.get('CLICKHOUSE_HOST'))
print(c.query('DESCRIBE TABLE sensor_data').result_rows)
"
    → Escalate P2 — needs schema migration

  IF "Memory limit exceeded" or resource error:
    → ClickHouse out of resources
    → Escalate P2

  IF "Too many parts":
    → ClickHouse merge backlog
    → Wait for auto-merge, monitor insert rate

External Service Issues

eyvax-ai Service Unavailable

Symptoms:

• Logs: Failed to compute metric via eyvax-ai
• Metrics computation fails for HR, HRV, BP, SpO2
• Consumer lag grows (blocked on API calls)
• Error code: EXTERNAL_SERVICE_ERROR

4xx vs 5xx Handling:

Decision Tree:

IF eyvax-ai errors causing consumer lag:
  → Check eyvax-ai health first:
    curl -s https://eyvax-ai.eyva.io/health | python3 -m json.tool

  IF eyvax-ai unhealthy:
    → Fix eyvax-ai first — see ai-service.md
    → Data pipeline will automatically retry on recovery
    → Consumer lag will drain once eyvax-ai is back

  IF eyvax-ai healthy but slow (latency > 2s):
    → eyvax-ai under load, HPA may be scaling
    → Wait 5 min for scaling to take effect
    → If still slow → check eyvax-ai resource usage

  IF eyvax-ai healthy and fast:
    → Network issue between namespaces
    → Rolling restart consumer:
      kubectl rollout restart deployment/eyvax-sensor-raw-data-processor -n eyvax-data

Performance Issues

Key Metrics

Decision Tree: Slow Processing

IF consumer lag growing due to slow processing:
  → Step 1: Check resource usage
    kubectl top pods -n eyvax-data

  IF CPU near limit (>400m of 500m):
    → CPU throttled
    → Check HPA: kubectl get hpa -n eyvax-data
    → If vitals-server at max replicas → escalate P2

  IF Memory near limit:
    → OOM risk
    → Rolling restart: kubectl rollout restart deployment/eyvax-sensor-raw-data-processor -n eyvax-data

  IF resources normal:
    → Bottleneck is likely external (eyvax-ai, ClickHouse)
    → Check eyvax-ai latency
    → Check ClickHouse insert latency

Batch Job Issues

CronJob Not Running

Expected Schedules:

• batch-metrics-processor: every 5 minutes (*/5 * * * *)
• daily-backfill-processor: midnight UTC (0 0 * * *)

Decision Tree:

IF batch-metrics-processor not running (no recent jobs):
  → Step 1: Check CronJob status
    kubectl get cronjob batch-metrics-processor -n eyvax-data

    IF SUSPEND=True:
      → CronJob was suspended
      → Resume: kubectl patch cronjob batch-metrics-processor -n eyvax-data -p '{"spec":{"suspend":false}}'

    IF SUSPEND=False but no recent LAST SCHEDULE:
      → Check recent job history
        kubectl get jobs -n eyvax-data --sort-by=.metadata.creationTimestamp | tail -5

      IF last successful run > 15 min ago:
        → Step 2: Check most recent job logs
          JOB=$(kubectl get jobs -n eyvax-data --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
          kubectl logs job/$JOB -n eyvax-data --tail=50

        IF "activeDeadlineSeconds exceeded":
          → Job ran too long (batch-metrics: no deadline, backfill: 2h max)
          → Check ClickHouse performance and eyvax-ai latency

        IF "backoffLimitExceeded" (backoff limit = 2):
          → Job failed all retries
          → Check API errors in logs

        → Step 3: Manual trigger
          kubectl create job --from=cronjob/batch-metrics-processor batch-manual-$(date +%s) -n eyvax-data
          → Watch: kubectl get jobs -n eyvax-data -w

      IF last run < 10 min ago → normal, next run is scheduled

Backfill / Reprocessing

# Manually trigger daily backfill for a specific date
kubectl create job --from=cronjob/daily-backfill-processor backfill-manual-$(date +%s) -n eyvax-data

# The backfill job processes all devices for yesterday by default
# Config: BACKFILL_MAX_WINDOWS_PER_RUN=288 (288 x 5min = 24h)
# Max runtime: 2 hours (activeDeadlineSeconds=7200)

# Monitor backfill progress
kubectl logs -f job/<job-name> -n eyvax-data

Deployment & Rollback

Rollback

# Rollback sensor processor
kubectl rollout undo deployment/eyvax-sensor-raw-data-processor -n eyvax-data

# Rollback vitals server
kubectl rollout undo deployment/eyvax-vitals-server -n eyvax-data

# Verify
kubectl rollout status deployment/eyvax-sensor-raw-data-processor -n eyvax-data
# Expected: "successfully rolled out"

CI/CD Failure Patterns

IF code merged but deployment not updated:
  → Check CI: gh run list --repo software-bluesemi/eyvax-data-pipeline --limit 3
  IF CI failing:
    → Check failure: gh run view <id> --log-failed
    → Fix on new branch, merge
  IF CI passing but ArgoCD not synced:
    → Force sync: argocd app sync eyvax-data-pipeline

Sentry Investigation

# List unresolved issues
curl -s -H "Authorization: Bearer $SENTRY_API_KEY" \
  "https://sentry.io/api/0/projects/bluesemi-qv/eyvax-data-pipeline/issues/?query=is:unresolved&sort=date&statsPeriod=24h&limit=10" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
if isinstance(data, list):
  for i in data:
    print(f'[{i[\"level\"]}] {i[\"count\"]}x | {i[\"title\"][:80]} | Last: {i[\"lastSeen\"][:19]}')
elif isinstance(data, dict) and 'detail' in data:
  print(f'Project not found or API error: {data[\"detail\"]}')
"

# Sentry Noise vs Real Issues
# NOISE (safe to ignore/resolve):
#   - 422 errors from eyvax-ai (bad signal quality, client issue)
#   - "No valid peak intervals detected" (hardware/user issue)
#   - Individual message parsing failures (malformed device data)
#
# REAL ISSUES (investigate):
#   - KafkaConnectionError (broker connectivity)
#   - ClickHouseConnectionError (database connectivity)
#   - Sustained ExternalServiceError (eyvax-ai down)
#   - DatabaseOperationError (data corruption)

PagerDuty Operations

# List open incidents
curl -s -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=25" | \
  python3 -c "
import json, sys
for i in json.load(sys.stdin).get('incidents', []):
  print(f'[{i[\"status\"]}] {i[\"title\"][:80]} | ID: {i[\"id\"]}')
"

# Acknowledge
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{"incidents": [{"id": "<ID>", "type": "incident_reference", "status": "acknowledged"}]}'

# Resolve
curl -s -X PUT "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "From: $PAGERDUTY_EMAIL" \
  -d '{"incidents": [{"id": "<ID>", "type": "incident_reference", "status": "resolved"}]}'

Escalation

If unable to resolve within 15 minutes:

1. Check if eyvax-ai is the root cause (cascade — see incident-response.md)
2. Check Kafka broker health (infrastructure team)
3. Check ClickHouse cluster health
4. Escalate to data engineering team via PagerDuty
5. If P1, follow escalation matrix in incident-response.md