swadhin / opencodehub public
/ opencodehub / docs-site / src / content / docs / administration / operations-drills.md
Swadhin Biswas feat(prod): production readiness — SLOs, runbooks, drills, and CI hardening · 4/21/2026
436debf
108 lines · 2.3 KB
Raw Download

title: "Operations Drills Runbook"

This runbook defines recurring resilience drills and backup rehearsals for OpenCodeHub staging/production readiness.

Scope

  • Redis outage behavior
  • PostgreSQL reconnect behavior
  • Backup + restore rehearsal
  • CI failure ownership and escalation

Prerequisites

  • Docker + Docker Compose available
  • Services runnable via docker-compose.yml
  • Access to app health endpoint (/api/health)

CI Failure Ownership

See .github/ci/failure-ownership-map.md for lane ownership, SLA, and escalation.

Drill 1: Redis Outage

Run:

npm run drill:redis

Artifacts:

  • test-results/drills/redis-drill-baseline-health.json
  • test-results/drills/redis-drill-degraded-health.json
  • test-results/drills/redis-drill-recovered-health.json
  • test-results/drills/redis-drill-app.log

Expected outcomes:

  1. Service remains responsive (may degrade, but should not crash loop).
  2. Health report shows redis/scaling impact while Redis is down.
  3. Health and behavior recover after Redis restart.

Drill 2: PostgreSQL Reconnect

Run:

npm run drill:postgres

Artifacts:

  • test-results/drills/postgres-drill-baseline-health.json
  • test-results/drills/postgres-drill-degraded-health.json
  • test-results/drills/postgres-drill-recovered-health.json
  • test-results/drills/postgres-drill-app.log

Expected outcomes:

  1. Health endpoint indicates database failure during outage.
  2. App reconnects when PostgreSQL is restored.
  3. No persistent crash state after recovery.

Drill 3: Backup + Restore Rehearsal

Safe (non-destructive) rehearsal:

npm run drill:backup-restore

Full rehearsal (destructive restore path):

RESTORE_MODE=full npm run drill:backup-restore

Artifacts:

  • test-results/drills/backup-drill.log
  • test-results/drills/backup-verify-drill.log
  • test-results/drills/restore-drill.log

Expected outcomes:

  1. Backup archive is generated.
  2. Backup verification passes checksum and manifest checks.
  3. Restore path executes successfully in selected mode.

Suggested Frequency

  • Redis outage drill: weekly
  • PostgreSQL reconnect drill: weekly
  • Backup+restore rehearsal: weekly (safe), monthly (full)

Reporting Template

Capture for each run:

  • Date/time
  • Operator
  • Environment
  • Result (pass/fail)
  • Mean time to recovery (if outage simulated)
  • Follow-up action items