Operations Runbook

This runbook distils the operational procedures documented in the README into a checklist that on-call responders can follow when maintaining a Crisis-Monitor deployment.

Contacts and escalation

Primary maintainers: the people listed in CONTRIBUTING.md.
Security incidents: follow the process in SECURITY.md and notify the maintainers privately.
Infrastructure problems: raise an internal ticket and document the incident in the /shift-log interface so the next shift can review the outcome.

Daily checklist

Confirm the application is reachable and log in with an admin account.
Open /admin/monitoring and verify that background tasks and external integrations are green.
For source diagnostics, use the authenticated GET /api/v1/sources/health contract. Read check_status together with state and event_freshness_state; do not infer a provider outage from a quiet event stream alone.
Check /admin/backups to ensure a recent archive exists in the directory defined by BACKUP_DIR.
Review the dashboard at /dashboard for unusual activity.
If teams are present, review the activation board for staffing, active load and overdue handoffs. Treat acknowledgement as unavailable until the dedicated audit contract exists; it is not inferred from assignment alone.
Review logs/ for errors and keep the directory under source control out of version control.

Public UI and responsive acceptance

After a UI or template release, keep the evidence layers separate: a local render, a healthy runtime and a public browser result are three different checks. The current public acceptance route is https://crisis.qdev.run.

Run the public contract check: .venv/bin/python scripts/self_check.py --scope prod --base-url https://crisis.qdev.run --default-json.
Run the bounded performance/indexing smoke: bash scripts/perf_smoke.sh https://crisis.qdev.run.
Run the existing Playwright route audit at 1440px, 1024px and 390px. A protected /admin or /tickets/new response may be 403 for a public viewer; it is not a layout failure.
For the very-wide pass, inspect the critical route family at 2560px and 3440px: /dashboard, /tickets, /kanban, /briefing, /cop/, /analytics/, /analytics/zones, /watch-zones/, /disinformation/dashboard, /teams/, /help/modules, a ticket detail route and a synthetic 404. Require bodyOverflow=0, no clipped controls and expected statuses at both widths.
Record the exact source SHA, public asset marker and residual warnings in a dated audit. The current evidence is CRISIS_BENCHMARK_WIDE_RELEASE_2026-08-04.md.

An external image or provider warning must remain labelled as an external dependency issue until it is independently fixed; it must not be silently reported as a CSS or route regression.

Scheduled jobs

Periodic tasks run via the ARQ worker (arq services.worker.WorkerSettings). In production, the same tasks can alternatively be invoked via system cron using CLI commands defined in cli.py. See crontab.example in the project root for a ready-to-use crontab configuration.

The commands rely on environment variables documented in the README (for example UPLOAD_FOLDER, BACKUP_DIR, LOG_DIR and TEMP_DIR). Ensure the cron user can read and write to those directories.

Configuration diff: staging vs production

Keep .env.example aligned with config.CONFIG_ENV_VARS and treat the values listed there as development defaults only. Secrets for long-lived deployments are sourced from the Secret Manager. The table below summarises the key flag differences we enforce between staging and production environments:

Variable	Staging	Production	Notes
`APP_ENV`	`staging`	`production`	Selects `BaseConfig` for staging and `ProductionConfig` for prod so that debugging is disabled everywhere but production-only overrides (mail validation, compression) run only in prod.
`ENABLE_SCHEDULER`	`false`	`true`	Staging jobs are triggered manually or via ad-hoc runs to avoid double processing; production runs the ARQ worker for periodic tasks.
`ALERTS_ENABLED`	`false`	`true`	Prevents staging health-check failures from paging the on-call rotation.
`AUDIT_LOG_SINK_ENABLED`	`false`	`true`	Keeps staging traffic out of the external audit sink, which in production receives a URL from the Secret Manager.
`MAIL_*` credentials	Sandbox SMTP credentials from the Secret Manager	Real provider credentials from the Secret Manager	Both environments load the secrets from the same source; staging points to a non-delivery sandbox.
`DATABASE_URL`, `REDIS_URL`, `CACHE_REDIS_URL`	Staging database/cache URLs from the Secret Manager	Production database/cache URLs from the Secret Manager	Ensures isolated data stores with identical schema migrations.
`TELEGRAM_TOKEN`/`TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID`	Test bot and chat IDs stored in the Secret Manager	Live bot/channel IDs from the Secret Manager	`TELEGRAM_TOKEN` is preferred, but legacy deployments setting only `TELEGRAM_BOT_TOKEN` continue to work.
`SENTRY_DSN`	Staging Sentry project DSN from the Secret Manager	Production Sentry DSN from the Secret Manager	Keeps error aggregation separated per environment.

Email alert configuration

Set ALERT_EMAIL_FROM, ALERT_EMAIL_TO, MAIL_SERVER and MAIL_PORT to route health check failures to the on-call inbox.
Enable encrypted transport by setting MAIL_USE_TLS=true for STARTTLS on the configured port or MAIL_USE_SSL=true for implicit TLS (usually on port 465). Only one of these options should be true at a time.
After updating credentials or toggling TLS/SSL, send a manual test email by running flask --app app:create_app shell and calling current_app.monitoring_scheduler._send_email('test message') or triggering a known-failing health check to confirm delivery.

Manual maintenance tasks

Backups: trigger POST /admin/backups/create from the admin UI or run the corresponding CLI/cron entrypoint in the production container. Verify the SQLite format 3\x00 header after extracting the file before replacing a live database.
Operational credibility pack: after a content refresh or before a demo, run python scripts/seed_operator_case_pack.py followed by python scripts/seed_operational_credibility_pack.py --apply inside the application runtime. The second command is idempotent and only fills operational context around existing records: response resources, static map objects, communication templates, assignment rules and shift-log guidance. It does not create incidents and does not mark evidence as verified.
Restoring data: stop the web container, restore the chosen backup into the mapped data location, then recreate the service with docker compose -f docker-compose.prod.yml up -d web.
Cleanup: call flask --app app:create_app shell and run cleanup_old_files() if old logs accumulate faster than the scheduled job can remove them.
Reminders: execute the worker/cron task inside the application runtime if scheduled notifications fall behind; do not rely on an ad-hoc python run.py process in production.

Deployments and upgrades

On the host, verify that /opt/crisis-monitor does not contain unrelated uncommitted changes. scripts/deploy.sh now aborts on a dirty worktree unless ALLOW_DIRTY_WORKTREE=1 is set explicitly. The script also reports OCI image label drift separately from source drift, so operators can distinguish a code-only restart from a planned image refresh.
The deploy script applies pending Alembic migrations before warming operator routes. Confirm flask --app wsgi.py db check is clean after the restart.
Use bash scripts/deploy.sh for the normal release path. The default DEPLOY_BUILD=auto rebuilds only when Docker/dependency/frontend build inputs changed. Python code, templates and direct static assets are shipped by pulling the source and restarting web/arq-worker, because production bind-mounts /opt/crisis-monitor into /app.
If the OCI image label must be refreshed for audit evidence, run DEPLOY_ENFORCE_IMAGE_VERSION=1 bash scripts/deploy.sh. Use DEPLOY_BUILD=always only during a planned application image refresh or after confirming Docker storage has enough free space. The database image is not rebuilt by DEPLOY_BUILD=always; set DEPLOY_BUILD_DB=1 only when intentionally refreshing docker/db/. On the shared production VPS, do not respond to BuildKit failures with broad Docker pruning. If BuildKit repeatedly exits with context canceled during the runtime dependency layer while the host is otherwise healthy, use the legacy builder for this project image only: DOCKER_BUILDKIT=0 docker build -q --build-arg APP_VERSION=${APP_VERSION} -t crisis-monitor-web:latest ., then docker tag crisis-monitor-web:latest crisis-monitor-arq-worker:latest and recreate only web/arq-worker with the production compose file. Verify the image label, both runtime dependency versions and /health before treating the release as complete.
If Docker reports a transient container-name conflict while recreating web/arq-worker, rerun bash scripts/deploy.sh after the first retry finishes. The deploy script retries the application recreate once and does not remove volumes or rebuild the database service. If Compose leaves temporary prefixed container names after a recreate race, first confirm the canonical names are free and the containers belong to this compose project, then rename them back to crisis-monitor-web-1 and crisis-monitor-arq-worker-1. Do not remove volumes or unrelated containers to solve a naming race.
If a schema change is included, run flask --app app:create_app db upgrade in the web container or release job before the smoke pass.
Run python scripts/verify_translations_runtime.py and flask --app wsgi.py db check in the web container. The standard scripts/deploy.sh performs both checks and stops on missing RU/KK catalogs or application-owned metadata drift.
Watch docker compose -f docker-compose.prod.yml logs web --tail 50 and verify /admin/monitoring plus the affected UI routes.

Docker storage maintenance

The normal release path must not prune Docker resources. Run storage maintenance as a separate window when df -h / or docker system df shows pressure before an enforced image rebuild.

Record the current state: df -h / /var/lib/docker 2>/dev/null || df -h / and docker system df.
Create the read-only inventory bundle: python scripts/docker_storage_inventory.py --pretty > /tmp/crisis-docker-storage-inventory.json. The script only runs df, docker system df, docker compose ps, docker image ls and docker volume ls; it never prunes or removes resources.
Confirm docker compose -f docker-compose.prod.yml ps shows db, redis, web and arq-worker healthy.
Clear only unused images and build cache after confirming no other active release is using them. Do not prune volumes during routine Crisis-Monitor maintenance.
Rerun bash scripts/deploy.sh, then repeat the public smoke pass for /dashboard, /kanban, /briefing, /cop/, /watch-zones/ and /disinformation/dashboard.

Document significant incidents, maintenance windows and follow-up actions in the shift log so the team shares the same operational picture.