Skip to content

Operations & Deployment Runbook

Field Value
Document ID DA-OPS-001
Pillar Software Engineering
Standard Alignment ISO/IEC/IEEE 26514 / SRE
Status Active / Synchronized with Codebase
Last Updated 2026-11-21
Source of Truth compose.yaml, compose.dashboard-public.yaml, compose.test.yaml, .github/workflows/ci-deploy.yml, scripts/bootstrap.py, scripts/configure.py, scripts/configure_cloudflare.py, scripts/pair.py, scripts/backup.sh, scripts/deploy-server.sh, scripts/setup-cd.py, scripts/smoke.py, app/operator.py, app/dashboard_auth.py, docs/CI-CD.md, docs/dashboard-deployment.md

  • Operating System & Runtime: Linux host (/opt/dirman-assist) with Docker Engine, Docker Compose v2, Python 3, rsync, flock, and curl.
  • Disk Space: scripts/deploy-server.sh enforces > 2 GiB (2,097,152 KiB) free disk space before staging a deployment.
  • Configured Container Resource Ceilings (compose.yaml):
Service Memory Limit CPU Limit PID Limit Persistent Volume
postgres 512m 0.5 128 postgres:/var/lib/postgresql/data
api 512m 0.5 128 Stateless (read_only: true, /tmp tmpfs)
worker 512m 0.5 128 Stateless (read_only: true, /tmp tmpfs)
dashboard 512m 0.5 128 Stateless (read_only: true, /tmp tmpfs)
topic-worker 192m 0.25 128 Stateless (read_only: true, /tmp tmpfs)
pdf-worker 640m 0.75 128 Stateless (read_only: true, /tmp tmpfs)
waha 2g 1.0 256 waha:/app/.sessions
Stack Total 4.87 GiB 4.0 vCPU — 2 named Docker volumes

Run from /opt/dirman-assist on the host:

Step 1: Bootstrap Protected .env & Build Image

Section titled “Step 1: Bootstrap Protected .env & Build Image”
Terminal window
python3 scripts/bootstrap.py
  • Pulls devlikeapro/waha:latest, postgres:17-alpine, and python:3.12-slim and pins each to its immutable RepoDigests[0].
  • Generates random 32-byte hex credentials (POSTGRES_PASSWORD, WAHA_API_KEY + WAHA_API_KEY_PLAIN, WEBHOOK_SECRET), writes .env under umask 0o077 (chmod 0600) with OUTBOUND_ENABLED=false and PUBLIC_INTAKE=false, validates Compose config, and builds dirman-assist-app:local.
Terminal window
python3 scripts/configure.py
  • Interactively prompts for GOOGLE_API_KEY (hidden input), atomically updates .env, and recreates api and worker.
  • Optional free-tier Cloudflare Workers AI fallback (@cf/google/gemma-4-26b-a4b-it):
Terminal window
python3 scripts/configure_cloudflare.py

Requires typing FREE to confirm a Workers Free account, runs a live JSON smoke test inside worker, writes CLOUDFLARE_* variables atomically to .env, restarts api/worker, and sets ai_provider_status (provider='cloudflare').

Step 3: Start Services & Provision Initial Admin Accounts

Section titled “Step 3: Start Services & Provision Initial Admin Accounts”
Terminal window
docker compose -f compose.yaml -f compose.dashboard-public.yaml up -d

Provision the initial administrator member/API token (app/operator.py) and dashboard login (app/dashboard_auth.py) inside private containers (umask 0o077, output files never overwritten):

Terminal window
docker compose exec -T api python -m app.operator bootstrap-admin --name "Admin Name" --phone 628XXXXXXXXXX --output /tmp/admin-token
docker compose exec -T dashboard python -m app.dashboard_auth --username adminuser --member-id <MEMBER_UUID> --output /tmp/dashboard-login.txt

Retrieve the generated credentials privately and delete temporary credential files after onboarding.

Section titled “Step 4: Link WhatsApp Session (scripts/pair.py)”
Terminal window
python3 scripts/pair.py
  • Prompts for PAIR confirmation, starts WAHA default session via loopback http://127.0.0.1:13000, polls for QR PNG, and saves secrets/whatsapp-qr.png (0o600).
  • Scan via WhatsApp -> Linked Devices, delete secrets/whatsapp-qr.png, and only then set OUTBOUND_ENABLED=true (and PUBLIC_INTAKE=true if admitting unknown WhatsApp numbers) in .env and recreate api and worker.

All secrets belong exclusively in protected server .env (0o600). Never commit .env or print credentials.

Variable Services Default / Format Purpose
POSTGRES_IMAGE postgres Pinned postgres:17-alpine@sha256:<digest> Immutable PostgreSQL image digest
PYTHON_IMAGE Build arg (x-app) Pinned python:3.12-slim@sha256:<digest> Immutable Python base image digest
WAHA_IMAGE waha Pinned devlikeapro/waha:latest@sha256:<digest> Immutable WAHA WEBJS image digest
POSTGRES_PASSWORD postgres 64-char hex PostgreSQL password for user dirman
DATABASE_URL x-app, dashboard, pdf-worker postgresql://dirman:<pass>@postgres:5432/dirman Primary DB DSN
WAHA_API_KEY waha sha512:<128-hex> WAHA server hashed API key
WAHA_API_KEY_PLAIN api, worker, waha 64-char hex Plain WAHA API key sent in X-Api-Key header
WEBHOOK_SECRET api, waha 64-char hex WHATSAPP_HOOK_HMAC_KEY for x-webhook-hmac SHA-512
GOOGLE_API_KEY api, worker, topic-worker AI Studio key (AIza<token>) Primary Gemma API key (empty disables AI calls)
GOOGLE_MODEL .env (bootstrap seed) gemma-4-26b-a4b-it Initial seed reference (runtime uses ai_configuration DB table)
AI_RPM worker, topic-worker 0 (disabled) Optional local cap on Google requests per UTC minute
AI_RPD worker, topic-worker 0 (disabled) Optional local cap on Google requests per Pacific day
CLOUDFLARE_AI_ENABLED worker 'true' or unset Enables Cloudflare Workers AI fallback
CLOUDFLARE_FREE_PLAN_CONFIRMED worker 'true' or unset Explicit confirmation of Workers Free allowance
CLOUDFLARE_ACCOUNT_ID worker 32-char hex Cloudflare account identifier
CLOUDFLARE_API_TOKEN worker Workers AI Read/Edit token Bearer token for Cloudflare AI fallback
OUTBOUND_ENABLED api, worker 'false' ('true' in prod) Master switch for all outbound WAHA messages
PUBLIC_INTAKE api 'false' ('true' to auto-register) Admits unknown WhatsApp senders as member (placeholder)
DASHBOARD_ORIGIN dashboard https://assist.p3mdi.id Exact allowed Origin for Same-Origin & reset URLs
DASHBOARD_TRUSTED_PROXY_IPS dashboard 127.0.0.1 (set Traefik IP in prod) Trusted reverse-proxy IP(s) for X-Forwarded-For throttling
TEST_DATABASE compose.test.yaml only 'true' Guard flag required by test harness; never set in production

4. Compose Overlays & Traefik Public Exposure

Section titled “4. Compose Overlays & Traefik Public Exposure”
  1. Base Stack (compose.yaml): Binds api (127.0.0.1:18081:8080), dashboard (127.0.0.1:18082:8082), and waha (127.0.0.1:13000:3000) exclusively to loopback with traefik.enable: 'false' on all services.
  2. Public Dashboard Overlay (compose.dashboard-public.yaml):
    • Attaches dashboard alone to the external coolify Docker network (traefik.docker.network: coolify).
    • Enables Traefik router dirman-dashboard-http (Host(\assist.p3mdi.id`), permanent HTTP -> HTTPS redirect) and dirman-dashboard (https, TLS certresolver letsencrypt, backend port 8082`).
    • Always invoke production Compose commands with both files:
      Terminal window
      docker compose -p dirman-assist -f compose.yaml -f compose.dashboard-public.yaml ps
    • Ensure DASHBOARD_TRUSTED_PROXY_IPS in .env includes the Traefik container IP on the coolify network so login throttling (dashboard_login_attempts) hashes real client IPs rather than proxy IPs (docs/dashboard-deployment.md).

5. Automated CI/CD & Manual Deployment Runbook

Section titled “5. Automated CI/CD & Manual Deployment Runbook”

5.1 One-Time CD Controller Setup (scripts/setup-cd.py)

Section titled “5.1 One-Time CD Controller Setup (scripts/setup-cd.py)”

Run from an operator workstation with root SSH and authenticated gh CLI:

  • Verifies the server’s ED25519 SSH host key fingerprint against the pinned constant before installing /usr/local/sbin/dirman-assist-deploy (scripts/deploy-server.sh).
  • Generates /etc/dirman-assist-cd/actions_key, appends restrict,command="/usr/local/sbin/dirman-assist-deploy" to /root/.ssh/authorized_keys, initializes bare mirror /var/lib/dirman-assist-cd/repository.git, and populates GitHub secrets DEPLOY_HOST, DEPLOY_KNOWN_HOSTS, and DEPLOY_SSH_KEY.

5.2 Automated Deployment Flow (scripts/deploy-server.sh)

Section titled “5.2 Automated Deployment Flow (scripts/deploy-server.sh)”
flowchart TD
    GHA["GitHub Actions (main verify passed)"] -->|"SSH forced command: deploy SHA"| Lock["flock -w 900 /var/lock/dirman-assist-deploy.lock"]
    Lock --> Fetch["git fetch origin & verify SHA == origin/main"]
    Fetch --> Disk["Check > 2 GiB free disk & extract release"]
    Disk --> Build["docker build -t dirman-assist-app:SHA"]
    Build --> Backup["Snapshot previous images + source.tgz + pg_dump database.dump"]
    Backup --> Sync["rsync app/ scripts/ docs/ tests/ & compose files (.env & secrets untouched)"]
    Sync --> Migrate["compose run --rm api migrate()"]
    Migrate --> Up["compose up -d --no-deps --wait (api, worker, dashboard, pdf-worker, topic-worker)"]
    Up --> Health["curl -fsS $DASHBOARD_ORIGIN/health (10 retries)"]
    Health -->|"Success"| Stamp["Write deployed-commit"]
    Health -->|"Failure (ERR trap)"| Rollback["Restore source.tgz + previous images via rollback.json (DB preserved)"]
  • Zero-Touch Stateful Services: postgres and waha are never restarted by deploy-server.sh, preserving WAHA sessions and PostgreSQL connections.
  • Standard Rollback: Revert the offending commit on main and push so CI builds and deploys the verified revert. For emergency server-side rollback, note deploy-server.sh automatically restores source.tgz and prior per-service images on failed healthcheck (database.dump is preserved without auto-restore to avoid losing user messages).

Terminal window
sh /opt/dirman-assist/scripts/backup.sh

Writes a custom-format PostgreSQL dump (pg_dump -U dirman -d dirman -Fc) with umask 077 to /opt/dirman-assist/backups/dirman-<UTC_TIMESTAMP>.dump. Automated deployment backups are stored under /var/lib/dirman-assist-cd/deployments/<TIMESTAMP>-<SHA>/database.dump.

6.2 Restore Verification Drill (Non-Destructive)

Section titled “6.2 Restore Verification Drill (Non-Destructive)”

Verify backup integrity periodically against a disposable PostgreSQL container (never overwrite live production blindly):

Terminal window
docker run --rm -d --name dirman-restore-drill -e POSTGRES_PASSWORD=drill -e POSTGRES_DB=dirman postgres:17-alpine
docker exec -i dirman-restore-drill pg_restore -U postgres -d dirman --no-owner < /opt/dirman-assist/backups/dirman-<UTC_TIMESTAMP>.dump
docker exec dirman-restore-drill psql -U postgres -d dirman -c "\dt" -c "SELECT count(*) FROM tickets;" -c "SELECT count(*) FROM knowledge_documents;"
docker rm -f dirman-restore-drill

Note: knowledge_documents.content (bytea) and case_messages.media_bytes (bytea) are included inside PostgreSQL dumps; WAHA session keys live in Docker volume waha:/app/.sessions.


Terminal window
# Container status & health
docker compose -f compose.yaml -f compose.dashboard-public.yaml ps
# Loopback API health
curl -fsS http://127.0.0.1:18081/health
# Public HTTPS dashboard health
curl -fsS https://assist.p3mdi.id/health
# WAHA session status (using WAHA_API_KEY_PLAIN from .env)
docker compose exec -T worker python -c "import os,httpx; print(httpx.get('http://waha:3000/api/sessions/default', headers={'X-Api-Key': os.environ['WAHA_API_KEY_PLAIN']}).json())"
Symptom Likely Cause Diagnostic / Resolution
WhatsApp replies not sending (not_sent / unconfirmed) OUTBOUND_ENABLED is 'false' or WAHA default session is STOPPED/FAILED/SCAN_QR_CODE Check /health (outbound_enabled). Check WAHA session state; if logged out, re-run python3 scripts/pair.py privately and restart default session.
Bot responds with BUSY_REPLY (quota-busy fallback message) Google 429 cooldown active in provider_cooldowns, or local AI_RPM/AI_RPD reached, and Cloudflare disabled or on 3036 cooldown Inspect Konfigurasi AI > Penggunaan in dashboard or query provider_cooldowns in PG. Configure Cloudflare free-tier fallback via scripts/configure_cloudflare.py if not enabled.
PDF upload stuck in queued or failed pdf-worker down, PDF encrypted/corrupted, > 250 pages, > 32 MB, or OCR hit 1,800s timeout Check docker compose logs --tail=50 pdf-worker. Split oversized PDFs or decrypt before retrying via dashboard (Coba proses ulang).
Topic Graph shows Belum dikategorikan topic-worker missing ai_quota_limits, active chat traffic within 10m, or 12-batch/day cap hit Save verified RPM/TPM/RPD limits under Konfigurasi AI > Penggunaan so topic_categorization.quota_ready() permits idle batches.
Dashboard login returns 429 or 403 Exceeded 5 failed logins in 10m, or Origin/DASHBOARD_TRUSTED_PROXY_IPS mismatch Wait 10 minutes or verify DASHBOARD_ORIGIN and DASHBOARD_TRUSTED_PROXY_IPS match Traefik proxy IP.
Admin locked out of dashboard Forgot password (passwords are scrypt-hashed and non-retrievable) Another active admin generates a 30m single-use /reset#TOKEN link via Anggota & akses or Akun administrator.