Operations & Deployment Runbook
| Field | Value |
|---|---|
| Document ID | DA-OPS-001 |
| Pillar | Software Engineering |
| Standard Alignment | ISO/IEC/IEEE 26514 / SRE |
| Status | Active / Synchronized with Codebase |
| Last Updated | 2026-11-21 |
| Source of Truth | compose.yaml, compose.dashboard-public.yaml, compose.test.yaml, .github/workflows/ci-deploy.yml, scripts/bootstrap.py, scripts/configure.py, scripts/configure_cloudflare.py, scripts/pair.py, scripts/backup.sh, scripts/deploy-server.sh, scripts/setup-cd.py, scripts/smoke.py, app/operator.py, app/dashboard_auth.py, docs/CI-CD.md, docs/dashboard-deployment.md |
1. Prerequisites & Resource Capacity
Section titled “1. Prerequisites & Resource Capacity”- Operating System & Runtime: Linux host (
/opt/dirman-assist) with Docker Engine, Docker Compose v2, Python 3,rsync,flock, andcurl. - Disk Space:
scripts/deploy-server.shenforces> 2 GiB(2,097,152 KiB) free disk space before staging a deployment. - Configured Container Resource Ceilings (
compose.yaml):
| Service | Memory Limit | CPU Limit | PID Limit | Persistent Volume |
|---|---|---|---|---|
postgres |
512m |
0.5 |
128 |
postgres:/var/lib/postgresql/data |
api |
512m |
0.5 |
128 |
Stateless (read_only: true, /tmp tmpfs) |
worker |
512m |
0.5 |
128 |
Stateless (read_only: true, /tmp tmpfs) |
dashboard |
512m |
0.5 |
128 |
Stateless (read_only: true, /tmp tmpfs) |
topic-worker |
192m |
0.25 |
128 |
Stateless (read_only: true, /tmp tmpfs) |
pdf-worker |
640m |
0.75 |
128 |
Stateless (read_only: true, /tmp tmpfs) |
waha |
2g |
1.0 |
256 |
waha:/app/.sessions |
| Stack Total | 4.87 GiB |
4.0 vCPU |
— | 2 named Docker volumes |
2. Initial Server Bootstrap & Onboarding
Section titled “2. Initial Server Bootstrap & Onboarding”Run from /opt/dirman-assist on the host:
Step 1: Bootstrap Protected .env & Build Image
Section titled “Step 1: Bootstrap Protected .env & Build Image”python3 scripts/bootstrap.py- Pulls
devlikeapro/waha:latest,postgres:17-alpine, andpython:3.12-slimand pins each to its immutableRepoDigests[0]. - Generates random 32-byte hex credentials (
POSTGRES_PASSWORD,WAHA_API_KEY+WAHA_API_KEY_PLAIN,WEBHOOK_SECRET), writes.envunderumask 0o077(chmod 0600) withOUTBOUND_ENABLED=falseandPUBLIC_INTAKE=false, validates Compose config, and buildsdirman-assist-app:local.
Step 2: Configure AI Provider Keys
Section titled “Step 2: Configure AI Provider Keys”python3 scripts/configure.py- Interactively prompts for
GOOGLE_API_KEY(hidden input), atomically updates.env, and recreatesapiandworker. - Optional free-tier Cloudflare Workers AI fallback (
@cf/google/gemma-4-26b-a4b-it):
python3 scripts/configure_cloudflare.pyRequires typing FREE to confirm a Workers Free account, runs a live JSON smoke test inside worker, writes CLOUDFLARE_* variables atomically to .env, restarts api/worker, and sets ai_provider_status (provider='cloudflare').
Step 3: Start Services & Provision Initial Admin Accounts
Section titled “Step 3: Start Services & Provision Initial Admin Accounts”docker compose -f compose.yaml -f compose.dashboard-public.yaml up -dProvision the initial administrator member/API token (app/operator.py) and dashboard login (app/dashboard_auth.py) inside private containers (umask 0o077, output files never overwritten):
docker compose exec -T api python -m app.operator bootstrap-admin --name "Admin Name" --phone 628XXXXXXXXXX --output /tmp/admin-tokendocker compose exec -T dashboard python -m app.dashboard_auth --username adminuser --member-id <MEMBER_UUID> --output /tmp/dashboard-login.txtRetrieve the generated credentials privately and delete temporary credential files after onboarding.
Step 4: Link WhatsApp Session (scripts/pair.py)
Section titled “Step 4: Link WhatsApp Session (scripts/pair.py)”python3 scripts/pair.py- Prompts for
PAIRconfirmation, starts WAHAdefaultsession via loopbackhttp://127.0.0.1:13000, polls for QR PNG, and savessecrets/whatsapp-qr.png(0o600). - Scan via WhatsApp -> Linked Devices, delete
secrets/whatsapp-qr.png, and only then setOUTBOUND_ENABLED=true(andPUBLIC_INTAKE=trueif admitting unknown WhatsApp numbers) in.envand recreateapiandworker.
3. Complete Environment Variable Catalog
Section titled “3. Complete Environment Variable Catalog”All secrets belong exclusively in protected server .env (0o600). Never commit .env or print credentials.
| Variable | Services | Default / Format | Purpose |
|---|---|---|---|
POSTGRES_IMAGE |
postgres |
Pinned postgres:17-alpine@sha256:<digest> |
Immutable PostgreSQL image digest |
PYTHON_IMAGE |
Build arg (x-app) |
Pinned python:3.12-slim@sha256:<digest> |
Immutable Python base image digest |
WAHA_IMAGE |
waha |
Pinned devlikeapro/waha:latest@sha256:<digest> |
Immutable WAHA WEBJS image digest |
POSTGRES_PASSWORD |
postgres |
64-char hex | PostgreSQL password for user dirman |
DATABASE_URL |
x-app, dashboard, pdf-worker |
postgresql://dirman:<pass>@postgres:5432/dirman |
Primary DB DSN |
WAHA_API_KEY |
waha |
sha512:<128-hex> |
WAHA server hashed API key |
WAHA_API_KEY_PLAIN |
api, worker, waha |
64-char hex | Plain WAHA API key sent in X-Api-Key header |
WEBHOOK_SECRET |
api, waha |
64-char hex | WHATSAPP_HOOK_HMAC_KEY for x-webhook-hmac SHA-512 |
GOOGLE_API_KEY |
api, worker, topic-worker |
AI Studio key (AIza<token>) |
Primary Gemma API key (empty disables AI calls) |
GOOGLE_MODEL |
.env (bootstrap seed) |
gemma-4-26b-a4b-it |
Initial seed reference (runtime uses ai_configuration DB table) |
AI_RPM |
worker, topic-worker |
0 (disabled) |
Optional local cap on Google requests per UTC minute |
AI_RPD |
worker, topic-worker |
0 (disabled) |
Optional local cap on Google requests per Pacific day |
CLOUDFLARE_AI_ENABLED |
worker |
'true' or unset |
Enables Cloudflare Workers AI fallback |
CLOUDFLARE_FREE_PLAN_CONFIRMED |
worker |
'true' or unset |
Explicit confirmation of Workers Free allowance |
CLOUDFLARE_ACCOUNT_ID |
worker |
32-char hex | Cloudflare account identifier |
CLOUDFLARE_API_TOKEN |
worker |
Workers AI Read/Edit token | Bearer token for Cloudflare AI fallback |
OUTBOUND_ENABLED |
api, worker |
'false' ('true' in prod) |
Master switch for all outbound WAHA messages |
PUBLIC_INTAKE |
api |
'false' ('true' to auto-register) |
Admits unknown WhatsApp senders as member (placeholder) |
DASHBOARD_ORIGIN |
dashboard |
https://assist.p3mdi.id |
Exact allowed Origin for Same-Origin & reset URLs |
DASHBOARD_TRUSTED_PROXY_IPS |
dashboard |
127.0.0.1 (set Traefik IP in prod) |
Trusted reverse-proxy IP(s) for X-Forwarded-For throttling |
TEST_DATABASE |
compose.test.yaml only |
'true' |
Guard flag required by test harness; never set in production |
4. Compose Overlays & Traefik Public Exposure
Section titled “4. Compose Overlays & Traefik Public Exposure”- Base Stack (
compose.yaml): Bindsapi(127.0.0.1:18081:8080),dashboard(127.0.0.1:18082:8082), andwaha(127.0.0.1:13000:3000) exclusively to loopback withtraefik.enable: 'false'on all services. - Public Dashboard Overlay (
compose.dashboard-public.yaml):- Attaches
dashboardalone to the externalcoolifyDocker network (traefik.docker.network: coolify). - Enables Traefik router
dirman-dashboard-http(Host(\assist.p3mdi.id`), permanent HTTP -> HTTPS redirect) anddirman-dashboard(https, TLS certresolverletsencrypt, backend port8082`). - Always invoke production Compose commands with both files:
Terminal window docker compose -p dirman-assist -f compose.yaml -f compose.dashboard-public.yaml ps - Ensure
DASHBOARD_TRUSTED_PROXY_IPSin.envincludes the Traefik container IP on thecoolifynetwork so login throttling (dashboard_login_attempts) hashes real client IPs rather than proxy IPs (docs/dashboard-deployment.md).
- Attaches
5. Automated CI/CD & Manual Deployment Runbook
Section titled “5. Automated CI/CD & Manual Deployment Runbook”5.1 One-Time CD Controller Setup (scripts/setup-cd.py)
Section titled “5.1 One-Time CD Controller Setup (scripts/setup-cd.py)”Run from an operator workstation with root SSH and authenticated gh CLI:
- Verifies the server’s ED25519 SSH host key fingerprint against the pinned constant before installing
/usr/local/sbin/dirman-assist-deploy(scripts/deploy-server.sh). - Generates
/etc/dirman-assist-cd/actions_key, appendsrestrict,command="/usr/local/sbin/dirman-assist-deploy"to/root/.ssh/authorized_keys, initializes bare mirror/var/lib/dirman-assist-cd/repository.git, and populates GitHub secretsDEPLOY_HOST,DEPLOY_KNOWN_HOSTS, andDEPLOY_SSH_KEY.
5.2 Automated Deployment Flow (scripts/deploy-server.sh)
Section titled “5.2 Automated Deployment Flow (scripts/deploy-server.sh)”flowchart TD
GHA["GitHub Actions (main verify passed)"] -->|"SSH forced command: deploy SHA"| Lock["flock -w 900 /var/lock/dirman-assist-deploy.lock"]
Lock --> Fetch["git fetch origin & verify SHA == origin/main"]
Fetch --> Disk["Check > 2 GiB free disk & extract release"]
Disk --> Build["docker build -t dirman-assist-app:SHA"]
Build --> Backup["Snapshot previous images + source.tgz + pg_dump database.dump"]
Backup --> Sync["rsync app/ scripts/ docs/ tests/ & compose files (.env & secrets untouched)"]
Sync --> Migrate["compose run --rm api migrate()"]
Migrate --> Up["compose up -d --no-deps --wait (api, worker, dashboard, pdf-worker, topic-worker)"]
Up --> Health["curl -fsS $DASHBOARD_ORIGIN/health (10 retries)"]
Health -->|"Success"| Stamp["Write deployed-commit"]
Health -->|"Failure (ERR trap)"| Rollback["Restore source.tgz + previous images via rollback.json (DB preserved)"]
- Zero-Touch Stateful Services:
postgresandwahaare never restarted bydeploy-server.sh, preserving WAHA sessions and PostgreSQL connections. - Standard Rollback: Revert the offending commit on
mainand push so CI builds and deploys the verified revert. For emergency server-side rollback, notedeploy-server.shautomatically restoressource.tgzand prior per-service images on failed healthcheck (database.dumpis preserved without auto-restore to avoid losing user messages).
6. Backup, Restore Drill & Verification
Section titled “6. Backup, Restore Drill & Verification”6.1 Database Backup (scripts/backup.sh)
Section titled “6.1 Database Backup (scripts/backup.sh)”sh /opt/dirman-assist/scripts/backup.shWrites a custom-format PostgreSQL dump (pg_dump -U dirman -d dirman -Fc) with umask 077 to /opt/dirman-assist/backups/dirman-<UTC_TIMESTAMP>.dump. Automated deployment backups are stored under /var/lib/dirman-assist-cd/deployments/<TIMESTAMP>-<SHA>/database.dump.
6.2 Restore Verification Drill (Non-Destructive)
Section titled “6.2 Restore Verification Drill (Non-Destructive)”Verify backup integrity periodically against a disposable PostgreSQL container (never overwrite live production blindly):
docker run --rm -d --name dirman-restore-drill -e POSTGRES_PASSWORD=drill -e POSTGRES_DB=dirman postgres:17-alpinedocker exec -i dirman-restore-drill pg_restore -U postgres -d dirman --no-owner < /opt/dirman-assist/backups/dirman-<UTC_TIMESTAMP>.dumpdocker exec dirman-restore-drill psql -U postgres -d dirman -c "\dt" -c "SELECT count(*) FROM tickets;" -c "SELECT count(*) FROM knowledge_documents;"docker rm -f dirman-restore-drillNote: knowledge_documents.content (bytea) and case_messages.media_bytes (bytea) are included inside PostgreSQL dumps; WAHA session keys live in Docker volume waha:/app/.sessions.
7. Health Checks & Troubleshooting
Section titled “7. Health Checks & Troubleshooting”Health & Smoke Commands
Section titled “Health & Smoke Commands”# Container status & healthdocker compose -f compose.yaml -f compose.dashboard-public.yaml ps
# Loopback API healthcurl -fsS http://127.0.0.1:18081/health
# Public HTTPS dashboard healthcurl -fsS https://assist.p3mdi.id/health
# WAHA session status (using WAHA_API_KEY_PLAIN from .env)docker compose exec -T worker python -c "import os,httpx; print(httpx.get('http://waha:3000/api/sessions/default', headers={'X-Api-Key': os.environ['WAHA_API_KEY_PLAIN']}).json())"Common Operational Issues & Remediation
Section titled “Common Operational Issues & Remediation”| Symptom | Likely Cause | Diagnostic / Resolution |
|---|---|---|
WhatsApp replies not sending (not_sent / unconfirmed) |
OUTBOUND_ENABLED is 'false' or WAHA default session is STOPPED/FAILED/SCAN_QR_CODE |
Check /health (outbound_enabled). Check WAHA session state; if logged out, re-run python3 scripts/pair.py privately and restart default session. |
Bot responds with BUSY_REPLY (quota-busy fallback message) |
Google 429 cooldown active in provider_cooldowns, or local AI_RPM/AI_RPD reached, and Cloudflare disabled or on 3036 cooldown |
Inspect Konfigurasi AI > Penggunaan in dashboard or query provider_cooldowns in PG. Configure Cloudflare free-tier fallback via scripts/configure_cloudflare.py if not enabled. |
PDF upload stuck in queued or failed |
pdf-worker down, PDF encrypted/corrupted, > 250 pages, > 32 MB, or OCR hit 1,800s timeout |
Check docker compose logs --tail=50 pdf-worker. Split oversized PDFs or decrypt before retrying via dashboard (Coba proses ulang). |
Topic Graph shows Belum dikategorikan |
topic-worker missing ai_quota_limits, active chat traffic within 10m, or 12-batch/day cap hit |
Save verified RPM/TPM/RPD limits under Konfigurasi AI > Penggunaan so topic_categorization.quota_ready() permits idle batches. |
Dashboard login returns 429 or 403 |
Exceeded 5 failed logins in 10m, or Origin/DASHBOARD_TRUSTED_PROXY_IPS mismatch |
Wait 10 minutes or verify DASHBOARD_ORIGIN and DASHBOARD_TRUSTED_PROXY_IPS match Traefik proxy IP. |
| Admin locked out of dashboard | Forgot password (passwords are scrypt-hashed and non-retrievable) |
Another active admin generates a 30m single-use /reset#TOKEN link via Anggota & akses or Akun administrator. |
8. Related Documentation
Section titled “8. Related Documentation”- ARCHITECTURE.md — Service topology and network rules.
- SECURITY_AND_PRIVACY.md — Hardening, secrets handling, and WAHA/AI risk disclosures.
- ENGINEERING_AND_SDLC.md — Verification pyramid and ADRs.
- CI-CD.md — Detailed GitHub Actions & SSH deployment specification.
- dashboard-deployment.md — Historical Traefik & DNS activation reference.
- TESTING.md — Running isolated verification tests safely.