Skip to content

System Architecture

Field Value
Document ID DA-ARCH-001
Pillar Design
Standard Alignment ISO/IEC/IEEE 42010 & IEEE 1016
Status Active / Synchronized with Codebase
Last Updated 2026-11-21
Source of Truth compose.yaml, compose.dashboard-public.yaml, app/Dockerfile, app/main.py, app/dashboard.py, app/worker.py, app/assistant.py, app/retrieval.py, app/documents.py, app/topic_categorization.py, app/broadcasts.py, app/analytics.py, app/cloudflare_ai.py, app/quota.py, app/message_batching.py, app/handoff.py

Dirman Assist is a self-hosted, PostgreSQL-backed WhatsApp helpdesk, management dashboard, ticket triage platform, and retrieval-augmented knowledge base built for P3MDI Sudirman.

The system prioritizes strict privilege separation, bounded resource usage, zero external vector/cache dependencies, and deterministic server-side enforcement of authorization, consent, citations, and outbound messaging.

  1. PostgreSQL as Single Source of State and Queues: All durable state—identities, cases, tickets, RAG indexes (tsvector + pg_trgm), PDF binary blobs, job queues, AI configuration revisions, rate-limit budgets, provider cooldowns, and telemetry—resides in a single PostgreSQL 17 instance (postgres).
  2. Privilege & Secret Isolation Across Containers:
    • dashboard (env_file: []) receives only DATABASE_URL and DASHBOARD_ORIGIN—never GOOGLE_API_KEY, CLOUDFLARE_API_TOKEN, WAHA_API_KEY_PLAIN, or WEBHOOK_SECRET.
    • pdf-worker (env_file: [], networks: [private]) has zero internet egress and zero AI/WAHA credentials; it runs Poppler and Tesseract in a resource-bounded local subprocess.
    • Only api, worker, and topic-worker hold AI provider credentials; only api and worker interact with waha.
  3. Loopback-Only Non-Dashboard Ports: api (127.0.0.1:18081:8080), dashboard (127.0.0.1:18082:8082), and waha (127.0.0.1:13000:3000) bind strictly to loopback on the host. Public ingress is routed exclusively to dashboard via Traefik HTTPS (compose.dashboard-public.yaml).
  4. Backend-Owned Decisions: LLMs never hold direct database handles, tool-calling access, or ticket/account privileges. The model returns a strictly schema-validated JSON decision object (scope, action, answer_basis, knowledge_ids, answer), which Python backend guards verify before executing any action or attaching verified citations.

2. Container Topology & Network Segmentation

Section titled “2. Container Topology & Network Segmentation”
flowchart LR
    WAUser["WhatsApp User"] <-->|"WhatsApp Protocol"| WAHA["WAHA (waha:3000 / WEBJS)"]
    AdminBrowser["Admin Browser"] -->|"HTTPS (assist.p3mdi.id)"| Traefik["Traefik Proxy (coolify network)"]
    AppHub["Sudirman App Hub"] -->|"Bearer Token (loopback 127.0.0.1:18081)"| API["FastAPI API (api:8080)"]

    subgraph DockerHost["Docker Compose Stack (dirman-assist)"]
        Traefik -->|"HTTP :8082"| Dashboard["Management Dashboard (dashboard:8082)"]
        WAHA -->|"POST /webhooks/waha (HMAC-SHA512)"| API
        API <-->|"Private PG"| PG[("PostgreSQL 17 (postgres:5432)")]
        Dashboard <-->|"Private PG (no AI/WAHA secrets)"| PG
        Dashboard -->|"GET http://api:8080/health"| API
        Worker["Conversational & Outbox Worker (app.worker)"] <-->|"Private PG"| PG
        Worker -->|"POST /api/sendText & GET chats"| WAHA
        PDFWorker["PDF Extraction Worker (app.documents)"] <-->|"Private PG (no egress)"| PG
        TopicWorker["Topic Categorizer (app.topic_categorization)"] <-->|"Private PG"| PG
    end

    Worker -->|"HTTPS generateContent"| GoogleAI["Google Gemini/Gemma API"]
    Worker -->|"HTTPS Fallback"| CF["Cloudflare Workers AI"]
    TopicWorker -->|"HTTPS Idle-Budgeted Calls"| GoogleAI
Network Driver / Flag Connected Services Purpose
private internal: true postgres, api, worker, dashboard, topic-worker, pdf-worker, waha Internal service mesh for PostgreSQL, WAHA webhook/API traffic, and dashboard-to-API health checks. Blocks all internet routing.
egress bridge (default) api, worker, dashboard, topic-worker, waha Outbound connectivity to WhatsApp servers (waha), Google AI Studio (worker, topic-worker), and Cloudflare Workers AI (worker). Note: pdf-worker and postgres are explicitly excluded from egress.
coolify external: true (in compose.dashboard-public.yaml) dashboard Attaches the public management dashboard to the host’s Traefik reverse proxy for TLS termination (https://assist.p3mdi.id).

All application containers share the hardened dirman-assist-app:local image (python:3.12-slim base with poppler-utils, tesseract-ocr, tesseract-ocr-ind, tesseract-ocr-eng, running as unprivileged UID 10001).

Service Entrypoint Host Binding mem_limit cpus pids_limit Hardening & Healthcheck
postgres ${POSTGRES_IMAGE} (postgres:17-alpine) None (private only) 512m 0.5 128 pg_isready -U dirman -d dirman (10s / 5s / 10 retries)
api uvicorn app.main:app --host 0.0.0.0 --port 8080 --no-access-log 127.0.0.1:18081:8080 512m 0.5 128 read_only: true, tmpfs: [/tmp], cap_drop: [ALL], no-new-privileges, HTTP /health (20s)
worker python -m app.worker None 512m 0.5 128 Same x-app hardening; waits on api: service_healthy
dashboard uvicorn app.dashboard:app --host 0.0.0.0 --port 8082 --no-access-log --proxy-headers 127.0.0.1:18082:8082 512m 0.5 128 Same x-app hardening; env_file: [] (DB + origin only); HTTP /health (20s)
topic-worker python -m app.topic_categorization None 192m 0.25 128 Same x-app hardening; isolated idle-only classifier
pdf-worker python -m app.documents None 640m 0.75 128 Same x-app hardening; env_file: []; networks: [private] only
waha ${WAHA_IMAGE} (devlikeapro/waha:latest) 127.0.0.1:13000:3000 2g 1.0 256 no-new-privileges, WEBJS engine, swagger/dashboard disabled, /api/sessions healthcheck (30s)

All services use the json-file Docker logging driver capped at max-size: 10m, max-file: 3.


4.1 WhatsApp Helpdesk & Grounded RAG Pipeline

Section titled “4.1 WhatsApp Helpdesk & Grounded RAG Pipeline”
sequenceDiagram
    autonumber
    participant User as WhatsApp User
    participant WAHA as WAHA (WEBJS)
    participant API as FastAPI API (app.main)
    participant PG as PostgreSQL
    participant Worker as Worker (app.worker)
    participant AI as Gemma / Cloudflare AI

    User->>WAHA: Send message / screenshot
    WAHA->>API: POST /webhooks/waha (x-webhook-hmac)
    API->>API: Verify HMAC-SHA512 & 300s timestamp window
    alt fromMe=true and source=app
        API->>PG: observe_native_event -> start_handoff & pause bot
    else fromMe=false
        API->>WAHA: resolve_sender (@lid -> @c.us via /api/default/lids)
        API->>PG: Auto-admit if PUBLIC_INTAKE=true, update_name, KB intake, consent check
        API->>PG: enqueue() -> jobs + support_cases + case_messages (ready_at +3s if batchable)
    end
    API-->>WAHA: 202 Accepted

    loop Worker Priority Poll
        Worker->>PG: Claim oldest ready job (advisory + SKIP LOCKED)
        Worker->>PG: message_batching.collect() (up to 6 msgs / 3s / 2000 chars)
        Worker->>PG: Check bot_mode (answer | ack | pause)
        alt bot_mode == answer
            Worker->>PG: Direct checks (KB intake receipt, consent, injection/unrelated, ticket direct action)
            Worker->>PG: retrieve() -> exact match or PG tsvector + pg_trgm (top 80 -> reranked top 5 <= 10k chars)
            alt Exact FAQ hit (not truncated)
                Worker-->>PG: Return deterministic FAQ answer + source
            else Model needed
                Worker->>AI: generate_with_fallback() (65% Google -> 35% Cloudflare if enabled)
                AI-->>Worker: JSON {scope, action, answer_basis, knowledge_ids, answer}
                Worker->>Worker: ticket_flow.decision() + retrieval.citations() URL/re-verification
            end
        end
        Worker->>PG: Save jobs(done), conversation_turns, case_messages, analytics_messages
        Worker->>WAHA: recent_native_reply() echo/takeover race check
        Worker->>WAHA: POST /api/sendText -> record waha_sent_messages + delivery telemetry
    end

Key operational stages:

  • Message Batching (app/message_batching.py): Ordinary WhatsApp text messages (<= 1,500 chars, excluding commands, ticket/intake/consent phrases, and screenshots) defer ready_at by WINDOW_SECONDS = 3. The worker merges up to MAX_MESSAGES = 6 consecutive queued messages from the same member and case (capped at 2,000 combined chars), marking earlier jobs with merged_into and case_messages.reply_state = 'grouped'.
  • PostgreSQL RAG (app/retrieval.py): Queries run purely inside PostgreSQL using normalized tokens (preserving 2-letter acronyms such as IB and WA), curated synonym expansion (GROUPS/CANON), weighted tsvector (search_vector), and pg_trgm trigram similarity (search_label, threshold 0.45). Up to 80 candidates are reranked locally in Python and packed into a 10,000-character JSON verified_evidence payload (max 5 items, with verbatim query-centered excerpts if oversized) plus a bounded 12-topic knowledge_catalog. See knowledge-retrieval.md and retrieval-acronyms-and-context.md.
  • Grounded JSON Contract & Citation Verification: The model receives responseSchema constraints enforcing scope (helpdesk | greeting | unrelated), action (none | create_ticket | offer_ticket | ticket_status), answer_basis (knowledge | general | insufficient), and knowledge_ids. When answer_basis == 'knowledge', retrieval.citations() re-queries PostgreSQL to verify that all cited IDs remain eligible, unexpired, and unmodified since retrieval, and verifies that any URLs in the generated answer are a subset of URLs present in the cited evidence.

4.2 Worker Scheduling Priorities & Background Daemons

Section titled “4.2 Worker Scheduling Priorities & Background Daemons”

The unified worker container (app/worker.py) processes tasks in strict deterministic priority order on every iteration, prioritizing interactive human replies and duty alerts ahead of AI work and bulk broadcasts:

flowchart TD
    Loop["Worker Main Loop"] --> Sweep{"60s elapsed?"}
    Sweep -->|"Yes"| CloseIdle["close_idle(): close non-handoff cases idle >= 24h"]
    Sweep -->|"No"| Hourly
    CloseIdle --> Hourly{"3600s elapsed?"}
    Hourly -->|"Yes"| Cleanup["Hourly TTL purge: cases 30d, turns 24h, analytics 180d, ai_usage 7d"]
    Hourly -->|"No"| P1
    Cleanup --> P1["1. send_staff_reply() — human handoff replies & notices"]
    P1 -->|"Idle"| P2["2. notify_admin() — on-duty admin alerts (5s shared gate)"]
    P2 -->|"Idle"| P3["3. run_one() — conversational helpdesk jobs"]
    P3 -->|"Idle"| P4["4. run_test() — AI config dry-run sandbox tests"]
    P4 -->|"Idle"| P5["5. broadcasts.run_one() — opt-in campaigns (09-18 WIB, 30s gap)"]
    P5 -->|"Idle"| Sleep["Sleep 1s"]
    Sleep --> Loop

4.3 Isolated PDF Ingestion Architecture (pdf-worker)

Section titled “4.3 Isolated PDF Ingestion Architecture (pdf-worker)”

PDF parsing is completely isolated from the API, conversational worker, and external networks (docs/pdf-knowledge.md):

  1. An authenticated administrator uploads a PDF (<= 32 MB, <= 250 pages; system cap 100 docs / 200 MB total) via POST /manage/api/documents.
  2. pdf-worker acquires session advisory lock 902722 (preventing concurrent workers and auto-releasing on crash), claims the oldest queued/processing document, writes the binary to a private TemporaryDirectory, and spawns python -m app.pdf_extract in a new process group (start_new_session=True).
  3. app/pdf_extract.py enforces OS resource.setrlimit caps (RLIMIT_AS = 384 MiB, RLIMIT_FSIZE = 32 MiB, RLIMIT_CPU = 210s), while the parent enforces an 1,800-second wall-clock deadline and 8 MiB stdout ceiling.
  4. Text extraction uses pdfinfo and batched pdftotext (25 pages/batch). Pages with fewer than 40 alphanumeric characters trigger pdftoppm (-scale-to 2000) and local tesseract (-l ind+eng --psm 3).
  5. Pages are split into <= 2,200-character searchable sections stored in faq with document_id, page_start, page_end, and locally extracted keyword aliases. OCR documents always enter review; non-OCR public documents with auto_publish=true publish automatically.

4.4 Background Content Topic Categorizer (topic-worker)

Section titled “4.4 Background Content Topic Categorizer (topic-worker)”

The topic-worker container (app/topic_categorization.py) classifies eligible KB entries into 1–3 content-based Indonesian topic labels (for Knowledge Base > Peta Pengetahuan) without competing with live chat (docs/content-topic-categorization.md):

  • Acquires session lock 902729 and sleeps 60 seconds between checks.
  • Idle-Only Guard: Defers whenever any job is queued/processing, any job was created in the last 10 minutes, any ai_config_tests job is queued, or a topic run occurred in the last 15 minutes.
  • Conservative Quota Gate (quota_ready): Requires operator-recorded ai_quota_limits (rpm, tpm, rpd updated within 30 days) and zero active cooldowns. Halts if daily runs reach min(12, int(rpd * 0.1)), if Google usage reaches 20% of RPD or RPM, or if estimated tokens would exceed 50% of TPM.
  • Sends sanitized titles and 650-char excerpts for up to 4 entries per batch; validates JSON IDs and rejects generic type labels (faq, agenda, announcement, etc.).

GET /manage/api/analytics (app/analytics.py) runs inside SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY. It derives ticket backlog, first-action and resolution percentiles (median and P90), WIB day/hour traffic heatmaps, privacy-preserving analytics_messages AI outcome telemetry, a local 7-day rolling MAE baseline forecast (gated at >= 28 complete days and >= 20 tickets), and what-if capacity planning simulations without calling any external AI API. See operational-analytics.md.


5. Resilience, Concurrency & Provider Fallback Design

Section titled “5. Resilience, Concurrency & Provider Fallback Design”

5.1 AI Provider Rolling & Cloudflare Fallback

Section titled “5.1 AI Provider Rolling & Cloudflare Fallback”
flowchart TD
    Req["AI Request (google_answer / extract_image_query)"] --> Guard{"Permit primary Google model?"}
    Guard -->|"Allowed"| Call1["POST Google generateContent"]
    Call1 -->|"200 OK"| Done["Parse & Validate JSON"]
    Call1 -->|"500/502/503/504 & remaining > 1s"| Retry1["Single retry with permit_request()"]
    Retry1 -->|"200 OK"| Done
    Retry1 -->|"Fail"| CFCheck
    Call1 -->|"429"| QuotaType{"model_quota_exhausted?"}
    QuotaType -->|"Project-wide 429"| CooldownProj["Record cooldown -> skip alternate Google model"]
    QuotaType -->|"Model-specific 429"| CooldownMod["Record model_specific cooldown -> try alternate Google model"]
    Guard -->|"Primary on model_specific cooldown"| CooldownMod
    CooldownMod --> AltCall["POST alternate Gemma model (26B <-> 31B)"]
    AltCall -->|"200 OK"| Done
    AltCall -->|"429 / Fail"| CFCheck
    CooldownProj --> CFCheck

    CFCheck{"Cloudflare configured & allow_roll?"}
    CFCheck -->|"Yes (35% timeout budget)"| CFCall["POST Cloudflare @cf/google/gemma-4-26b-a4b-it"]
    CFCall -->|"200 OK"| Done
    CFCall -->|"Error 3036 (daily cap)"| CFDayCooldown["Cooldown until next UTC midnight"]
    CFCall -->|"Fail"| StrongCheck
    CFCheck -->|"No"| StrongCheck

    StrongCheck{"Retrieved KB confidence == strong?"}
    StrongCheck -->|"Yes"| DetFallback["Return deterministic related guide + verified source"]
    StrongCheck -->|"No"| SafeFallback["Return BUSY_REPLY or fallback() + human handoff prompt"]
  • Timeout Budgets (app/assistant.py): 120s when thinking == 'high', 75s for screenshots, 30s for standard minimal-thinking text. When Cloudflare fallback is configured (CLOUDFLARE_AI_ENABLED=true and CLOUDFLARE_FREE_PLAN_CONFIRMED=true), Google receives 65% of the timeout budget so at least 35% remains for Cloudflare.
  • No Auto-Retry on External Sends: WhatsApp outbound deliveries (send_whatsapp, send_staff_reply, ticket_notifications.run_one, broadcasts.run_one) commit a sending claim to PostgreSQL before calling WAHA POST /api/sendText. If the worker crashes or WAHA times out, the record transitions to unconfirmed (not_sent) and is never automatically retried, preventing duplicate messages.

Concurrent API workers, background workers, and migrations serialize critical sections using explicit PostgreSQL advisory locks:

Lock ID Mode Module(s) Purpose
902711 pg_advisory_xact_lock app/main.py Serialize job enqueue deduplication and global queue cap (< 100 active jobs).
902712 pg_advisory_xact_lock app/quota.py Serialize per-minute/per-day AI quota budget checks and cooldown lookups.
902713 pg_advisory_xact_lock app/operator.py Serialize initial bootstrap-admin CLI creation.
902714 pg_advisory_xact_lock app/dashboard_auth.py Serialize dashboard login attempt counting, throttling (5 / 10 min), and session issuance.
902715 pg_advisory_xact_lock app/dashboard.py, app/management.py, app/admin_accounts.py Serialize admin profile/role/account mutations and prevent removing the last active admin.
902719 pg_advisory_xact_lock app/ai_management.py Serialize dry-run AI config test rate limits (5 / min / admin, max 5 queued).
902720 pg_advisory_xact_lock app/db.py Serialize idempotent schema.sql execution during multi-container startup.
902721 pg_advisory_xact_lock app/documents.py Serialize PDF quota checks (100 docs / 200 MB), publish/retire transitions, and retries.
902722 pg_try_advisory_lock app/documents.py Session-level single-active-PDF-worker guard across progress commits.
902723 pg_advisory_xact_lock app/knowledge_authority.py, app/knowledge_intake.py Serialize KB source authority graph cycle checks, addenda reviews, and auto-publication.
902725 pg_try_advisory_xact_lock app/broadcasts.py Non-blocking single-runner lock for scheduled broadcast dispatch.
902729 pg_try_advisory_lock app/topic_categorization.py Session-level single-instance lock for the topic-worker scheduler.