System Architecture
| Field | Value |
|---|---|
| Document ID | DA-ARCH-001 |
| Pillar | Design |
| Standard Alignment | ISO/IEC/IEEE 42010 & IEEE 1016 |
| Status | Active / Synchronized with Codebase |
| Last Updated | 2026-11-21 |
| Source of Truth | compose.yaml, compose.dashboard-public.yaml, app/Dockerfile, app/main.py, app/dashboard.py, app/worker.py, app/assistant.py, app/retrieval.py, app/documents.py, app/topic_categorization.py, app/broadcasts.py, app/analytics.py, app/cloudflare_ai.py, app/quota.py, app/message_batching.py, app/handoff.py |
1. Architectural Overview
Section titled “1. Architectural Overview”Dirman Assist is a self-hosted, PostgreSQL-backed WhatsApp helpdesk, management dashboard, ticket triage platform, and retrieval-augmented knowledge base built for P3MDI Sudirman.
The system prioritizes strict privilege separation, bounded resource usage, zero external vector/cache dependencies, and deterministic server-side enforcement of authorization, consent, citations, and outbound messaging.
Core Architectural Principles
Section titled “Core Architectural Principles”- PostgreSQL as Single Source of State and Queues: All durable state—identities, cases, tickets, RAG indexes (
tsvector+pg_trgm), PDF binary blobs, job queues, AI configuration revisions, rate-limit budgets, provider cooldowns, and telemetry—resides in a single PostgreSQL 17 instance (postgres). - Privilege & Secret Isolation Across Containers:
dashboard(env_file: []) receives onlyDATABASE_URLandDASHBOARD_ORIGIN—neverGOOGLE_API_KEY,CLOUDFLARE_API_TOKEN,WAHA_API_KEY_PLAIN, orWEBHOOK_SECRET.pdf-worker(env_file: [],networks: [private]) has zero internet egress and zero AI/WAHA credentials; it runs Poppler and Tesseract in a resource-bounded local subprocess.- Only
api,worker, andtopic-workerhold AI provider credentials; onlyapiandworkerinteract withwaha.
- Loopback-Only Non-Dashboard Ports:
api(127.0.0.1:18081:8080),dashboard(127.0.0.1:18082:8082), andwaha(127.0.0.1:13000:3000) bind strictly to loopback on the host. Public ingress is routed exclusively todashboardvia Traefik HTTPS (compose.dashboard-public.yaml). - Backend-Owned Decisions: LLMs never hold direct database handles, tool-calling access, or ticket/account privileges. The model returns a strictly schema-validated JSON decision object (
scope,action,answer_basis,knowledge_ids,answer), which Python backend guards verify before executing any action or attaching verified citations.
2. Container Topology & Network Segmentation
Section titled “2. Container Topology & Network Segmentation”flowchart LR
WAUser["WhatsApp User"] <-->|"WhatsApp Protocol"| WAHA["WAHA (waha:3000 / WEBJS)"]
AdminBrowser["Admin Browser"] -->|"HTTPS (assist.p3mdi.id)"| Traefik["Traefik Proxy (coolify network)"]
AppHub["Sudirman App Hub"] -->|"Bearer Token (loopback 127.0.0.1:18081)"| API["FastAPI API (api:8080)"]
subgraph DockerHost["Docker Compose Stack (dirman-assist)"]
Traefik -->|"HTTP :8082"| Dashboard["Management Dashboard (dashboard:8082)"]
WAHA -->|"POST /webhooks/waha (HMAC-SHA512)"| API
API <-->|"Private PG"| PG[("PostgreSQL 17 (postgres:5432)")]
Dashboard <-->|"Private PG (no AI/WAHA secrets)"| PG
Dashboard -->|"GET http://api:8080/health"| API
Worker["Conversational & Outbox Worker (app.worker)"] <-->|"Private PG"| PG
Worker -->|"POST /api/sendText & GET chats"| WAHA
PDFWorker["PDF Extraction Worker (app.documents)"] <-->|"Private PG (no egress)"| PG
TopicWorker["Topic Categorizer (app.topic_categorization)"] <-->|"Private PG"| PG
end
Worker -->|"HTTPS generateContent"| GoogleAI["Google Gemini/Gemma API"]
Worker -->|"HTTPS Fallback"| CF["Cloudflare Workers AI"]
TopicWorker -->|"HTTPS Idle-Budgeted Calls"| GoogleAI
Network Zones
Section titled “Network Zones”| Network | Driver / Flag | Connected Services | Purpose |
|---|---|---|---|
private |
internal: true |
postgres, api, worker, dashboard, topic-worker, pdf-worker, waha |
Internal service mesh for PostgreSQL, WAHA webhook/API traffic, and dashboard-to-API health checks. Blocks all internet routing. |
egress |
bridge (default) | api, worker, dashboard, topic-worker, waha |
Outbound connectivity to WhatsApp servers (waha), Google AI Studio (worker, topic-worker), and Cloudflare Workers AI (worker). Note: pdf-worker and postgres are explicitly excluded from egress. |
coolify |
external: true (in compose.dashboard-public.yaml) |
dashboard |
Attaches the public management dashboard to the host’s Traefik reverse proxy for TLS termination (https://assist.p3mdi.id). |
3. Service Inventory & Resource Budgets
Section titled “3. Service Inventory & Resource Budgets”All application containers share the hardened dirman-assist-app:local image (python:3.12-slim base with poppler-utils, tesseract-ocr, tesseract-ocr-ind, tesseract-ocr-eng, running as unprivileged UID 10001).
| Service | Entrypoint | Host Binding | mem_limit |
cpus |
pids_limit |
Hardening & Healthcheck |
|---|---|---|---|---|---|---|
postgres |
${POSTGRES_IMAGE} (postgres:17-alpine) |
None (private only) |
512m |
0.5 |
128 |
pg_isready -U dirman -d dirman (10s / 5s / 10 retries) |
api |
uvicorn app.main:app --host 0.0.0.0 --port 8080 --no-access-log |
127.0.0.1:18081:8080 |
512m |
0.5 |
128 |
read_only: true, tmpfs: [/tmp], cap_drop: [ALL], no-new-privileges, HTTP /health (20s) |
worker |
python -m app.worker |
None | 512m |
0.5 |
128 |
Same x-app hardening; waits on api: service_healthy |
dashboard |
uvicorn app.dashboard:app --host 0.0.0.0 --port 8082 --no-access-log --proxy-headers |
127.0.0.1:18082:8082 |
512m |
0.5 |
128 |
Same x-app hardening; env_file: [] (DB + origin only); HTTP /health (20s) |
topic-worker |
python -m app.topic_categorization |
None | 192m |
0.25 |
128 |
Same x-app hardening; isolated idle-only classifier |
pdf-worker |
python -m app.documents |
None | 640m |
0.75 |
128 |
Same x-app hardening; env_file: []; networks: [private] only |
waha |
${WAHA_IMAGE} (devlikeapro/waha:latest) |
127.0.0.1:13000:3000 |
2g |
1.0 |
256 |
no-new-privileges, WEBJS engine, swagger/dashboard disabled, /api/sessions healthcheck (30s) |
All services use the json-file Docker logging driver capped at max-size: 10m, max-file: 3.
4. End-to-End Runtime Workflows
Section titled “4. End-to-End Runtime Workflows”4.1 WhatsApp Helpdesk & Grounded RAG Pipeline
Section titled “4.1 WhatsApp Helpdesk & Grounded RAG Pipeline”sequenceDiagram
autonumber
participant User as WhatsApp User
participant WAHA as WAHA (WEBJS)
participant API as FastAPI API (app.main)
participant PG as PostgreSQL
participant Worker as Worker (app.worker)
participant AI as Gemma / Cloudflare AI
User->>WAHA: Send message / screenshot
WAHA->>API: POST /webhooks/waha (x-webhook-hmac)
API->>API: Verify HMAC-SHA512 & 300s timestamp window
alt fromMe=true and source=app
API->>PG: observe_native_event -> start_handoff & pause bot
else fromMe=false
API->>WAHA: resolve_sender (@lid -> @c.us via /api/default/lids)
API->>PG: Auto-admit if PUBLIC_INTAKE=true, update_name, KB intake, consent check
API->>PG: enqueue() -> jobs + support_cases + case_messages (ready_at +3s if batchable)
end
API-->>WAHA: 202 Accepted
loop Worker Priority Poll
Worker->>PG: Claim oldest ready job (advisory + SKIP LOCKED)
Worker->>PG: message_batching.collect() (up to 6 msgs / 3s / 2000 chars)
Worker->>PG: Check bot_mode (answer | ack | pause)
alt bot_mode == answer
Worker->>PG: Direct checks (KB intake receipt, consent, injection/unrelated, ticket direct action)
Worker->>PG: retrieve() -> exact match or PG tsvector + pg_trgm (top 80 -> reranked top 5 <= 10k chars)
alt Exact FAQ hit (not truncated)
Worker-->>PG: Return deterministic FAQ answer + source
else Model needed
Worker->>AI: generate_with_fallback() (65% Google -> 35% Cloudflare if enabled)
AI-->>Worker: JSON {scope, action, answer_basis, knowledge_ids, answer}
Worker->>Worker: ticket_flow.decision() + retrieval.citations() URL/re-verification
end
end
Worker->>PG: Save jobs(done), conversation_turns, case_messages, analytics_messages
Worker->>WAHA: recent_native_reply() echo/takeover race check
Worker->>WAHA: POST /api/sendText -> record waha_sent_messages + delivery telemetry
end
Key operational stages:
- Message Batching (
app/message_batching.py): Ordinary WhatsApp text messages (<= 1,500chars, excluding commands, ticket/intake/consent phrases, and screenshots) deferready_atbyWINDOW_SECONDS = 3. The worker merges up toMAX_MESSAGES = 6consecutive queued messages from the same member and case (capped at 2,000 combined chars), marking earlier jobs withmerged_intoandcase_messages.reply_state = 'grouped'. - PostgreSQL RAG (
app/retrieval.py): Queries run purely inside PostgreSQL using normalized tokens (preserving 2-letter acronyms such asIBandWA), curated synonym expansion (GROUPS/CANON), weightedtsvector(search_vector), andpg_trgmtrigram similarity (search_label, threshold0.45). Up to 80 candidates are reranked locally in Python and packed into a 10,000-character JSONverified_evidencepayload (max 5 items, with verbatim query-centered excerpts if oversized) plus a bounded 12-topicknowledge_catalog. See knowledge-retrieval.md and retrieval-acronyms-and-context.md. - Grounded JSON Contract & Citation Verification: The model receives
responseSchemaconstraints enforcingscope(helpdesk | greeting | unrelated),action(none | create_ticket | offer_ticket | ticket_status),answer_basis(knowledge | general | insufficient), andknowledge_ids. Whenanswer_basis == 'knowledge',retrieval.citations()re-queries PostgreSQL to verify that all cited IDs remain eligible, unexpired, and unmodified since retrieval, and verifies that any URLs in the generated answer are a subset of URLs present in the cited evidence.
4.2 Worker Scheduling Priorities & Background Daemons
Section titled “4.2 Worker Scheduling Priorities & Background Daemons”The unified worker container (app/worker.py) processes tasks in strict deterministic priority order on every iteration, prioritizing interactive human replies and duty alerts ahead of AI work and bulk broadcasts:
flowchart TD
Loop["Worker Main Loop"] --> Sweep{"60s elapsed?"}
Sweep -->|"Yes"| CloseIdle["close_idle(): close non-handoff cases idle >= 24h"]
Sweep -->|"No"| Hourly
CloseIdle --> Hourly{"3600s elapsed?"}
Hourly -->|"Yes"| Cleanup["Hourly TTL purge: cases 30d, turns 24h, analytics 180d, ai_usage 7d"]
Hourly -->|"No"| P1
Cleanup --> P1["1. send_staff_reply() — human handoff replies & notices"]
P1 -->|"Idle"| P2["2. notify_admin() — on-duty admin alerts (5s shared gate)"]
P2 -->|"Idle"| P3["3. run_one() — conversational helpdesk jobs"]
P3 -->|"Idle"| P4["4. run_test() — AI config dry-run sandbox tests"]
P4 -->|"Idle"| P5["5. broadcasts.run_one() — opt-in campaigns (09-18 WIB, 30s gap)"]
P5 -->|"Idle"| Sleep["Sleep 1s"]
Sleep --> Loop
4.3 Isolated PDF Ingestion Architecture (pdf-worker)
Section titled “4.3 Isolated PDF Ingestion Architecture (pdf-worker)”PDF parsing is completely isolated from the API, conversational worker, and external networks (docs/pdf-knowledge.md):
- An authenticated administrator uploads a PDF (
<= 32 MB,<= 250pages; system cap100docs /200 MBtotal) viaPOST /manage/api/documents. pdf-workeracquires session advisory lock902722(preventing concurrent workers and auto-releasing on crash), claims the oldestqueued/processingdocument, writes the binary to a privateTemporaryDirectory, and spawnspython -m app.pdf_extractin a new process group (start_new_session=True).app/pdf_extract.pyenforces OSresource.setrlimitcaps (RLIMIT_AS = 384 MiB,RLIMIT_FSIZE = 32 MiB,RLIMIT_CPU = 210s), while the parent enforces an 1,800-second wall-clock deadline and 8 MiB stdout ceiling.- Text extraction uses
pdfinfoand batchedpdftotext(25 pages/batch). Pages with fewer than 40 alphanumeric characters triggerpdftoppm(-scale-to 2000) and localtesseract(-l ind+eng --psm 3). - Pages are split into
<= 2,200-character searchable sections stored infaqwithdocument_id,page_start,page_end, and locally extracted keywordaliases. OCR documents always enterreview; non-OCR public documents withauto_publish=truepublish automatically.
4.4 Background Content Topic Categorizer (topic-worker)
Section titled “4.4 Background Content Topic Categorizer (topic-worker)”The topic-worker container (app/topic_categorization.py) classifies eligible KB entries into 1–3 content-based Indonesian topic labels (for Knowledge Base > Peta Pengetahuan) without competing with live chat (docs/content-topic-categorization.md):
- Acquires session lock
902729and sleeps 60 seconds between checks. - Idle-Only Guard: Defers whenever any job is
queued/processing, any job was created in the last 10 minutes, anyai_config_testsjob isqueued, or a topic run occurred in the last 15 minutes. - Conservative Quota Gate (
quota_ready): Requires operator-recordedai_quota_limits(rpm,tpm,rpdupdated within 30 days) and zero active cooldowns. Halts if daily runs reachmin(12, int(rpd * 0.1)), if Google usage reaches 20% of RPD or RPM, or if estimated tokens would exceed 50% of TPM. - Sends sanitized titles and 650-char excerpts for up to 4 entries per batch; validates JSON IDs and rejects generic type labels (
faq,agenda,announcement, etc.).
4.5 Repeatable-Read Operational Analytics
Section titled “4.5 Repeatable-Read Operational Analytics”GET /manage/api/analytics (app/analytics.py) runs inside SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY. It derives ticket backlog, first-action and resolution percentiles (median and P90), WIB day/hour traffic heatmaps, privacy-preserving analytics_messages AI outcome telemetry, a local 7-day rolling MAE baseline forecast (gated at >= 28 complete days and >= 20 tickets), and what-if capacity planning simulations without calling any external AI API. See operational-analytics.md.
5. Resilience, Concurrency & Provider Fallback Design
Section titled “5. Resilience, Concurrency & Provider Fallback Design”5.1 AI Provider Rolling & Cloudflare Fallback
Section titled “5.1 AI Provider Rolling & Cloudflare Fallback”flowchart TD
Req["AI Request (google_answer / extract_image_query)"] --> Guard{"Permit primary Google model?"}
Guard -->|"Allowed"| Call1["POST Google generateContent"]
Call1 -->|"200 OK"| Done["Parse & Validate JSON"]
Call1 -->|"500/502/503/504 & remaining > 1s"| Retry1["Single retry with permit_request()"]
Retry1 -->|"200 OK"| Done
Retry1 -->|"Fail"| CFCheck
Call1 -->|"429"| QuotaType{"model_quota_exhausted?"}
QuotaType -->|"Project-wide 429"| CooldownProj["Record cooldown -> skip alternate Google model"]
QuotaType -->|"Model-specific 429"| CooldownMod["Record model_specific cooldown -> try alternate Google model"]
Guard -->|"Primary on model_specific cooldown"| CooldownMod
CooldownMod --> AltCall["POST alternate Gemma model (26B <-> 31B)"]
AltCall -->|"200 OK"| Done
AltCall -->|"429 / Fail"| CFCheck
CooldownProj --> CFCheck
CFCheck{"Cloudflare configured & allow_roll?"}
CFCheck -->|"Yes (35% timeout budget)"| CFCall["POST Cloudflare @cf/google/gemma-4-26b-a4b-it"]
CFCall -->|"200 OK"| Done
CFCall -->|"Error 3036 (daily cap)"| CFDayCooldown["Cooldown until next UTC midnight"]
CFCall -->|"Fail"| StrongCheck
CFCheck -->|"No"| StrongCheck
StrongCheck{"Retrieved KB confidence == strong?"}
StrongCheck -->|"Yes"| DetFallback["Return deterministic related guide + verified source"]
StrongCheck -->|"No"| SafeFallback["Return BUSY_REPLY or fallback() + human handoff prompt"]
- Timeout Budgets (
app/assistant.py):120swhenthinking == 'high',75sfor screenshots,30sfor standard minimal-thinking text. When Cloudflare fallback is configured (CLOUDFLARE_AI_ENABLED=trueandCLOUDFLARE_FREE_PLAN_CONFIRMED=true), Google receives65%of the timeout budget so at least35%remains for Cloudflare. - No Auto-Retry on External Sends: WhatsApp outbound deliveries (
send_whatsapp,send_staff_reply,ticket_notifications.run_one,broadcasts.run_one) commit asendingclaim to PostgreSQL before calling WAHAPOST /api/sendText. If the worker crashes or WAHA times out, the record transitions tounconfirmed(not_sent) and is never automatically retried, preventing duplicate messages.
5.2 PostgreSQL Advisory Lock Allocation
Section titled “5.2 PostgreSQL Advisory Lock Allocation”Concurrent API workers, background workers, and migrations serialize critical sections using explicit PostgreSQL advisory locks:
| Lock ID | Mode | Module(s) | Purpose |
|---|---|---|---|
902711 |
pg_advisory_xact_lock |
app/main.py |
Serialize job enqueue deduplication and global queue cap (< 100 active jobs). |
902712 |
pg_advisory_xact_lock |
app/quota.py |
Serialize per-minute/per-day AI quota budget checks and cooldown lookups. |
902713 |
pg_advisory_xact_lock |
app/operator.py |
Serialize initial bootstrap-admin CLI creation. |
902714 |
pg_advisory_xact_lock |
app/dashboard_auth.py |
Serialize dashboard login attempt counting, throttling (5 / 10 min), and session issuance. |
902715 |
pg_advisory_xact_lock |
app/dashboard.py, app/management.py, app/admin_accounts.py |
Serialize admin profile/role/account mutations and prevent removing the last active admin. |
902719 |
pg_advisory_xact_lock |
app/ai_management.py |
Serialize dry-run AI config test rate limits (5 / min / admin, max 5 queued). |
902720 |
pg_advisory_xact_lock |
app/db.py |
Serialize idempotent schema.sql execution during multi-container startup. |
902721 |
pg_advisory_xact_lock |
app/documents.py |
Serialize PDF quota checks (100 docs / 200 MB), publish/retire transitions, and retries. |
902722 |
pg_try_advisory_lock |
app/documents.py |
Session-level single-active-PDF-worker guard across progress commits. |
902723 |
pg_advisory_xact_lock |
app/knowledge_authority.py, app/knowledge_intake.py |
Serialize KB source authority graph cycle checks, addenda reviews, and auto-publication. |
902725 |
pg_try_advisory_xact_lock |
app/broadcasts.py |
Non-blocking single-runner lock for scheduled broadcast dispatch. |
902729 |
pg_try_advisory_lock |
app/topic_categorization.py |
Session-level single-instance lock for the topic-worker scheduler. |
6. Related Documentation
Section titled “6. Related Documentation”- DESIGN_SYSTEM.md — Visual identity, responsive layout, and accessibility standards.
- DATA_MODEL.md — Complete PostgreSQL schema, 53-table dictionary, and state machines.
- API_AND_INTERFACES.md — Complete contract for all 103 HTTP routes and internal queues.
- SECURITY_AND_PRIVACY.md — Security controls, threat boundaries, and UU PDP compliance.
- OPERATIONS_AND_DEPLOYMENT.md — Production deployment, backups, and runbooks.
- Existing feature deep dives: dashboard-plan.md, dashboard-deployment.md, natural-ticket-plan.md, knowledge-retrieval.md, pdf-knowledge.md, content-topic-categorization.md, ai-configuration.md, broadcasts.md, operational-analytics.md, admin-duty-notifications.md.