Engineering Standards, Module Map & SDLC
| Field | Value |
|---|---|
| Document ID | DA-ENG-001 |
| Pillar | Software Engineering |
| Standard Alignment | ISO/IEC/IEEE 12207 & 29119 |
| Status | Active / Synchronized with Codebase |
| Last Updated | 2026-11-21 |
| Source of Truth | app/*.py, app/schema.sql, app/requirements.txt, app/Dockerfile, app/dashboard_static/*, tests/*.py, scripts/*.py, scripts/run-verification.sh, compose.test.yaml, .github/workflows/ci-deploy.yml, docs/TESTING.md, docs/CI-CD.md |
1. Repository & Module Map
Section titled “1. Repository & Module Map”1.1 Backend Python Modules (app/ — 36 Files + schema.sql)
Section titled “1.1 Backend Python Modules (app/ — 36 Files + schema.sql)”| Domain | Module | Responsibility |
|---|---|---|
| Core & DB | app/__init__.py |
Package marker |
app/db.py |
psycopg connection context manager (dict_row), advisory-locked migrate(), budget(), per-member rate_limit() |
|
app/schema.sql |
Idempotent PostgreSQL 17 DDL (53 tables, pg_trgm, generated search columns, indexes) |
|
| HTTP Entrypoints | app/main.py |
Private FastAPI API (13 routes: /health, /webhooks/waha, /v1/* App Hub endpoints), 65,536-byte streamed body limit |
app/dashboard.py |
Public FastAPI Management Dashboard (19 core routes + 11 sub-routers + /assets), CSP/security headers, 32 MiB PDF upload gate |
|
app/dashboard_auth.py |
scrypt password hashing, __Host- session/CSRF cookies, origin checks, 5/10m login lock, CLI provision() |
|
app/operator.py |
CLI for initial bootstrap-admin, member, and scoped API credential provisioning |
|
| Workers & Queueing | app/worker.py |
Priority daemon loop (staff_replies -> ticket_notifications -> jobs -> ai_config_tests -> broadcasts), 60s idle close, 3600s TTL cleanup |
app/message_batching.py |
3-second WhatsApp text burst collector (up to 6 messages / 2,000 chars) |
|
| AI & Policy | app/assistant.py |
Core helpdesk orchestration (_answer), Google Gemma client (google_attempt, post_google, generate_google), fallback, send_whatsapp |
app/cloudflare_ai.py |
Optional Cloudflare Workers AI (@cf/google/gemma-4-26b-a4b-it) fallback adapter & 3036 UTC-midnight backoff |
|
app/ai_settings.py |
Pydantic AIConfig, 10 editable prompt sections, immutable FIXED_RULES / ACTION_RULES / TEXT_RULES / IMAGE_RULES / KNOWLEDGE_RULES |
|
app/ai_management.py |
/manage/api/ai-config* endpoints, usage aggregation, revision control, sandboxed worker dry-run runner (run_test) |
|
app/quota.py |
Local RPM/RPD budgets (budgets), 429 Google RPC QuotaFailure/RetryInfo parser, provider_cooldowns |
|
app/policy.py |
Scope guards (clearly_unrelated, injection_attempt, unrelated_reply), non-action parse_reply |
|
app/security.py |
WAHA HMAC-SHA512 & timestamp verifier (verify_event), PII redact(), can_access() RBAC helper |
|
app/conversation.py |
sanitize() PII/token masker, compact_history(), 15-turn / 24h / 75,000-char conversation_turns store |
|
| Media & Identity | app/identity.py |
WAHA @c.us & @lid sender phone resolver |
app/media.py |
Private WAHA file fetcher (5 MiB, 12 MP decompression-bomb guard, EXIF strip & JPEG normalization) |
|
| Knowledge & RAG | app/retrieval.py |
PG tsvector + pg_trgm retrieval (80 -> 5 within 10k chars), IB/WA synonym canonicalization, verbatim citations() & URL verification |
app/documents.py |
pdf-worker daemon (run_document) & /manage/api/documents* upload, preview, section editing, publish/retire |
|
app/pdf_extract.py |
Subprocess PDF extractor (pdfinfo, pdftotext, pdftoppm, tesseract ind+eng) with RLIMIT_AS/RLIMIT_FSIZE/RLIMIT_CPU |
|
app/knowledge_authority.py |
MD5 revision() fingerprinting, cycle detection, prefer/replaces/coexists/conflict governance |
|
app/knowledge_review.py |
Case-derived FAQ drafts & saved retrieval checks (/manage/api/knowledge-checks*) |
|
app/knowledge_graph.py |
/manage/api/knowledge-graph topic/source/check/relation graph builder (8s statement timeout) |
|
app/knowledge_intake.py |
WhatsApp allowlisted submissions, /pengetahuan & /adendum, trusted auto_publish, pushName//nama resolution |
|
app/intake_language.py |
Deterministic Indonesian/English natural-language intake & choice parser (natural_intent, selection) |
|
app/topic_categorization.py |
topic-worker daemon for idle-only, quota-capped AI topic labeling (knowledge_topics) |
|
| Tickets, Cases & Ops | app/cases.py |
30-day support_cases / case_messages archive, 24h idle auto-closure, media storage, retention cleanup |
app/handoff.py |
Human takeover (claim/release/resume), 2-working-hour SLA target (reply_deadline), native WAHA app reply detection, staff_replies outbox |
|
app/ticket_flow.py |
Consent-enforced ticket creation/status (direct, decision), 15m ticket_conversations state |
|
app/ticket_language.py |
Literal consent/cancellation/status regexes & strict parse_ticket_decision JSON validator |
|
app/ticket_notifications.py |
Opt-in on-duty admin alerts (admin_duty, ticket_open_events, ticket_notifications) |
|
app/management.py |
Bulk ticket updates, password reset links, case inbox/CRM/notes/canned responses endpoints | |
app/admin_accounts.py |
Dedicated /manage/api/admins* lifecycle CRUD |
|
app/broadcasts.py |
Opt-in consent commands, campaign preview digests, and rate-limited broadcast dispatcher | |
app/analytics.py |
REPEATABLE READ READ ONLY metrics engine, analytics_messages writer, 7-day MAE baseline forecast, capacity simulator |
1.2 Dashboard Frontend Modules (app/dashboard_static/ — 1 HTML, 13 CSS, 14 JS, 5 Media/Font Assets)
Section titled “1.2 Dashboard Frontend Modules (app/dashboard_static/ — 1 HTML, 13 CSS, 14 JS, 5 Media/Font Assets)”- Shell & Entry:
index.html,p3md-crest.png,support-illustration.svg,icons.svg,fonts/Manrope-Variable.ttf,fonts/OFL.txt. - Stylesheets (13):
dashboard.css,dashboard-brand.css,management.css,tickets.css,ai-config.css,knowledge-search.css,documents-admin.css,analytics.css,mobile-tickets.css,knowledge-base.css,broadcasts.css,admin-duty.css,interface.css. - Scripts (14, zero-build vanilla JS with
defer):- Core & Navigation:
dashboard.js,mobile-navigation.js - Cases & Accounts:
management.js - Tickets:
tickets.js,ticket-ux.js - Knowledge Base & PDFs:
knowledge-search.js,documents-admin.js,knowledge-base.js,knowledge-intake.js,knowledge-graph.js - Analytics, Duty, Broadcasts & AI:
analytics.js,admin-duty.js,broadcasts.js,ai-config.js
- Core & Navigation:
1.3 Runtime Dependencies (app/requirements.txt & app/Dockerfile)
Section titled “1.3 Runtime Dependencies (app/requirements.txt & app/Dockerfile)”- Python 3.12 (
python:3.12-slim):fastapi==0.121.1,uvicorn==0.38.0,psycopg[binary]==3.2.12,httpx==0.28.1,Pillow==13.0.0. - OS Packages (
app/Dockerfile):poppler-utils,tesseract-ocr,tesseract-ocr-ind,tesseract-ocr-eng.
2. Engineering & Coding Conventions
Section titled “2. Engineering & Coding Conventions”- Direct Parameterized SQL (
psycopg): Database queries use explicit%sor%(name)splaceholders withdict_row. Critical updates acquire row locks (FOR UPDATE,FOR UPDATE SKIP LOCKED) or explicit advisory locks (902711–902729) rather than relying on implicit isolation alone. - Idempotent Additive Migrations (
app/schema.sql): Schema additions must useCREATE TABLE IF NOT EXISTS,ALTER TABLE <table> ADD COLUMN IF NOT EXISTS, andCREATE INDEX IF NOT EXISTS, executed safely underpg_advisory_xact_lock(902720)so old app code can roll back without breaking database compatibility (tests/test_migration.py). - Strict Input/Output Contracts: Request payloads use Pydantic v2 models with
ConfigDict(extra='forbid'), explicitLiteralenums, regex patterns, and bounded field lengths. Model outputs useresponseSchemaplus defensive Python structural validation. - CSP-Compliant DOM Manipulation: Frontend modules execute in
'use strict'mode, construct nodes strictly viadocument.createElement/.textContent/document.createElementNS, and read the CSRF cookie for every non-GET fetch.
3. Verification & Testing Pyramid
Section titled “3. Verification & Testing Pyramid”flowchart TB
L4["Level 4: Controlled Live Probes (scripts/probe_grounded_model.py, scripts/smoke.py)"]
L3["Level 3: Playwright Browser & Responsive UX Suites (scripts/check_*_browser.py, check_mobile_ticket_ux.py)"]
L2["Level 2: Isolated Disposable-PG Integration & Journey Suite (35 test_*.py / 243 tests via compose.test.yaml)"]
L1["Level 1: Static Syntax Checks (python3 -m compileall, bash -n, node --check)"]
L1 --> L2 --> L3 --> L4
3.1 Level 1 — Static Syntax Gates
Section titled “3.1 Level 1 — Static Syntax Gates”Executed in .github/workflows/ci-deploy.yml:
python3 -m compileall -q app testsbash -n scripts/deploy-server.shfor file in app/dashboard_static/*.js; do node --check "$file"; done3.2 Level 2 — Isolated Disposable-PostgreSQL Suite (35 Test Modules, 243 Tests)
Section titled “3.2 Level 2 — Isolated Disposable-PostgreSQL Suite (35 Test Modules, 243 Tests)”Orchestrated via scripts/run-verification.sh and compose.test.yaml (docs/TESTING.md):
- Spins up an ephemeral Compose project (
dirman-assist-verify-<timestamp>-<pid>) withpostgres:17-alpineontmpfs, enforcingTEST_DATABASE=true,OUTBOUND_ENABLED=false, andGOOGLE_API_KEY=''. tests/run_verification.pyrefuses execution if the connected database name is nottest, mocks external Google/Cloudflare/WAHA transports, runs either--suite all(243tests across35test_*.pyfiles) or--suite journeys(14connected journeys intests/test_workflow_journeys.py), and extracts logs and JSON reports (review/test-results/verification-*.json).
docker build -t dirman-assist-app:local -f app/Dockerfile .sh scripts/run-verification.sh allsh scripts/run-verification.sh journeysNotable coverage areas:
- End-to-End Journeys (
test_workflow_journeys.py): Natural-language ticketing, duplicate webhook idempotency, out-of-scope refusal, human handoff claim/reply/resume, KB CRUD, WhatsApp intake/addendum & staleness, real PDF + OCR ingestion, and citation race rejection. - Security & Auth (
test_security.py,test_dashboard.py,test_admin_crud.py,test_media.py): HMAC/replay rejection, CSRF/Same-Origin enforcement, scrypt/reset tokens, login rate limits, decompression-bomb/SSRF rejection, self/last-admin guards. - Retrieval & AI (
test_retrieval.py,test_retrieval_limits.py,test_ai_settings.py,test_cloudflare_ai.py,test_quota.py,test_topic_categorization.py): Short acronym (IB/WA) matching, excerpt budgeting, prompt bounds, Cloudflare adapter, cooldowns, topic classifier caps. - Operations & Concurrency (
test_load.py,test_migration.py,test_deployment_command.py,test_worker.py,test_broadcasts.py,test_operational_analytics.py): 20-thread concurrent/v1/assistburst, schema upgrade compatibility, SSH forced-command input validation, broadcast limits, and analytics forecasts.
3.3 Level 3 & 4 — Browser Checks & Live Probes (scripts/)
Section titled “3.3 Level 3 & 4 — Browser Checks & Live Probes (scripts/)”- 14 Mocked / Disposable-DB Playwright Checks:
check_admin_duty_browser.py,check_ai_browser.py,check_analytics_browser.py,check_broadcasts_browser.py,check_dashboard_browser.py,check_handoff_browser.py,check_knowledge_base_browser.py,check_knowledge_browser.py,check_knowledge_graph_browser.py,check_knowledge_intake_browser.py,check_management_browser.py,check_mobile_ticket_ux.py,check_pdf_admin_browser.py,check_ticket_browser.py. - 3 Live Read-Only HTTPS Verification Scripts:
check_analytics_live.py,check_mobile_ticket_live.py,check_pdf_admin_live.py. - Probes & Fixtures:
probe_grounded_model.py(live Google Gemma checks ontestDB),seed_management_ui.py(synthetic staging seed assertingcurrent_database() == 'test'),smoke.py(host loopback health & auth boundary check).
4. CI/CD Quality Gates (.github/workflows/ci-deploy.yml)
Section titled “4. CI/CD Quality Gates (.github/workflows/ci-deploy.yml)”- Verification Job (
verify): Runs on everypull_request,pushtomain, andworkflow_dispatchonubuntu-24.04(pinned action SHAs,persist-credentials: false, cancels superseded PR runs). Executes syntax checks, Docker build, fullrun-verification.sh all, and uploadsreview/test-results/artifacts for 7 days. - Deployment Job (
deploy): Runs only whenverifysucceeds onrefs/heads/main(event_name != 'pull_request'). Uses non-interruptibleconcurrency: dirman-productionand invokesscripts/deploy-server.shvia restricted SSH forced command withDEPLOY_SSH_KEY,DEPLOY_KNOWN_HOSTS,DEPLOY_HOST, and short-livedgithub.token. See CI-CD.md and OPERATIONS_AND_DEPLOYMENT.md.
5. Architecture Decision Records (ADRs)
Section titled “5. Architecture Decision Records (ADRs)”ADR-001: PostgreSQL Lexical/Trigram RAG (tsvector + pg_trgm) Instead of External Vector Database
Section titled “ADR-001: PostgreSQL Lexical/Trigram RAG (tsvector + pg_trgm) Instead of External Vector Database”- Context: Self-hosted helpdesk operating within strict server RAM limits (
512mforpostgres) with a curated P3MDI corpus requiring exact acronyms (IB,WA, 3-digit error codes) and instant publication/withdrawal consistency. - Decision: Implement weighted
tsvector(GIN) +pg_trgm(gin_trgm_ops) generated columns directly in PostgreSQL (app/retrieval.py), paired with curated synonym groups and Python reranking (80candidates ->5articles within10,000chars). - Consequences: Eliminates external embedding APIs, vector daemon memory overhead, and index sync lag; deterministic exact/citation re-verification runs in the same SQL database. Trade-off: unseen semantic paraphrases without shared roots or curated aliases may require manual alias tuning (retrieval-acronyms-and-context.md).
ADR-002: Bounded Single-Call JSON Action Enum Instead of Arbitrary Tool Calling
Section titled “ADR-002: Bounded Single-Call JSON Action Enum Instead of Arbitrary Tool Calling”- Context: WhatsApp messages and KB passages are untrusted user/document inputs susceptible to prompt injection.
- Decision: Constrain Gemma to emit a single JSON object (
scope,action,answer_basis,knowledge_ids,answer) viaresponseSchema(app/retrieval.py,app/ticket_language.py), while Python backend modules enforce sender identity, literal ticket consent, member-filtered queries, and verified URL citations. - Consequences: Prompt injections cannot trigger ticket creation, read foreign data, alter priorities/assignees, or fabricate citations. Trade-off: new assistant capabilities require adding explicit enum values and Python handlers.
ADR-003: Network-Isolated pdf-worker Without Egress or Provider Secrets
Section titled “ADR-003: Network-Isolated pdf-worker Without Egress or Provider Secrets”- Context: Parsing arbitrary uploaded PDFs (
poppler-utils,tesseract-ocr) carries memory/CPU DoS and vulnerability risks. - Decision: Isolate PDF extraction into
pdf-workeronnetworks: [private]withenv_file: [],640mRAM limit, and OSrlimitcaps (384 MiBAS,32 MiBFSIZE,210sCPU) insideapp/pdf_extract.py. - Consequences: Compromised PDF parsers have zero internet egress and zero access to AI keys, WAHA keys, or webhook secrets. Trade-off: complex scanned PDFs exceeding limits must be split by operators before upload (pdf-knowledge.md).
ADR-004: Zero-Build Vanilla HTML/CSS/JS Management Dashboard
Section titled “ADR-004: Zero-Build Vanilla HTML/CSS/JS Management Dashboard”- Context: The management UI requires high security (
script-src 'self'), fast hot-patching of dashboard containers, and minimal operational complexity. - Decision: Build the entire UI as modular static HTML, 13 CSS files, and 14
'use strict'JS modules served via FastAPIStaticFileswith self-hostedManrope-Variable.ttfand SVG sprites (icons.svg). - Consequences: Zero Node build toolchain inside production Docker builds, zero third-party JS supply-chain exposure, strict CSP compliance, and straightforward rollback. Trade-off: UI state/DOM rendering relies on disciplined manual DOM helpers (
el(),svgNode()) rather than reactive framework bindings.
ADR-005: Repeatable-Read Operational Analytics with Gated Local Forecasting
Section titled “ADR-005: Repeatable-Read Operational Analytics with Gated Local Forecasting”- Context: Administrators need reliable operational backlog, latency, AI outcome, and 7-day capacity views without external BI tools or AI token spend.
- Decision: Compute
GET /manage/api/analyticsinsideSET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY(app/analytics.py) and gate the 7-day rolling MAE baseline forecast at>= 28complete days and>= 20tickets. - Consequences: Charts, tables, and KPIs reflect a single point-in-time consistent database snapshot with explicit quality caveats and zero AI cost. Trade-off: live queries execute directly against PostgreSQL, relying on indexed tables and 30d/180d retention caps (operational-analytics.md).
6. Related Documentation
Section titled “6. Related Documentation”- ARCHITECTURE.md — Runtime topology and service budgets.
- API_AND_INTERFACES.md — Full HTTP and worker queue contracts.
- TESTING.md — Detailed test runner and connected journey assertions.
- CI-CD.md — GitHub Actions and server deployment controller details.
- OPERATIONS_AND_DEPLOYMENT.md — Operational runbooks and troubleshooting.