Skip to content

Engineering Standards, Module Map & SDLC

Field Value
Document ID DA-ENG-001
Pillar Software Engineering
Standard Alignment ISO/IEC/IEEE 12207 & 29119
Status Active / Synchronized with Codebase
Last Updated 2026-11-21
Source of Truth app/*.py, app/schema.sql, app/requirements.txt, app/Dockerfile, app/dashboard_static/*, tests/*.py, scripts/*.py, scripts/run-verification.sh, compose.test.yaml, .github/workflows/ci-deploy.yml, docs/TESTING.md, docs/CI-CD.md

1.1 Backend Python Modules (app/ — 36 Files + schema.sql)

Section titled “1.1 Backend Python Modules (app/ — 36 Files + schema.sql)”
Domain Module Responsibility
Core & DB app/__init__.py Package marker
app/db.py psycopg connection context manager (dict_row), advisory-locked migrate(), budget(), per-member rate_limit()
app/schema.sql Idempotent PostgreSQL 17 DDL (53 tables, pg_trgm, generated search columns, indexes)
HTTP Entrypoints app/main.py Private FastAPI API (13 routes: /health, /webhooks/waha, /v1/* App Hub endpoints), 65,536-byte streamed body limit
app/dashboard.py Public FastAPI Management Dashboard (19 core routes + 11 sub-routers + /assets), CSP/security headers, 32 MiB PDF upload gate
app/dashboard_auth.py scrypt password hashing, __Host- session/CSRF cookies, origin checks, 5/10m login lock, CLI provision()
app/operator.py CLI for initial bootstrap-admin, member, and scoped API credential provisioning
Workers & Queueing app/worker.py Priority daemon loop (staff_replies -> ticket_notifications -> jobs -> ai_config_tests -> broadcasts), 60s idle close, 3600s TTL cleanup
app/message_batching.py 3-second WhatsApp text burst collector (up to 6 messages / 2,000 chars)
AI & Policy app/assistant.py Core helpdesk orchestration (_answer), Google Gemma client (google_attempt, post_google, generate_google), fallback, send_whatsapp
app/cloudflare_ai.py Optional Cloudflare Workers AI (@cf/google/gemma-4-26b-a4b-it) fallback adapter & 3036 UTC-midnight backoff
app/ai_settings.py Pydantic AIConfig, 10 editable prompt sections, immutable FIXED_RULES / ACTION_RULES / TEXT_RULES / IMAGE_RULES / KNOWLEDGE_RULES
app/ai_management.py /manage/api/ai-config* endpoints, usage aggregation, revision control, sandboxed worker dry-run runner (run_test)
app/quota.py Local RPM/RPD budgets (budgets), 429 Google RPC QuotaFailure/RetryInfo parser, provider_cooldowns
app/policy.py Scope guards (clearly_unrelated, injection_attempt, unrelated_reply), non-action parse_reply
app/security.py WAHA HMAC-SHA512 & timestamp verifier (verify_event), PII redact(), can_access() RBAC helper
app/conversation.py sanitize() PII/token masker, compact_history(), 15-turn / 24h / 75,000-char conversation_turns store
Media & Identity app/identity.py WAHA @c.us & @lid sender phone resolver
app/media.py Private WAHA file fetcher (5 MiB, 12 MP decompression-bomb guard, EXIF strip & JPEG normalization)
Knowledge & RAG app/retrieval.py PG tsvector + pg_trgm retrieval (80 -> 5 within 10k chars), IB/WA synonym canonicalization, verbatim citations() & URL verification
app/documents.py pdf-worker daemon (run_document) & /manage/api/documents* upload, preview, section editing, publish/retire
app/pdf_extract.py Subprocess PDF extractor (pdfinfo, pdftotext, pdftoppm, tesseract ind+eng) with RLIMIT_AS/RLIMIT_FSIZE/RLIMIT_CPU
app/knowledge_authority.py MD5 revision() fingerprinting, cycle detection, prefer/replaces/coexists/conflict governance
app/knowledge_review.py Case-derived FAQ drafts & saved retrieval checks (/manage/api/knowledge-checks*)
app/knowledge_graph.py /manage/api/knowledge-graph topic/source/check/relation graph builder (8s statement timeout)
app/knowledge_intake.py WhatsApp allowlisted submissions, /pengetahuan & /adendum, trusted auto_publish, pushName//nama resolution
app/intake_language.py Deterministic Indonesian/English natural-language intake & choice parser (natural_intent, selection)
app/topic_categorization.py topic-worker daemon for idle-only, quota-capped AI topic labeling (knowledge_topics)
Tickets, Cases & Ops app/cases.py 30-day support_cases / case_messages archive, 24h idle auto-closure, media storage, retention cleanup
app/handoff.py Human takeover (claim/release/resume), 2-working-hour SLA target (reply_deadline), native WAHA app reply detection, staff_replies outbox
app/ticket_flow.py Consent-enforced ticket creation/status (direct, decision), 15m ticket_conversations state
app/ticket_language.py Literal consent/cancellation/status regexes & strict parse_ticket_decision JSON validator
app/ticket_notifications.py Opt-in on-duty admin alerts (admin_duty, ticket_open_events, ticket_notifications)
app/management.py Bulk ticket updates, password reset links, case inbox/CRM/notes/canned responses endpoints
app/admin_accounts.py Dedicated /manage/api/admins* lifecycle CRUD
app/broadcasts.py Opt-in consent commands, campaign preview digests, and rate-limited broadcast dispatcher
app/analytics.py REPEATABLE READ READ ONLY metrics engine, analytics_messages writer, 7-day MAE baseline forecast, capacity simulator

1.2 Dashboard Frontend Modules (app/dashboard_static/ — 1 HTML, 13 CSS, 14 JS, 5 Media/Font Assets)

Section titled “1.2 Dashboard Frontend Modules (app/dashboard_static/ — 1 HTML, 13 CSS, 14 JS, 5 Media/Font Assets)”
  • Shell & Entry: index.html, p3md-crest.png, support-illustration.svg, icons.svg, fonts/Manrope-Variable.ttf, fonts/OFL.txt.
  • Stylesheets (13): dashboard.css, dashboard-brand.css, management.css, tickets.css, ai-config.css, knowledge-search.css, documents-admin.css, analytics.css, mobile-tickets.css, knowledge-base.css, broadcasts.css, admin-duty.css, interface.css.
  • Scripts (14, zero-build vanilla JS with defer):
    • Core & Navigation: dashboard.js, mobile-navigation.js
    • Cases & Accounts: management.js
    • Tickets: tickets.js, ticket-ux.js
    • Knowledge Base & PDFs: knowledge-search.js, documents-admin.js, knowledge-base.js, knowledge-intake.js, knowledge-graph.js
    • Analytics, Duty, Broadcasts & AI: analytics.js, admin-duty.js, broadcasts.js, ai-config.js

1.3 Runtime Dependencies (app/requirements.txt & app/Dockerfile)

Section titled “1.3 Runtime Dependencies (app/requirements.txt & app/Dockerfile)”
  • Python 3.12 (python:3.12-slim): fastapi==0.121.1, uvicorn==0.38.0, psycopg[binary]==3.2.12, httpx==0.28.1, Pillow==13.0.0.
  • OS Packages (app/Dockerfile): poppler-utils, tesseract-ocr, tesseract-ocr-ind, tesseract-ocr-eng.

  1. Direct Parameterized SQL (psycopg): Database queries use explicit %s or %(name)s placeholders with dict_row. Critical updates acquire row locks (FOR UPDATE, FOR UPDATE SKIP LOCKED) or explicit advisory locks (902711–902729) rather than relying on implicit isolation alone.
  2. Idempotent Additive Migrations (app/schema.sql): Schema additions must use CREATE TABLE IF NOT EXISTS, ALTER TABLE <table> ADD COLUMN IF NOT EXISTS, and CREATE INDEX IF NOT EXISTS, executed safely under pg_advisory_xact_lock(902720) so old app code can roll back without breaking database compatibility (tests/test_migration.py).
  3. Strict Input/Output Contracts: Request payloads use Pydantic v2 models with ConfigDict(extra='forbid'), explicit Literal enums, regex patterns, and bounded field lengths. Model outputs use responseSchema plus defensive Python structural validation.
  4. CSP-Compliant DOM Manipulation: Frontend modules execute in 'use strict' mode, construct nodes strictly via document.createElement / .textContent / document.createElementNS, and read the CSRF cookie for every non-GET fetch.

flowchart TB
    L4["Level 4: Controlled Live Probes (scripts/probe_grounded_model.py, scripts/smoke.py)"]
    L3["Level 3: Playwright Browser & Responsive UX Suites (scripts/check_*_browser.py, check_mobile_ticket_ux.py)"]
    L2["Level 2: Isolated Disposable-PG Integration & Journey Suite (35 test_*.py / 243 tests via compose.test.yaml)"]
    L1["Level 1: Static Syntax Checks (python3 -m compileall, bash -n, node --check)"]
    L1 --> L2 --> L3 --> L4

Executed in .github/workflows/ci-deploy.yml:

Terminal window
python3 -m compileall -q app tests
bash -n scripts/deploy-server.sh
for file in app/dashboard_static/*.js; do node --check "$file"; done

3.2 Level 2 — Isolated Disposable-PostgreSQL Suite (35 Test Modules, 243 Tests)

Section titled “3.2 Level 2 — Isolated Disposable-PostgreSQL Suite (35 Test Modules, 243 Tests)”

Orchestrated via scripts/run-verification.sh and compose.test.yaml (docs/TESTING.md):

  • Spins up an ephemeral Compose project (dirman-assist-verify-<timestamp>-<pid>) with postgres:17-alpine on tmpfs, enforcing TEST_DATABASE=true, OUTBOUND_ENABLED=false, and GOOGLE_API_KEY=''.
  • tests/run_verification.py refuses execution if the connected database name is not test, mocks external Google/Cloudflare/WAHA transports, runs either --suite all (243 tests across 35 test_*.py files) or --suite journeys (14 connected journeys in tests/test_workflow_journeys.py), and extracts logs and JSON reports (review/test-results/verification-*.json).
Terminal window
docker build -t dirman-assist-app:local -f app/Dockerfile .
sh scripts/run-verification.sh all
sh scripts/run-verification.sh journeys

Notable coverage areas:

  • End-to-End Journeys (test_workflow_journeys.py): Natural-language ticketing, duplicate webhook idempotency, out-of-scope refusal, human handoff claim/reply/resume, KB CRUD, WhatsApp intake/addendum & staleness, real PDF + OCR ingestion, and citation race rejection.
  • Security & Auth (test_security.py, test_dashboard.py, test_admin_crud.py, test_media.py): HMAC/replay rejection, CSRF/Same-Origin enforcement, scrypt/reset tokens, login rate limits, decompression-bomb/SSRF rejection, self/last-admin guards.
  • Retrieval & AI (test_retrieval.py, test_retrieval_limits.py, test_ai_settings.py, test_cloudflare_ai.py, test_quota.py, test_topic_categorization.py): Short acronym (IB/WA) matching, excerpt budgeting, prompt bounds, Cloudflare adapter, cooldowns, topic classifier caps.
  • Operations & Concurrency (test_load.py, test_migration.py, test_deployment_command.py, test_worker.py, test_broadcasts.py, test_operational_analytics.py): 20-thread concurrent /v1/assist burst, schema upgrade compatibility, SSH forced-command input validation, broadcast limits, and analytics forecasts.

3.3 Level 3 & 4 — Browser Checks & Live Probes (scripts/)

Section titled “3.3 Level 3 & 4 — Browser Checks & Live Probes (scripts/)”
  • 14 Mocked / Disposable-DB Playwright Checks: check_admin_duty_browser.py, check_ai_browser.py, check_analytics_browser.py, check_broadcasts_browser.py, check_dashboard_browser.py, check_handoff_browser.py, check_knowledge_base_browser.py, check_knowledge_browser.py, check_knowledge_graph_browser.py, check_knowledge_intake_browser.py, check_management_browser.py, check_mobile_ticket_ux.py, check_pdf_admin_browser.py, check_ticket_browser.py.
  • 3 Live Read-Only HTTPS Verification Scripts: check_analytics_live.py, check_mobile_ticket_live.py, check_pdf_admin_live.py.
  • Probes & Fixtures: probe_grounded_model.py (live Google Gemma checks on test DB), seed_management_ui.py (synthetic staging seed asserting current_database() == 'test'), smoke.py (host loopback health & auth boundary check).

4. CI/CD Quality Gates (.github/workflows/ci-deploy.yml)

Section titled “4. CI/CD Quality Gates (.github/workflows/ci-deploy.yml)”
  1. Verification Job (verify): Runs on every pull_request, push to main, and workflow_dispatch on ubuntu-24.04 (pinned action SHAs, persist-credentials: false, cancels superseded PR runs). Executes syntax checks, Docker build, full run-verification.sh all, and uploads review/test-results/ artifacts for 7 days.
  2. Deployment Job (deploy): Runs only when verify succeeds on refs/heads/main (event_name != 'pull_request'). Uses non-interruptible concurrency: dirman-production and invokes scripts/deploy-server.sh via restricted SSH forced command with DEPLOY_SSH_KEY, DEPLOY_KNOWN_HOSTS, DEPLOY_HOST, and short-lived github.token. See CI-CD.md and OPERATIONS_AND_DEPLOYMENT.md.

ADR-001: PostgreSQL Lexical/Trigram RAG (tsvector + pg_trgm) Instead of External Vector Database

Section titled “ADR-001: PostgreSQL Lexical/Trigram RAG (tsvector + pg_trgm) Instead of External Vector Database”
  • Context: Self-hosted helpdesk operating within strict server RAM limits (512m for postgres) with a curated P3MDI corpus requiring exact acronyms (IB, WA, 3-digit error codes) and instant publication/withdrawal consistency.
  • Decision: Implement weighted tsvector (GIN) + pg_trgm (gin_trgm_ops) generated columns directly in PostgreSQL (app/retrieval.py), paired with curated synonym groups and Python reranking (80 candidates -> 5 articles within 10,000 chars).
  • Consequences: Eliminates external embedding APIs, vector daemon memory overhead, and index sync lag; deterministic exact/citation re-verification runs in the same SQL database. Trade-off: unseen semantic paraphrases without shared roots or curated aliases may require manual alias tuning (retrieval-acronyms-and-context.md).

ADR-002: Bounded Single-Call JSON Action Enum Instead of Arbitrary Tool Calling

Section titled “ADR-002: Bounded Single-Call JSON Action Enum Instead of Arbitrary Tool Calling”
  • Context: WhatsApp messages and KB passages are untrusted user/document inputs susceptible to prompt injection.
  • Decision: Constrain Gemma to emit a single JSON object (scope, action, answer_basis, knowledge_ids, answer) via responseSchema (app/retrieval.py, app/ticket_language.py), while Python backend modules enforce sender identity, literal ticket consent, member-filtered queries, and verified URL citations.
  • Consequences: Prompt injections cannot trigger ticket creation, read foreign data, alter priorities/assignees, or fabricate citations. Trade-off: new assistant capabilities require adding explicit enum values and Python handlers.

ADR-003: Network-Isolated pdf-worker Without Egress or Provider Secrets

Section titled “ADR-003: Network-Isolated pdf-worker Without Egress or Provider Secrets”
  • Context: Parsing arbitrary uploaded PDFs (poppler-utils, tesseract-ocr) carries memory/CPU DoS and vulnerability risks.
  • Decision: Isolate PDF extraction into pdf-worker on networks: [private] with env_file: [], 640m RAM limit, and OS rlimit caps (384 MiB AS, 32 MiB FSIZE, 210s CPU) inside app/pdf_extract.py.
  • Consequences: Compromised PDF parsers have zero internet egress and zero access to AI keys, WAHA keys, or webhook secrets. Trade-off: complex scanned PDFs exceeding limits must be split by operators before upload (pdf-knowledge.md).

ADR-004: Zero-Build Vanilla HTML/CSS/JS Management Dashboard

Section titled “ADR-004: Zero-Build Vanilla HTML/CSS/JS Management Dashboard”
  • Context: The management UI requires high security (script-src 'self'), fast hot-patching of dashboard containers, and minimal operational complexity.
  • Decision: Build the entire UI as modular static HTML, 13 CSS files, and 14 'use strict' JS modules served via FastAPI StaticFiles with self-hosted Manrope-Variable.ttf and SVG sprites (icons.svg).
  • Consequences: Zero Node build toolchain inside production Docker builds, zero third-party JS supply-chain exposure, strict CSP compliance, and straightforward rollback. Trade-off: UI state/DOM rendering relies on disciplined manual DOM helpers (el(), svgNode()) rather than reactive framework bindings.

ADR-005: Repeatable-Read Operational Analytics with Gated Local Forecasting

Section titled “ADR-005: Repeatable-Read Operational Analytics with Gated Local Forecasting”
  • Context: Administrators need reliable operational backlog, latency, AI outcome, and 7-day capacity views without external BI tools or AI token spend.
  • Decision: Compute GET /manage/api/analytics inside SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY (app/analytics.py) and gate the 7-day rolling MAE baseline forecast at >= 28 complete days and >= 20 tickets.
  • Consequences: Charts, tables, and KPIs reflect a single point-in-time consistent database snapshot with explicit quality caveats and zero AI cost. Trade-off: live queries execute directly against PostgreSQL, relying on indexed tables and 30d/180d retention caps (operational-analytics.md).