Skip to content

Verification

Run from /opt/dirman-assist on the server:

Terminal window
sh scripts/run-verification.sh all

For the connected user journeys only:

Terminal window
sh scripts/run-verification.sh journeys

Each run creates its own uniquely named Compose project and PostgreSQL database on tmpfs. It runs the currently built dirman-assist-app:local image with the current tests mounted read-only. It removes only that run’s containers/network on exit. It does not restart the production app, modify production data, enable duty status, or send WhatsApp messages. Rebuild the application image first if testing changed application code.

The runner refuses to run unless TEST_DATABASE=true, the actual database name is test, OUTBOUND_ENABLED=false, and no Google API key is set in the test environment. The new journey fixtures also check the actual database name before any cleanup. Real model credentials are never copied into the test environment. The test container retains its unprivileged service UID; reports are copied out after it stops.

Results are saved under review/test-results/ as a text log and JSON report with individual test IDs, statuses, durations, failure details, aggregate counts, and an explicit statement that external providers are mocked. The command exits nonzero for failures, errors, skipped tests, an empty suite, or inability to save the report. A fresh database is used on every run, preventing stale fixture collisions.

tests/test_workflow_journeys.py exercises real FastAPI endpoints, signed webhook validation, queue handling, PostgreSQL transactions, model prompt construction and JSON parsing, source governance, and dashboard APIs. Only the external model transport and WhatsApp sends are mocked.

Journey Assertions
Natural-language ticket to duty alert Signed incoming message creates one ticket and opening event; duplicate delivery cannot duplicate either; case archive includes the provided sender name; one on-duty admin alert contains the correct ticket link; dashboard status changes are reflected in member status queries; reopening creates one new alert cycle.
Complaint and consent A complaint produces an offer without a ticket; cancellation leaves no ticket; later explicit consent creates one ticket without another model call.
Out-of-scope and invalid delivery Unrelated requests are politely denied without ticket creation or a model call; invalid signatures and expired events create no queued job.
Missing knowledge Unsupported organization information produces clarification and a human-help option, never an invented ticket; an explicit request then activates handoff.
Human takeover and return Claim and staff reply retries do not duplicate notices/messages; AI stops for the affected member while another member remains served; closing the manual case sends the resume notice and subsequent messages use the bot again.
Dashboard KB CRUD Drafts stay out of retrieval; publishing exposes the article; editing replaces the retrieved content; deleting removes it from search.
Trusted WhatsApp knowledge Natural-language submission stays a draft until admin approval; publication makes it retrievable and available in the model payload; the WhatsApp answer contains a backend-verified citation; the proposal and review are logged.
Natural-language addendum Sender selects the correct target; approval publishes the replacement; original text stays preserved and excluded from current retrieval; the unselected source remains valid; audit history contains before/after content.
Knowledge permissions Unauthorized, revoked, and rejected submissions never become published evidence.
Stale addendum A source changed after target selection causes review to return 409 without partial publication or overwriting the newer source.
Real PDF ingestion A generated PDF is actually extracted using the installed PDF toolchain; review sections are not public until publication; a saved KB check finds its page; a grounded answer cites its PDF filename and page; withdrawal removes it from retrieval and causes the saved check to fail.
KB checks and AI previews Saved checks find the expected article, including IB/Izin Bermalam; dry-run configuration testing reports retrieved IDs and verified citations without creating tickets or sending WhatsApp messages.
Citation race A source withdrawn while the model is generating cannot be cited as current evidence.

The broader suite additionally covers retrieval ranking and budgets, short acronyms, long sources, expiry/authority rules, screenshot handling, conversation history, PDF limits/preview, ticket filters/bulk updates, duty delivery failures, account security, analytics, and broadcast consent. Consult the individual test_*.py modules for the assertions.

On the Windows development machine, with Playwright and Chrome installed:

Terminal window
python deploy/scripts/check_knowledge_intake_browser.py
python deploy/scripts/check_admin_duty_browser.py

Both use synthetic mocked endpoints. They check contributor CRUD, proposal approval/audit history, duty opt-in/off, delivery history, ticket links, mobile navigation, and small/desktop viewport overflow. They do not log into the production dashboard or send messages.

These are application integration and regression tests. Real PDF extraction and database operations are exercised. Provider transport and WhatsApp delivery are mocked, so passing does not establish current Google/Cloudflare availability, the accuracy of every answer produced by a live model, real WhatsApp delivery/read receipts, or the absence of account restrictions. Those require a separately controlled live acceptance check. No live sends are part of this command.