Retrieval diagnosis and fix: IB questions
Verified production evidence (2026-11-08)
Section titled “Verified production evidence (2026-11-08)”The database contained 429 eligible published KB sections, including current IB / Izin Bermalam guidance. The longest answer contained 2,910 characters. The active model configuration used minimal thinking and a 2,048-token output allowance. Read-only probes showed IB generated zero query terms; ketentuan IB generated only ketentuan and returned unrelated regulations. This particular failure occurred before the model request and was not caused by the provider context window.
The tokenizer discarded every term shorter than three characters. The same tokenization was used for article scoring, so IB was lost on both sides. Expanded names also lacked a canonical phrase relationship to IB. Query processing considered only the first 12 terms, and aliases had the same weight as actual headings. On the live corpus, generic articles with broad IB aliases competed with dedicated IB guidance.
Changes
Section titled “Changes”- Preserve two-letter words/acronyms after stop-word and numeric filtering. Canonicalize Izin Bermalam to IB, and retrieve its original full-text phrase as well as the acronym. WA/WhatsApp is also canonicalized.
- Require a match on a short acronym when one is explicitly requested. A generic match on ketentuan alone is insufficient.
- Keep up to 32 search terms, preserving short acronyms and the ends of unusually long queries. Remove conversational filler and weigh real headings above alias keywords.
- Keep the existing 10,000-character evidence budget. If a relevant source cannot fit whole, select a contiguous verbatim query-centered passage rather than silently dropping it. Preserve source identity, dates, applicability and citation validation. Mark excerpts explicitly, avoid cut URLs, and prevent the exact-answer fast path from presenting an excerpt as a complete article.
- Expose candidate/matched/selected/omitted counts, excerpt counts and serialized evidence size. Knowledge Base > Uji Jawaban includes expandable selection details. AI dry-run results also include selection diagnostics. No user text or secrets are added to logs.
- Tell the model when the evidence selection is incomplete and prohibit claims that an excerpt is the entire policy. The model is still responsible for checking applicability and exceptions.
Limits and verification
Section titled “Limits and verification”The application retrieves at most five evidence items from an 80-row candidate pool. The catalog is a bounded topic index, not proof. The new diagnostics distinguish application selection limits from actual model token limits. These limits are reported rather than silently claiming complete coverage. There is no new model call, automatic quota increase, re-ingestion requirement, or replacement of authority/expiry checks.
Validation uses a disposable PostgreSQL database and mocked providers. Regression coverage includes acronym-only queries, expanded names, verbose questions with trailing acronyms, actual-title versus alias ranking, oversized sources with relevant facts far into the answer, excerpt selection under repeated generic words, and observable context omissions. Six read-only production probes use the real published KB without model calls or WhatsApp sends. Production rollout is app-only, with a database and app-source backup made before changing deployed sources.
Reproduce production probes with docker compose exec -T api python -u < review/retrieval_live_verify.py. The script starts a READ ONLY database transaction and invokes retrieval only. Do not use test database reset commands against the production compose project.