Why naive chunking fails on Indonesian documents
Fixed-size chunking at 512 tokens works acceptably on clean English prose and poorly on the documents Indonesian companies actually keep. SOPs are numbered clause hierarchies where clause 4.2.3 only means anything underneath heading 4.2. Split them arbitrarily and you retrieve a sentence saying the approval limit is Rp 50 juta, without the context that this applies only to branch managers in the Jakarta region.
Chunk on structure instead: headings, numbered clauses, and table boundaries. We land on 700 to 900 tokens with roughly fifteen percent overlap, and we prepend the document title and full section path to every chunk before embedding. That single prepend is the cheapest quality win in the entire pipeline. Budget tokens honestly too — Indonesian text runs fifteen to twenty percent longer than equivalent English under most tokenizers.
Scanned documents are their own project. A meaningful share of any Indonesian corpus is a photographed SK or a signed perjanjian saved as an image PDF. Run OCR with an Indonesian language model, keep the confidence score, and refuse to index anything below your threshold. Garbled OCR text embeds perfectly happily and then surfaces as a confident, wrong answer nobody can trace back.
Mixed-language retrieval is the default, not the edge case
Users ask in Indonesian. Contracts, technical specifications, and vendor policies are frequently in English, and plenty of internal documents mix both inside a single paragraph. Keyword search fails here for a structural reason: Indonesian affixation means mendaftarkan, pendaftaran, and daftar are one concept and three different strings. Stemming with a library like Sastrawi fixes part of that; embeddings fix the rest.
- Use a genuinely multilingual embedding model and test retrieval in both directions before committing
- Run hybrid retrieval — BM25 with Indonesian stemming plus dense vectors — then rerank the merged set
- Add a query rewriting step that expands abbreviations and department jargon before retrieval runs
- Keep product names, module names, and legal terms untranslated in both the query and the index
Reranking is where a mediocre pipeline becomes a good one. Retrieve twenty candidates cheaply, rerank down to the best five, and pass only those into the model. On a 4.000-document corpus we measured answer accuracy moving from roughly 71 to 88 percent by adding a reranker alone, without touching the embedding model or the prompt.
PII filtering before anything reaches the index
Personal data is the risk nobody scopes. Indonesian corpora are full of it: NIK on scanned KTP copies, NPWP on invoices, BPJS numbers in HR letters, salary figures in appraisal documents, phone numbers everywhere. A retrieval index is the fastest mechanism ever invented for making the payroll spreadsheet searchable by the whole company, and it will happen on day one if you index by folder without checking permissions.
- Run a regex and NER pass at ingestion for NIK, NPWP, BPJS, phone numbers, and bank accounts
- Mirror source-system permissions into the index and filter at query time by the user’s real groups
- Keep HR and finance in a separate index with separate access, not a metadata flag on one index
- Log which chunks were retrieved for which user so an access complaint can be investigated
- Re-sync permissions on a schedule — people change roles faster than anyone updates the index
Budget for a periodic access audit as well. Pick ten questions that should be refused for a given role, run them once a month, and record the result. It takes twenty minutes, and it is the only evidence you will have that the permission model still works after six months of folders being reshuffled and people changing teams.
UU PDP, residency, and what legal will ask
UU 27/2022 applies to this system in ways your legal team will ask about specifically. You are the controller; your model vendor and hosting provider are processors, and each needs a data processing agreement covering purpose limitation, sub-processors, security, deletion at termination, and an explicit no-training clause. Cross-border transfer needs an equivalent protection level, binding safeguards, or consent — pick one and write down which.
In practice this pushes most clients toward a model endpoint in a region they can name and a vector store hosted in a Jakarta data centre. Set a retention window for conversation logs — ninety days is defensible — and make sure a scheduled job enforces it rather than a paragraph in a policy document. Record the processing activity in your RoPA before the pilot starts, not afterwards.
Evaluation and a rollout that sticks
Evaluate with real questions. Collect fifty to eighty queries from the people who will actually use the system, have a subject-matter expert write the correct answer and cite the source document, then score retrieval and generation separately. Retrieval failures and generation failures need completely different fixes, and one blended accuracy number hides which of the two you have.
Track two numbers over time rather than one. Retrieval recall at five tells you whether the correct chunk even reached the context window; answer faithfulness tells you whether the model stayed inside what it was given. High recall with low faithfulness is a prompting problem. Low recall means no amount of prompt engineering is going to save the answer.
Roll out to a single department first, put a thumbs-down button on every answer, and read every piece of negative feedback yourself for the first month. Most of what you learn will be about documents rather than about the model: the SOP is three versions out of date, two departments hold contradictory policies, and nobody knew. Fixing that is worth more than the assistant.