Build the eval set before you build the agent
Before writing a single prompt, pull 100 to 150 real cases out of the system the agent will replace — closed support tickets, past purchase requests, last quarter’s supplier invoices — and label the correct outcome for each one. That labelled set is your eval. It takes two or three days to build, and it is the only thing standing between you and a project where nobody can say whether last week’s prompt change helped or hurt.
Run the eval on every change: prompt edits, model swaps, tool schema changes, retrieval tweaks. Track one headline pass rate with failure categories underneath it. We ship at 92 percent on the eval set and treat anything below 85 as not production-ready. The number matters less than the discipline — teams without an eval argue about vibes and ship regressions they discover from customers.
Tool design matters more than model choice
Everyone wants to talk about which model to use. Across our projects, swapping between frontier models moved the eval pass rate by about four points. Rewriting the tool layer moved it nineteen. Models improve on their own schedule; your tool schema does not. If you have one day to spend on quality, spend it on the tools, their descriptions, and the errors they return.
- One job per tool — a single do-everything tool with a mode parameter fails in ways you cannot debug
- Typed, narrow parameters beat free text; an enum of six statuses removes an entire class of error
- Never expose raw SQL; expose twelve parameterised queries the business actually asks for
- Return errors as instructions: name the invalid field and show what a valid value looks like
- Make every write idempotent with a key, because the agent will retry and you will not notice
The error-message point deserves expanding. A tool that returns HTTP 400 teaches the model nothing. A tool that returns a sentence explaining that the posting date falls in a closed period, and that the earliest open period starts on 1 April, lets the model correct itself in a single turn. We now write tool errors the way we would write them for a new junior analyst on their first week.
Human-in-the-loop is a product decision
Human review is not a safety net you bolt on when the agent underperforms. It is a product decision you make per action, based on how expensive a mistake is and how easily it can be reversed. Reading data needs no approval. Drafting a reply needs a glance. Posting a journal entry or messaging a customer needs a person whose name is attached to the outcome.
We usually set thresholds in both rupiah and confidence. Supplier invoice matching posts automatically under Rp 5 juta when the three-way match is exact, queues for approval above that, and always queues when the vendor is new. The approval queue is a first-class screen with keyboard shortcuts, not a buried notification, because a queue people dread becomes a rubber stamp within a fortnight.
One more rule: never let the agent be the only record of what happened. Every automated action writes a document in ERPNext with the agent named as the author, so a supervisor can see the decision, the inputs it used, and the reasoning without opening a log viewer. Auditability is what lets finance sign off on giving it more autonomy later.
Guardrails, budgets, and the boring plumbing
The unglamorous parts decide whether an agent survives contact with production. Cap tool calls per conversation — twelve is a reasonable ceiling for most workflows — and fail loudly rather than looping. Set a token budget per session and alert when it is exceeded. Redact NIK, NPWP, and phone numbers before anything reaches your logging platform, because logs get copied to places your data map does not cover.
- An allow-list for outbound channels, so a bug cannot message ten thousand customers at 02:00
- A kill switch any operations lead can press, tested monthly like a fire drill
- Per-session token and tool-call budgets with real alerting, not dashboards nobody opens
- PII redaction at the logging boundary covering NIK, NPWP, BPJS numbers, and phone numbers
- Full trace capture of every prompt, tool call, and result, retained long enough to investigate a complaint
Version everything that can change behaviour: prompts, tool schemas, retrieval configuration, and model identifiers. Pin the model version explicitly instead of following a floating alias, because a silent provider upgrade can move your outputs on a Tuesday afternoon with no deployment on your side and no obvious place to look. Treat a model bump like any other dependency upgrade and run the eval first.
Cost control matters more than people expect. A well-designed support agent resolves a ticket for Rp 300 to 900 in model spend. A badly designed one loops through retrieval, re-reads the same document nine times, and costs Rp 6.000 for the same answer. Both feel identical to the user. Only the trace tells you which one you actually deployed.
What we would do differently
Three things. We would build the approval interface in week one instead of week six, because it shapes how much autonomy the agent can safely be given. We would instrument traces before the first pilot user rather than after the first complaint. And we would start narrower — one workflow done properly earns the political capital for the next five, while a broad assistant that does eight things adequately gets quietly abandoned.