An AI agent that answers well in a demo can still fail in its first week connected to your ERP. A demo uses clean questions, and production uses the documents your team actually has. The cheap way to find this out is a staged evaluation: two weeks to decide whether the agent is worth building, eight weeks to prove it on a bounded scope, then a written acceptance gate before the production budget is spent. This article covers what each stage should test, which numbers belong in the sign-off criteria, and who owns the test suite once the vendor has left.
WHY PILOTS FAIL
What Goes Wrong When An Agent Skips The Pilot
An agent that skips the pilot usually reaches production as a system nobody trusts enough to use, because a proof of concept was taken as evidence. A proof of concept shows that a model can do a task once. A pilot shows how often it does the task correctly on your data, what it does when it is wrong, and whether the people who must approve its work will actually approve it.
The ways agents fail are well documented. In December 2025 the OWASP GenAI Security Project published its Top 10 for Agentic Applications. It covers risks in tool use, identity and privilege, memory and context, and cascading failures between agents. In January 2026 Singapore’s IMDA launched its Model AI Governance Framework for Agentic AI. It advises organisations to test agents for execution accuracy, policy adherence and tool use before deployment, then roll out gradually with monitoring. In the United States, NIST’s Generative AI Profile, AI 600-1, lists measurement actions such as testing whether a system reveals personal or confidential information, and checking that fine-tuning has not weakened safety controls.
These frameworks say what to test and leave the pass mark to you. The pass mark is a business decision. Write it down before the pilot starts, or it will be negotiated afterwards against money already spent.
THE THREE STAGES
What Each Stage Of An Agent Evaluation Actually Tests
Each stage answers one question and ends in a decision. The feasibility assessment asks whether the agent should be built, the pilot asks whether it works on a bounded scope, and production asks whether it keeps working at scale. Skipping a stage pushes its question into a later, more expensive one.
Feasibility: Should This Be Built
Two weeks against your real documents and systems. The output is a diagnosis of the process the agent would replace, a judgement on whether your data can support it, a cost model, and a go or no-go. A no is a useful result. It means the data is too inconsistent, the process too rare or the saving too small, and you learned that for the price of an assessment.
Pilot: Does It Work Here
Eight weeks on one process, one team and one target system. The agent runs on live inputs. Every write it proposes passes through a human approval gate, and every proposal is logged with the reviewer’s verdict. That log is the evidence the acceptance decision rests on.
Production: Does It Keep Working
Twelve weeks or more to integrate fully, train operators and hand over. The main question shifts from accuracy to drift: whether document formats change, whether a model update changes behaviour, and whether the regression suite still passes after each change.
At ARSA the three stages are separate, priced engagements, listed on the LLM and AI agent development page.
| Stage | Duration | From | Decision It Produces |
|---|---|---|---|
| Feasibility Assessment | 2 weeks | $4,500 | Go or no-go, with a cost model |
| Pilot Deployment | 8 weeks | $20,000 | Accept, extend or stop, measured against agreed KPIs |
| Single-Site Production | 12 weeks or more | $60,000 | Handover with support |
The feasibility fee is deducted from the project fee if you contract within 90 days. List prices are in USD and invoicing is available in USD or EUR. Buyers outside Indonesia are served the same way as domestic ones. For a fuller breakdown of build and running costs, see what AI agent development costs on premise compared with SaaS.
BUILDING THE TEST SET
How To Build A Test Set The Agent Has Never Seen
A useful test set is a few hundred real cases from your own archive. Each case is paired with the answer a competent employee gave, and the set is kept away from anyone tuning the agent. Questions the vendor writes only show how well the vendor writes questions.
Take the cases from the last six to twelve months so the awkward ones are included: the scanned invoice with a coffee stain, the purchase order in two currencies, the contract with a handwritten amendment. Weight the set toward cases where a mistake is expensive. Then split it. One part goes to the engineers to tune retrieval and prompts. The other part stays locked until the acceptance run, so the result reflects work the agent has never practised on.
Build a smaller adversarial set beside it. This set should contain documents with hidden instructions, requests for records the user is not entitled to see, and tasks that would need a write the agent must refuse to prepare. These cases map directly onto the OWASP agentic risks. A pilot that has never been attacked has only been tested on good days.
ACCEPTANCE CRITERIA
What Numbers Should Gate Sign-Off
Sign-off should rest on four or five measured numbers agreed before the pilot starts. Each threshold is set by the person who owns that risk and justified by what an error costs. A single headline accuracy figure hides the failures that matter, so measure them separately.
| Measure | How It Is Counted | Who Sets The Threshold |
|---|---|---|
| Task Accuracy | Share of held-out cases where the prepared output matches the reference answer | Process owner, against the error rate of the current manual process |
| Approval Rate | Share of proposed writes the reviewer accepts without editing | Process owner |
| Critical Error Rate | Proposals that would have caused financial, legal or safety harm if committed | Risk or compliance, usually at or near zero |
| Refusal Correctness | Share of out-of-scope or unauthorised requests the agent declines | IT security |
| Time Per Task | Reviewer minutes per item, compared with the manual baseline | Operations |
The baseline matters more than the threshold. Suppose your accounts team misreads 3 invoices in every 100. An agent that misreads 2 is an improvement. An agent that misreads 1 but takes longer to check than the original task took saves nothing. Measure the manual process during feasibility so the pilot has something to beat.
Track critical errors separately because the approval gate is what contains them. In every ARSA build the agent prepares transactions and a person commits them, and the gate is part of the architecture with no setting to switch it off. The pilot shows whether reviewers catch what the agent gets wrong, and how long that takes. How those approvals are recorded for auditors is covered in AI agent audit trails and human approval for write access.
DATA DURING THE PILOT
Where Pilot Data Should Live
Pilot data should live where production data will live. A pilot run on a cloud endpoint proves nothing about an on-premise deployment, and it may break the rules the production system is built to respect. Under UK GDPR, Singapore’s PDPA and the UAE’s PDPL, testing with real customer records is processing personal data like any other use.
ARSA treats the deployment target as a design input from the first week. Retrieval, inference and any fine-tuning run on infrastructure you control: on premise, air-gapped, or single-tenant hosted in a region you choose. The pilot and the production system therefore share the same data boundary. If data residency, cost at scale and vendor deprecation risk do not apply to you, a cloud API is a reasonable place to run, and the assessment will say so.
AFTER HANDOVER
Who Owns The Test Suite After Go-Live
You should own the test suite after go-live, and the vendor should be contracted to run it on every change. The held-out cases, the adversarial set and the acceptance thresholds define correct behaviour in your business, and they outlast any one model.
Rerun the suite whenever the model is updated, a connector changes, a document template changes or the process itself changes. Add every production error the reviewers catch as a new test case, so the suite grows from real failures. Keep the results with the approval log. Together they show an auditor what the agent was tested against and what it did afterwards.
FAQ
Frequently Asked Questions
How Long Should An AI Agent Pilot Run?
Eight weeks is a sound default for one process on one system. That allows about two weeks for integration and tuning and five or six weeks of live operation to collect enough approved and rejected proposals to measure. The final week is for the acceptance run on the held-out set.
How Many Test Cases Does An Agent Evaluation Need?
A few hundred real cases is a practical minimum for one process, weighted toward the expensive errors. With fewer, a handful of lucky or unlucky documents can move the result by several percentage points.
What Accuracy Should An AI Agent Reach Before Production?
It should beat the measured error rate of the manual process it replaces, while the approval gate holds critical errors at or near zero. No published industry figure fits every process, so the threshold belongs to the process owner and should be fixed before the pilot begins.
How Do You Choose An AI Agent Development Company?
Choose one that charges for a feasibility assessment and will tell you when the answer is no. Ask to see the acceptance criteria from a previous pilot, ask who owns the test suite afterwards, and ask where your data sits during testing.
Can ARSA Run A Pilot Outside Indonesia?
Yes. List prices are in USD and invoicing is available in USD or EUR. Where visa conditions prevent ARSA engineers from travelling, a qualified local partner installs under ARSA supervision.
NEXT STEP
Start With A Two-Week Feasibility Assessment
If you have a process that takes hours of skilled staff time each week and data that cannot leave your network, start with a fixed-price feasibility assessment against your real documents. It ends in a go or no-go and a cost model. See the engagement stages on LLM and AI agent development, or contact ARSA to scope one.