Evaluate the approval workflow, not just the agent
Accuracy alone cannot establish operational readiness. Test exceptions, reviewer decisions and the audit trail.
Executive summary
- Measure missed exceptions separately from false positives.
- Human approval is a workflow with its own evidence and latency.
- Draft system output must remain bounded by permission and review.
Define the decision boundary
State what the software may read, suggest and draft. Assign a named human role to consequential action. For invoice workflows, extraction is distinct from deterministic order checks, exception resolution and authorized posting.
Build a versioned evaluation set
Use authorized synthetic examples covering normal cases, ambiguous fields, duplicate invoices, missing receipts and inconsistent purchase order records. Keep expected outcomes and adjudication notes separate from model output.
Measure both error directions
A missed material exception and an unnecessary review have different operating consequences. Track each against human adjudicated labels. Report coverage and abstention so a model is not rewarded for answering unsupported questions.
Evaluate the reviewer path
Measure approval latency, review effort, source visibility and override reasons. Verify that rejected drafts cannot reach writeback, permission failures are logged and the audit connects source, version, suggestion and approver.
Reevaluate when the system changes
Changes to source formats, prompts, models, permissions or business rules can change behavior. Run the same regression set and record the new limitations. Pilot thresholds should be agreed with the operating owner, not invented from a benchmark.
Decision sequence
- 01
Bound
Define authorized inputs and draft outputs.
- 02
Label
Human adjudicators record expected decisions.
- 03
Compare
Count missed exceptions and false positives.
- 04
Review
Test approval, rejection and escalation.
- 05
Audit
Trace sources, versions and decisions.
Illustrative application. No customer result.
Synthetic evaluation illustration. A fictional invoice is matched to an order but lacks receipt evidence. A three way check should route it to review, not infer that delivery occurred.
The reviewer records the unresolved evidence and rejects posting. This describes a test expectation, not a deployed finance integration or measured customer result.
Action checklist
- Name the human approver and stop authority.
- Version the test set and expected labels.
- Measure accuracy, false positives and missed exceptions separately.
- Test rejected access, rejected drafts and audit completeness.
- Include review latency and effort in the business baseline.