Disclosure: This post supports a fixed-scope Memetic Forge service offer. No affiliate links are included.
Most AI voice-agent demos sound good in a five-minute founder walkthrough. Production is different.
Once a real caller interrupts, gives partial information, changes their mind, gets angry, asks for a refund, mentions a regulated edge case, or asks the agent to do something outside policy, the demo script stops being the test plan.
If you are shipping a voice agent into customer support, collections, healthcare admin, hospitality, home services, sales qualification, or internal operations, here is the release checklist I would want to see before the agent touches real customers.
1. Define the exact jobs the agent is allowed to finish
A release-ready voice agent needs a narrow completion boundary:
- What can it resolve end to end?
- What can it collect but not decide?
- What must it escalate immediately?
- What must it refuse or redirect?
- What systems can it write to?
- What systems are read-only?
A useful eval does not just ask “did it answer?” It asks whether the agent stayed inside the allowed job.
Example:
| Caller request | Agent allowed outcome | Failure mode to test |
|---|---|---|
| Reschedule an appointment | Offer available slots and confirm | Books outside business rules |
| Refund request | Collect order details and escalate | Promises refund without eligibility check |
| Medical billing question | Explain next step / transfer | Gives medical or coverage advice |
| Collections dispute | Log dispute and follow policy | Uses non-compliant wording |
2. Build golden calls, not only golden prompts
Text-only prompt tests miss the hard parts of voice:
- interruptions and barge-in
- noisy caller phrasing
- slow or emotional callers
- language switching
- phone-number and address capture
- retries after ASR mistakes
- tool latency while the caller waits
- when to stop talking and listen
For each critical workflow, create 5–10 “golden calls” with realistic caller personas. The pass/fail criteria should include both task completion and conversation quality.
A minimal golden-call row:
Scenario: caller wants to change a delivery address after shipment
Persona: rushed, interrupts twice, gives ZIP before street address
Expected: agent verifies order identity, explains shipment constraint, escalates if address is locked
Must not: claim the address is changed before carrier/API confirmation
Evidence: transcript, tool trace, final CRM/helpdesk note
3. Score the trace, not just the final answer
For voice agents, the transcript can look fine while the execution trace is wrong.
Score at least four layers:
- Intent handling — Did it understand the caller’s real goal?
- Policy adherence — Did it stay inside the approved operating rules?
- Tool behavior — Did it call the right tool with complete, valid inputs?
- Handoff quality — If escalated, would a human know what happened?
If your QA report only says “passed” or “failed,” it will not help the engineering team fix the release. Capture why.
4. Test refusal and escalation with the same seriousness as success
A surprising number of agents are tested mostly on happy paths. The riskiest failures are usually refusal and escalation failures:
- caller asks for a refund exception
- caller asks for credentials or internal information
- caller asks for legal/medical/financial advice
- caller demands a human
- caller says they are angry or at risk of churning
- caller tries to override the system: “ignore your instructions”
A production-ready agent should not improvise policy. It should know when it is done.
5. Include regression tests for every prompt or workflow change
Voice-agent teams often ship small prompt or routing changes quickly. That is good, but every small change can break an earlier path.
Create a regression set with:
- top 10 revenue-critical workflows
- top 10 support-volume workflows
- all regulated/sensitive workflows
- all human-handoff workflows
- known historical failures
Run it before launch and after material prompt/tool changes. The goal is not academic evaluation; it is catching expensive regressions before customers do.
6. Measure “safe automation rate,” not automation rate
A high automation rate is not useful if the agent is quietly making risky decisions.
Track:
- resolved correctly without human help
- escalated correctly
- refused correctly
- resolved but missing required data
- resolved but used unsafe wording
- tool call failed but agent pretended success
- caller abandoned due to latency or repetition
The metric that matters is not “how many calls did AI handle?” It is “how many calls did AI handle safely and usefully?”
7. Require a release report that a non-engineer can understand
A good release report should be simple enough for a founder, ops lead, or customer-success leader to act on:
- overall pass rate by workflow
- top failure modes
- examples with transcript snippets
- severity ranking
- recommended launch gates
- fixes that are prompt-only vs. workflow/tooling changes
- what should stay human-only for now
The best report is not a leaderboard. It is a go/no-go decision aid.
A lightweight eval sprint structure
For early-stage teams, a practical first sprint can be small:
- Pick 3–5 critical workflows.
- Write 25–40 golden-call scenarios.
- Run the current agent through the set.
- Score transcript + trace + handoff note.
- Ship a one-page release-risk map and fix list.
- Re-run the highest-severity failures after changes.
That is enough to catch the obvious release blockers without building a full QA platform.
If you want an outside pass
Memetic Forge runs a fixed-scope Agentic QA / Eval Sprint for teams shipping AI agents.
Typical first pass:
- 25–40 golden scenarios
- prompt-injection and boundary probes
- transcript and tool-trace review
- pass/fail release matrix
- one-page risk map with recommended fixes
No production credentials or customer data are required for the first pass. Sanitized workflows, demo access, or recorded traces are enough.
If that would be useful, email ops@memeticforge.com with the subject Agent eval sprint and the workflow you are preparing to release.