Fixed-scope diagnostic
A passing test suite is not a working phone line.
The Voice Agent Production Readiness Audit: five business days, $5,500 fixed, half to schedule and half on delivery. We audit the wiring, the auth, the state races, the compliance defaults, and the conversation itself, then hand you a ranked report with evidence and a repro for every finding, and a regression suite you keep. Vapi, Retell, ElevenLabs, or homegrown, and we assert against your database, not the vendor's dashboard.
Why the green dashboard lies
The vendor’s suite mocks the part that matters.
A passing "Tool Called" assertion means the model chose the tool. It does not mean an HTTP request left the building. An agent can pass a full behavioural suite while every webhook fails, and the dashboard stays green the whole time.
The console sets defaults you never chose.
Recording on, retention unlimited, endpoints that answer anyone. Those are vendor defaults, not decisions. In several jurisdictions the recording one is not a fine; it is a criminal matter.
Nothing watches the destination.
The code path throws no errors while the table sits empty for weeks. The most expensive failures are silent: the call ends politely and no record of the caller survives.
What five days buys
- 01
Probe the wiring and the config
Automated probes against the agent object and the tool endpoints: routing, parameter passing, auth, HTTP method handling, retry behaviour, recording and retention settings. Read-only, no access to your callers, no writes to your systems.
- 02
Drive real conversations
Behavioural passes for truthfulness and boundaries: invented prices, commitments nobody authorised, regulated advice, prompt injection through caller speech and through tool results.
- 03
Assert against your database, not the transcript
A live session driven end to end, then the check that matters: exactly one row where the booking landed. Simulation tells you the model chose the tool. We check the far end.
- 04
Hand you the report and the suite
Findings ranked by severity, each with evidence and a repro. Criticals fixed or precisely scoped. And a regression suite pinned to your live config that runs on every change, because the finding is worth one release and the suite is worth every one after it.
The catalog
39 probes across nine failure classes, twenty-three of them automated, including the one that matters most: a live session driven end to end and asserted against the destination database. Every probe has cost somebody a call that mattered. Severity one means a real caller loses something: a booking, a matter, their data.
Transport and tool wiring
The pipes. The most expensive class, because a simulated test suite reports it green.
Identity and auth
Endpoints that answer anyone, errors that teach the credential format, configs that echo their own secrets.
Idempotency and state
Retries that book twice, parallel callers who claim one slot, failures that leave no record behind.
Conversation control
Dead air during tool calls, ignored interruptions, emails captured without a spoken read-back.
Truthfulness and commitments
Invented prices, hedged ones, and promises that bind a human who was not on the call.
Adversarial input
Injection through caller speech, injection through tool results, social engineering for internal detail.
Failure and degradation
Status codes reaching the caller’s ear, retry storms, a concurrency cap with no overflow path.
Data protection
Consent, retention, PII in logs, and processing jurisdictions nobody declared.
Observability and evidence
Orphaned transcripts, silent-death writers, and releases with no gate between an edit and production.
One finding, exactly as the report renders it
This is a real finding from the first run against our own line, formatted the way every finding in your report arrives. Nothing here is a mockup.
- Evidence
platform_settings.privacy.record_voice: truewhile the agent's opening turn carried no recording disclosure. Recording was a console default, not a decision.- Impact
- In all-party-consent jurisdictions this is unlawful recording of every caller who reaches the line. In several it is a criminal matter rather than a fine, and the exposure accrues per call.
- Repro
- One read of the agent configuration plus one call to the number. The opening turn is the disclosure surface; listen for it.
- Fix
- Disclosure added to the opening turn, a caller-objection path added to the prompt (stop recording, continue voice-only, honour it in the follow-up), and the fix pinned by a regression test so a console edit cannot silently undo it. Re-run the same night: clean.
We ran it on our own line first
Our front desk is a production voice agent on a real phone number. The first full run of this audit against it surfaced six findings,3 of them severity one: recording with no disclosure in the opening turn, retention set to unlimited, and a platform behaviour that echoes credentials on a config read. Two were fixed the same night and verified by re-running the harness; the credential was rotated. A fourth severity-one claim did not survive our own verification pass, so it was downgraded instead of reported. That is the standard the report holds.
Questions buyers actually ask
How do you audit without touching production?
The config and HTTP layers are read-only: assertions against the agent object and unauthenticated probes of the tool endpoints. Live-session probes that write anything run against a quarantined clone of your agent, or with marked test identities agreed in advance. Nothing we run creates load your callers would feel.
Which platforms do you cover?
Vapi, Retell, ElevenLabs, Bland, or a homegrown stack. The catalog is organized by failure class, not by vendor, and the probes adapt to wherever your agent runs. We run our own production line on this discipline daily.
What do we keep when the five days are over?
The ranked report with evidence and a repro for every finding, the agreed critical fixes shipped or precisely scoped, and the regression suite pinned to your live configuration. The suite is the deliverable that outlives the engagement: a finding is worth one release, the suite is worth every release after it.
What do you need from us to start?
Read access to the agent configuration, the tool endpoint URLs, and a staging target or clone for the probes that write. Most teams grant all three in an afternoon, and the audit starts the next business day.
How is this different from our platform’s built-in test suite?
Vendor test suites simulate tool execution, so a green suite proves the model chose the tool and never that a request left the building. We assert at the destination: the database row, the calendar entry, the webhook delivery. That is the layer where production actually fails.
Five days from access to ranked report.
$5,500 fixed, half to schedule and half on delivery, no retainer. You get the findings, the fixes or exact scoping for the criticals, and the regression suite. Thirty minutes on your agent first, free, and you leave with direction whether or not you hire us.