Voice agents fail in ways text tests never catch — the agent talks over the caller, answers slowly, or drifts off-script. Agent evaluation lets you pin that behavior down: script a conversation, assert what the agent should do, and get a pass/fail plus latency and quality metrics — reproducibly, in CI. A simulated caller connects to your agent over a WebSocket (no phone number, no browser, no SIP), speaks your scripted turns with text-to-speech, and the agent’s replies are transcribed and graded. The run exits 0 only if every assertion passes, and writes a JSON report with per-turn metrics.
Evaluation runs against a real agent over the same runtime path a live call uses — the same speech-to-text, language, and speech providers your agent runs with — so a green run reflects real behavior, not a mock.

​
Write a scenario

A scenario is JSON: who the caller is, which agent to test, the scripted turns, and the assertions on each turn (and on the call as a whole). 
{
  "name": "billing_flow",
  "agent_key": "agent_abc",                      // your agent; "" runs the default pipeline
  "user_speech": { "provider": "elevenlabs", "voice": "" },
  "asr":         { "provider": "deepgram" },     // transcribes the agent's replies
  "judge":       { "provider": "openai", "model": "gpt-4o-mini" },
  "turns": [
    {
      "user": "Hi, I was double charged.",       // scripted text (spoken via TTS) — or "wav": "fixtures/x.wav"
      "expect": [
        { "type": "response_within_ms", "ms": 3000 },
        { "type": "regex",  "pattern": "refund|charge" },
        { "type": "judge",  "criterion": "acknowledges the double charge" }
      ]
    }
  ],
  "expectations": [ { "type": "judge", "criterion": "stayed polite throughout" } ],
  "timeouts": { "call_ms": 120000, "turn_ms": 20000 }
}
Set agent_key to the agent you built in the console (or "" to smoke-test the default speech-to-text → language → speech pipeline with no agent configured). A turn is either scripted user text (spoken through the TTS provider) or a wav fixture (PCM16, mono or stereo, any sample rate) when you need a specific voice, accent, or noise profile.

​
Assertion types

Each turn carries expect assertions; the call carries call-level expectations judged against the whole transcript.
​
response_within_ms
{ ms: number }
Latency gate: the gap from the caller’s last speech frame to the agent’s first audio frame must be under ms. Catches a slow agent before your users do.
​
regex
{ pattern: string }
Case-insensitive regex over the agent’s transcript for that turn. Good for must-say phrases, disclosures, or IDs.
​
judge
{ criterion: string }
LLM-as-judge: the configured judge model returns a strict { "pass": bool, "reason": string } verdict against your plain-English criterion. It fails closed — a malformed or flaky judge reply counts as a fail, so a bad judge can never silently green a run.
​
function_call
{ name: string, args_contains?: object }
Assert the agent invoked a tool by the end of the turn (optionally with certain argument values).
Preview — pairs with the runtime’s tool-call surface.

​
Metrics in the report

Every run writes a JSON report with per-turn timings, all measured client-side:
MetricWhat it is
latency_to_first_audio_msCaller’s last speech frame → agent’s first audio frame (the number your callers feel).
agent_turn_duration_msWall-clock length of the agent’s turn.
agent_audio_msHow much audio the agent actually spoke.
greeting latencyConnect → the agent’s first audio out.
call durationTotal scripted call length.
Pair these with the live-call latency breakdown and MOS / jitter / loss to compare test-bench numbers against production.

​
Run an evaluation

1

Point at an agent

Build and save the agent in the console so its config is live, and note its agent_key. To smoke-test with no agent, use agent_key: "" — the default pipeline runs off your provider keys.
2

Provide credentials

The simulated caller reuses the same provider clients your agent uses, so it needs keys for the TTS, ASR, and judge providers in your scenario (e.g. Deepgram, ElevenLabs, OpenAI) — set them in the environment.
3

Run the evaluator

Point the evaluator at your engine and scenario and run it. It connects, plays the scripted call, grades every assertion, and writes the report.
telequick eval run ./scenarios/billing_flow.json --report ./out
Today the evaluator ships as a standalone binary you run against a self-hosted or managed engine; the telequick eval wrapper is the convenience entry point. See Self-Hosted deployment for running it against your own engine.
4

Read the result

The process exits 0 only if every assertion passed; the JSON report lands in your output directory with the per-turn metrics above and, for each failed assertion, the judge’s reason.

​
Gate your CI on it

Because the run is a single command with a meaningful exit code, drop it into CI to block a regression before it ships: 
- name: Evaluate voice agent
  run: telequick eval run ./scenarios/*.json --report ./eval-out
  # non-zero exit fails the job; upload ./eval-out for the per-turn report
Keep a suite of scenarios per agent — one per critical flow (billing, refund, escalation) plus a latency-only smoke scenario. The agent_key: "" default pipeline gives you a zero-config smoke test that needs only provider keys, so CI can catch a broken transport or provider outage without a saved agent.

Turn Detection & Barge-in

Tune what your latency and barge-in assertions are testing.

Tool Calling

The tools that function_call assertions check for.

Runtime Configuration

The agent config an evaluation runs against.

Latency Breakdown

Compare bench latency to production.