Configuring Your Voice Agent for Speed

How fast a voice agent feels has less to do with the underlying model than with a handful of specific levers: what the agent says while it's working, how much it has to read and decide on each turn, and how the voice line itself detects when a caller has stopped talking. This guide covers all three.

Say something before a slow tool call

Dead air while a tool runs reads as the agent hanging up or freezing, even if the model itself is fast. The fix is to instruct the agent directly in its prompt to narrate as it is calling a tool that takes noticeable time:

While calling `check_order_status`, say something like:
"Let me look that up for you now."

Keep prompts short

Longer prompts cost first-token time and often reduce instruction-following. If you have a large knowledge base, rely on retrieval (vector search / Auto-RAG) for the few relevant chunks per turn instead of pasting everything into the prompt. Aim to keep prompts under 7,000 tokens for best results and reduced hallucination risk.

Constrain the tool set

Ten tools available at once invites wrong or slow choices — the model has to reason over all of them on every turn just to pick the right one. Use tool filters to expose only what's reachable from the current point in the conversation, or split into a multi-agent team so each agent only ever sees the tools relevant to its own job (see the Prompt Engineering Guide).

Let Auto-RAG fetch before the model has to decide

When a retrieval tool is marked as forced (force_tool_calls), the platform runs it before the LLM call and injects the results directly as context — the model never spends a turn deciding whether to call it, waiting on it, then deciding what to do with the result. If a tool's output is something the agent needs on every relevant turn rather than only sometimes, forcing it removes a full decision-and-round-trip cycle.

Note: While this can improve speed, it can also increase unnecesary noise to the agent. Use this selectively when knowledge sources are narrow in scope.

Tune End of Turn Wait Time

This setting controls how long the voice line waits after the caller stops talking before treating their turn as over. It's a direct speed/safety tradeoff:

Lower value: the agent responds faster, but risks cutting the caller off mid-thought — especially if they pause to think or stutter.
Higher value: safer against interruptions, but every turn feels slower, since the agent is waiting longer before it even starts thinking about a response.

Tune this per use case: a quick order-status line can run tighter than a line where callers are expected to explain something complex.

Note: this is configured at the voice line level (not in the prompt) and is owned outside chat-agents-api — set it in the platform's voice line settings.

Tune Background Denoising

This is primarily an audio-quality setting, but it affects perceived speed too: when background noise leaks into the audio, speech recognition can misfire on turn detection — either cutting the caller off early because it mistook noise for silence, or failing to recognize the caller has stopped talking at all. Either failure mode reads as the agent being slow or confused, even when the model itself responded instantly. If callers are on noisy lines (retail floor, warehouse, driving), check this setting alongside End of Turn Wait Time rather than assuming a slow-feeling call is a prompt or model problem.

Note: like End of Turn Wait Time, this is configured at the voice line level, not in the prompt.

Was this article helpful?