Real-time avatar agents: what is inside them

An avatar agent is an assistant with a face and a voice: the user speaks, the avatar listens, thinks and answers in real time, with gestures and lip sync. We built one for a client and it runs in production. This article explains what is inside, without kitchen secrets but with the honesty of someone who has wrestled with every piece.

The pieces

An agent like this is five coordinated systems:

1. Speech to text (STT): transcribes the user while they speak, streaming.
2. The brain (LLM + system prompt): decides what to answer.
3. Retrieval (RAG): feeds the brain the company's knowledge.
4. Text to speech (TTS): turns the answer into audio with a consistent voice.
5. The avatar: renders face, gestures and lip sync over that audio.

Every piece works on its own in any demo. The real project is making them work together, fast, without saying nonsense.

The system prompt: personality is engineering work

The system prompt defines who the agent is: what it knows, what it must never do, how the brand speaks, when it should admit it does not know and when to hand over to a person. A few things we learned:

• Limits first. Before writing "you are a friendly assistant", write what it is forbidden to do: promise prices, invent policies, comment on competitors, step outside its scope.
• Examples, not adjectives. "Friendly yet professional tone" means nothing to a model. Three examples of a question answered well are worth more than a paragraph of adjectives.
• Versioning and regressions. Every prompt change runs through a battery of test questions. An innocent tone adjustment can break a safety limit that took effort to tune.

RAG: answering with your business, not with the internet

Without retrieval, the model answers from its general knowledge and sounds like a brochure. With RAG, we index the client's knowledge (catalog, policies, FAQs, internal documentation) in a vector database; each question retrieves the relevant fragments and the model answers grounded in that base.

The details matter more than they seem: how documents are chunked, which metadata is stored for filtering, and what the agent does when retrieval finds nothing (the right answer is to admit it and offer the human contact, not to improvise).

Latency rules

In text, a two-second wait is forgiven. In a conversation with voice and a face, half a second of silence is noticeable and two seconds kill the illusion. The whole design revolves around that:

• Streaming at every stage: transcription happens while the user speaks, the model starts answering before the full text exists, TTS synthesizes sentence by sentence and the avatar starts with the first audio fragment.
• Tiered models: a small, fast one classifies and answers the trivial; the large one steps in only for substantive questions.
• A millisecond budget per piece, measured in production, not in the demo.

Guardrails and handover to humans

A public-facing agent needs: input and output filters (banned topics, personal data), a confidence threshold below which it does not improvise, full conversation logging for review, and a visible "talk to a person" button. The goal is not for the agent to solve everything: it is to solve the frequent and route the difficult well.

When is it worth it?

An avatar agent pays off when there is a volume of repetitive conversations with real value (first-level support, sales qualification, guided onboarding) and a brand that benefits from having a face. For a contact form, it is overkill: a text assistant with RAG delivers 80% of the value at a fraction of the cost.

If you are considering something like this (with or without the avatar), let's talk. We have walked the full path: system prompt, RAG, voice and production, and we know where the potholes are.