How to Build a Real-Time AI Avatar (Under 500ms)

A real-time avatar is an AI-driven digital character that responds to voice, text, or motion input with sub-500ms latency. Unlike pre-rendered video avatars, real-time avatars generate lip-synced animation, facial expressions, and speech dynamically during live interaction, enabling two-way conversation for customer support, education, and interactive applications.

That 47-word definition describes what most avatar platforms promise but few deliver. The real-time AI avatar market is projected to grow from $0.80 billion in 2025 to $5.93 billion by 2032 (MarketsandMarkets, 33.1% CAGR), yet every result in the top 10 search results is a product page, a glossary entry, or a forum thread. Not a single tutorial with working code exists.

This guide changes that. In the next 30 minutes, you will build a real-time AI avatar — a live avatar that responds in under 500 milliseconds — fast enough for live conversations, kiosks, and customer support. Whether you are building a voice AI chatbot interface or a kiosk character, you will understand the full streaming pipeline, measure latency at every stage, and deploy a working prototype from a companion GitHub template.

Last updated May 2026. Tested with @mascotbot/react ^0.3.x, OpenAI Realtime, Google Gemini Live, ElevenLabs Conversational AI, and Node.js 18+.

The Problem: Why Latency Kills Avatar Experiences

Research by Jacoby et al. (2024) confirms that humans prefer conversational response delays of 200-500ms. Beyond one second, the interaction feels broken. Beyond three seconds, users disengage entirely.

Pre-rendered video avatar platforms -- and even so-called live avatar solutions -- cannot meet this threshold. They generate video clips server-side: record full audio, render full video, download the result. This sequential pipeline guarantees 2-5 seconds of latency at minimum. HeyGen community forums document users reporting 7-8 second lag times when connecting their streaming avatar to a custom LLM. D-ID's rendering is described as "noticeably slower than other tools" in third-party comparisons.

The frustration is universal. Developers building live event kiosks, voice AI chatbot interfaces, or support bots all discover the same problem: existing platforms were designed for async video content, not live avatar conversation with real-time face animation. The architecture is fundamentally wrong for the use case.

Beyond Presence claims sub-100ms latency for their speech-to-avatar step, but provides no architecture explanation, no pipeline breakdown, and no benchmarking methodology. Claims without proof do not help developers who need to understand where the time goes and how to reduce it.

What You'll Build

By the end of this guide, you will have a working interactive avatar powered by the Mascotbot avatar SDK, 2D Rive animation, ElevenLabs streaming TTS, and an LLM of your choice, responding to voice input in under 500ms total round-trip:

• Sub-500ms end-to-end latency from user speech to avatar response
• 120fps lip-synced 2D character animation driven by a licensed ML model that runs on-device — so audio never round-trips
• Works with any voice AI provider -- ElevenLabs, OpenAI, Google Gemini, or your own backend
• No GPU server required -- client-side rendering means zero infrastructure cost for the avatar layer
• Companion GitHub template -- clone and deploy in 15 minutes

Time required: 30 minutes for the full tutorial. 15 minutes if you clone the template directly.

Prerequisites

Before starting, make sure you have the avatar SDK and chatbot SDK credentials ready for the AI avatar API integration:

1. Node.js 18+ installed -- verify with node --version
2. Mascotbot publishable key -- a browser-safe mascot_pub_… key (use a mascot_dev_… key on localhost) from app.mascot.bot/api-keys, plus a registry token for the private npm install
3. A voice AI provider key -- an OpenAI key for the Realtime API, a Google Gemini key, or an ElevenLabs key + Agent ID. Each provider's free or trial tier works for the tutorial.
4. Basic JavaScript/TypeScript knowledge -- familiarity with React and APIs

For the fastest path to a working demo, see the SDK Quick Start in 10 minutes.

Step 1: Understand the Real-Time Avatar Pipeline

Before writing code, you need to understand the architecture that makes sub-500ms latency possible. This is the section that no competitor explains -- everyone claims numbers, but nobody shows the pipeline.

The 7-Stage Streaming Architecture

Mascotbot uses a hybrid architecture, and the first thing to understand is what is not in the audio path: no Mascotbot server. The lip-sync engine is a trained ML model that Mascotbot licenses and delivers to your app. On first load, the SDK does a short licensing handshake with the Mascotbot edge, which returns a time-boxed license and the model itself as a WebAssembly runtime — a background license refresh keeps it live. From then on that model runs on-device: your app talks to the voice AI provider directly, the provider plays its own audio, and the licensed inference engine reads the same audio stream that is hitting the speakers and infers a mouth-shape frame every ~10ms, entirely in the browser. You get a production-grade model you do not have to build or train, executing locally — so once it has loaded, no audio and no viseme data ever round-trip to a Mascotbot server. The animation capture point is the playback point, so the avatar's mouth is driven by the exact audio the user hears.

Stages 6 and 7 happen locally on the audio the provider already plays, which is why they add sub-10ms instead of a server hop. The licensing handshake and the model download are one-time work at startup; once the model is loaded, the audio and the visemes never round-trip to a Mascotbot server. The edge does three things — authorize the license, deliver the model, meter usage — and none of them sit in your audio path.

In our benchmarks across 1,000 test conversations, this pipeline achieves a median end-to-end latency of 340ms (P50) and 480ms (P95), measured from the moment the user stops speaking to the first frame of avatar response animation. Because the loaded model runs on-device, the entire latency budget is spent on the provider round-trip you already pay for — adding the avatar costs you only sub-10ms of on-device inference and render.

Why Streaming Beats Sequential Processing

The key insight is streaming at every stage. The LLM does not wait to finish the full response before TTS starts. TTS does not wait for the full text before generating audio. Each stage begins processing as soon as partial data arrives from the previous stage.

Traditional video avatar pipelines work sequentially: wait for full speech, transcribe all of it, generate the complete LLM response, synthesize the entire audio clip, then render the full video. This serial approach guarantees multi-second latency.

As Gustavo Garcia's comparative review of 10 leading avatar providers (January 2026) notes, most platforms now use WebRTC via LiveKit or Daily for real-time streaming. The W3C WebRTC specification enables sub-500ms glass-to-glass delivery -- the WebRTC avatar transport layer is fast. The bottleneck is the processing pipeline, and streaming eliminates the wait at each stage.

For comparison, the LiveAvatar framework from Alibaba Group (Huang et al., 2025) achieves 20 FPS on 5 H800 GPUs using a diffusion-based 3D approach. That level of GPU infrastructure is what server-side real-time face animation demands. The 2D Rive approach eliminates the rendering bottleneck entirely by computing animation client-side at 120fps with sub-10ms per frame.

For a deeper look at the viseme mapping pipeline, see the lip sync API tutorial.

Step 2: Set Up the Project and Install the SDK

With the architecture clear, let's build. This step gets you from zero to a 2D animated character rendering in the browser.

Clone the Template

Clone the companion template (15-minute path). This guide builds on the OpenAI Realtime avatar template; the Gemini Live template is wired the same way with a raw-PCM player instead of an element tap.

git clone https://github.com/mascotbot-templates/openai-realtime-avatar.git
cd openai-realtime-avatar

# Configure environment
cp .env.example .env.local
# Edit .env.local:
#   NEXT_PUBLIC_MASCOT_KEY=mascot_pub_xxxxxxxxxxxxxx   # browser-safe publishable key
#   MASCOT_NPM_TOKEN=mascot_xxxxxxxxxxxxxx             # private-registry install token
#   OPENAI_API_KEY=sk_xxxxxxxxxxxxxx                   # stays server-side

pnpm install && pnpm dev

The SDK installs from the private registry npm.mascot.bot, not from a tarball. The template ships a .npmrc that points npm at the registry and reads your token from the environment:

# .npmrc (committed; the token comes from MASCOT_NPM_TOKEN, never hardcoded)
@mascotbot:registry=https://npm.mascot.bot/
//npm.mascot.bot/:_authToken=${MASCOT_NPM_TOKEN}

Start from Scratch

Or start from scratch (full tutorial path). Add the private registry and install the packages — @mascotbot/react plus the optional Rive peer deps you need to render an avatar:

# .npmrc at the project root
echo '@mascotbot:registry=https://npm.mascot.bot/' >> .npmrc
echo '//npm.mascot.bot/:_authToken=${MASCOT_NPM_TOKEN}' >> .npmrc

pnpm add @mascotbot/react @rive-app/react-webgl2 @rive-app/webgl2

Mount one <MascotProvider apiKey> at the top of the app — it owns the single licensed inference client — and render the avatar with <Mascot src>, which draws the Rive canvas by default:

"use client";
import { MascotProvider } from "@mascotbot/react";
import { Mascot, Fit, Alignment } from "@mascotbot/react/rive";

export default function Home() {
  return (
    <MascotProvider apiKey={process.env.NEXT_PUBLIC_MASCOT_KEY!}>
      <Mascot
        src="/mascot.riv"
        artboard="Character"
        stateMachine="mascotStateMachine"
        layout={{
          fit: Fit.Contain,
          alignment: Alignment.Center,
        }}
      />
    </MascotProvider>
  );
}

A single <MascotProvider apiKey> wrapping <Mascot src> is the whole mount. The apiKey is your browser-safe publishable key (mascot_pub_…, or mascot_dev_… on localhost) — it is scoped to your origins and meant to ship in the client bundle. The .riv file must contain an artboard named Character and a mascotStateMachine state machine; the SDK writes only the mouth, is_speaking, and stress inputs and leaves everything else to you.

After this step: A browser window showing a 2D animated character in idle state -- breathing, blinking, waiting to speak.

From our testing with 50+ developers, this step takes under 5 minutes. The most common issue is a missing or environment-mismatched key. If nothing renders, check that NEXT_PUBLIC_MASCOT_KEY is set and that useMascot() reaches status === "ready".

Step 3: Connect Voice AI for Real-Time Speech

Now wire up the audio pipeline so the avatar can listen and respond. This is where you connect microphone capture, the LLM, and streaming TTS.

Wire Up the Streaming Pipeline

You connect the voice AI provider with its own official SDK — @openai/agents-realtime for OpenAI, @google/genai for Gemini, @elevenlabs/client for ElevenLabs. The Mascotbot SDK never sits in that connection. Your only server-side job is to mint a short-lived provider credential so the standing provider key never reaches the browser. For OpenAI Realtime that credential is a single-use client secret:

// app/api/openai/token/route.ts — mints a single-use OpenAI Realtime client secret.
// There is NO Mascotbot endpoint in this path; the standing OPENAI_API_KEY
// never reaches the browser.
export const runtime = "nodejs";
export const dynamic = "force-dynamic";

export async function POST() {
  const key = process.env.OPENAI_API_KEY;
  if (!key) return Response.json({ error: "OPENAI_API_KEY not set" }, { status: 400 });

  const res = await fetch("https://api.openai.com/v1/realtime/client_secrets", {
    method: "POST",
    headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
    body: JSON.stringify({
      session: {
        type: "realtime",
        model: "gpt-realtime",
        instructions: "You are a friendly assistant. Keep responses brief and conversational.",
        output_modalities: ["audio"],
        audio: {
          input: { turn_detection: { type: "server_vad", threshold: 0.5, silence_duration_ms: 500 } },
          output: { voice: "marin" },
        },
      },
    }),
    cache: "no-store",
  });

  if (!res.ok) return Response.json({ error: `OpenAI ${res.status}` }, { status: 502 });
  const json = (await res.json()) as { value: string };
  return Response.json({ clientSecret: json.value });
}

Note the key handling: the provider key (OPENAI_API_KEY) is the only secret that stays on the server. The Mascotbot publishable key lives on <MascotProvider apiKey> in the browser, because it never touches the audio path.

Pipe Audio to the Avatar SDK for Lip Sync

OpenAI Realtime over WebRTC plays its own audio into an <audio> element you supply. You tap that element with createElementTap() — a cross-browser helper (Safari has no HTMLMediaElement.captureStream()) — and hand the resulting MediaStream to useLipsyncStream. The SDK infers visemes locally from that stream:

"use client";
import { useCallback, useRef, useState } from "react";
import { useMascot, createElementTap, type ElementTap } from "@mascotbot/react";
import {
  Mascot, MascotRive, Fit, Alignment,
  useMascotPlayback, useLipsyncStream,
} from "@mascotbot/react/rive";

// STABLE module constant — a fresh object every render reinitializes the
// post-processor and breaks lip sync after the first audio chunk (the #1
// integration bug). Define it once, outside the component.
const NATURAL_LIP_SYNC_CONFIG = {
  minVisemeInterval: 60,
  mergeWindow: 80,
  keyVisemePreference: 0.7,
  preserveSilence: true,
  similarityThreshold: 0.6,
  preserveCriticalVisemes: true,
} as const;

function AvatarWithVoice() {
  const { client, status } = useMascot();
  const playback = useMascotPlayback({
    stream: true,
    enableNaturalLipSync: true,
    naturalLipSyncConfig: NATURAL_LIP_SYNC_CONFIG,
  });

  // The tapped provider voice is the lip-sync source. capture point == playback point.
  const [stream, setStream] = useState<MediaStream | null>(null);
  useLipsyncStream({ client, playback, source: { kind: "mediaStream", stream } });

  const tapRef = useRef<ElementTap | null>(null);

  const start = useCallback(async () => {
    if (status !== "ready") return;

    // 1. Create the tap synchronously in the click, before any await, so its
    //    AudioContext is born running. tap.stream is usable immediately.
    const tap = createElementTap();
    tapRef.current = tap;
    setStream(tap.stream);

    // 2. Mint a single-use client secret server-side.
    const res = await fetch("/api/openai/token", { method: "POST", cache: "no-store" });
    const { clientSecret } = await res.json();

    // 3. Connect OpenAI Realtime over WebRTC into an <audio> we own.
    const { RealtimeAgent, RealtimeSession, OpenAIRealtimeWebRTC } =
      await import("@openai/agents-realtime");
    const audioEl = new Audio();
    audioEl.autoplay = true;
    const agent = new RealtimeAgent({ name: "Assistant" });
    const transport = new OpenAIRealtimeWebRTC({ audioElement: audioEl });
    const session = new RealtimeSession(agent, { transport });
    await session.connect({ apiKey: clientSecret });

    // 4. Tap the self-playing element. It stays audible AND drives the avatar.
    tap.attach(audioEl);
    tap.resume();
  }, [status]);

  return (
    <div style={{ width: "100%", height: "100vh" }}>
      <MascotRive />
      <button onClick={start} disabled={status !== "ready"}>
        {status === "ready" ? "Start Voice Mode" : "Loading…"}
      </button>
    </div>
  );
}

export default function Page() {
  return (
    <MascotProvider apiKey={process.env.NEXT_PUBLIC_MASCOT_KEY!}>
      <Mascot
        src="/mascot.riv"
        artboard="Character"
        stateMachine="mascotStateMachine"
        layout={{ fit: Fit.Contain, alignment: Alignment.Center }}
      >
        <AvatarWithVoice />
      </Mascot>
    </MascotProvider>
  );
}

The pattern is the same for every provider — only how you obtain the MediaStream differs. Self-playing providers (OpenAI Realtime over WebRTC, ElevenLabs Conversational AI) are tapped with createElementTap(). Providers that hand you raw PCM and do not play it (Gemini Live, OpenAI Realtime over WebSocket) feed a createPCMStreamPlayer({ sampleRate }), which plays the audio gap-tolerantly and exposes the same audio as a tappable outputStream:

import { createPCMStreamPlayer } from "@mascotbot/core";

// Gemini Live streams raw base64 PCM16 at 24 kHz and does NOT play it.
const player = createPCMStreamPlayer({ sampleRate: 24000 });
setStream(player.outputStream); // → useLipsyncStream source: { kind: "mediaStream", stream }

session.onmessage = (m) => {
  const b64 = m?.serverContent?.modelTurn?.parts?.[0]?.inlineData?.data;
  if (typeof b64 === "string") player.pushBase64PCM16(b64);
  if (m?.serverContent?.interrupted) player.stop(); // barge-in
};

Never route a self-playing provider through createPCMStreamPlayer — the voice would play twice. The player is only for providers that stream raw PCM. In both cases the audio is captured at the exact point it plays, so nothing is sent to a Mascotbot server and the avatar's mouth always matches what the user hears.

After this step: The avatar listens to your speech, responds through the LLM, and animates its mouth in sync with the audio. You should see the character speaking with natural lip movement.

The streaming avatar API stays fast because the only network round-trip is the one to your voice provider — the lip-sync work is local. For deeper ElevenLabs-specific setup, see how to add a talking avatar to ElevenLabs.

Step 4: Add Expressions and Emotional Responses

With lip sync working, the next step is making the interactive avatar feel alive beyond mouth movement.

Expressions split into two families. Emphasis is SDK-driven: the stress input is one of the three inputs the SDK owns (mouth, is_speaking, stress), and you drive it on the speech envelope through useMascotPlayback().stress([{ offset, stress }]), where offset: 0 means "apply now." Gestures are consumer-fired: a custom gesture trigger is yours, declared on <Mascot inputs={["gesture"]}> and fired by you — the SDK never touches it. The SDK never auto-fires gestures; you decide when a gesture plays.

The natural trigger for both is speech onset, which useLipsyncStream hands you through its onFrame callback. Raise stress and fire one gesture when the assistant starts talking, ease stress back when it stops:

import { useRef } from "react";
import { useMascot } from "@mascotbot/react";
import { useMascotPlayback, useMascotInputs, useLipsyncStream } from "@mascotbot/react/rive";

function AvatarReactions({ stream }: { stream: MediaStream | null }) {
  const { client } = useMascot();
  const playback = useMascotPlayback({ stream: true, enableNaturalLipSync: true });
  const { has, custom } = useMascotInputs(); // custom is never undefined; gate writes with has()
  const speaking = useRef(false);

  useLipsyncStream({
    client,
    playback,
    source: { kind: "mediaStream", stream }, // tapped element or player.outputStream
    onFrame: (f) => {
      const isSpeech = !f.silenceDetected;
      if (isSpeech && !speaking.current) {
        speaking.current = true;
        playback.stress([{ offset: 0, stress: 1 }]);  // SDK-driven emphasis while speaking
        if (has("gesture")) custom.gesture.fire?.();   // consumer-fired one-shot reaction
      } else if (!isSpeech && speaking.current) {
        speaking.current = false;
        playback.stress([{ offset: 0, stress: 0 }]);   // ease back to neutral
      }
    },
  });
  return null;
}

For providers that expose their own turn events, you can fire the gesture there instead. With ElevenLabs Conversational AI, for example, Conversation.startSession({ onModeChange }) reports when the agent begins speaking, so you fire the consumer-owned trigger per turn:

// inside Conversation.startSession({ ... })
onModeChange: ({ mode }) => {
  if (mode === "speaking" && has("gesture")) custom.gesture.fire?.();
},

The actual animations live in the .riv file's state machine — a nod, a wave, an acknowledgment. Driven this way, the avatar emphasizes its delivery while it speaks and reacts at the start of each turn, so it reads as an AI agent avatar engaged in the conversation rather than a static image with a moving mouth.

After this step: The avatar shows natural body language and facial reactions during conversation.

Step 5: Measure and Optimize Latency

You have a working avatar. Now make it fast. This section provides the benchmarking methodology no competitor offers.

Instrument the Pipeline

interface PipelineMetrics {
  urlFetchStart: number;
  urlFetchEnd: number;
  connectionStart: number;
  connectionEstablished: number;
  firstAudioChunk: number;
  firstViseme: number;
}

function useLatencyMeasurement() {
  const metrics = useRef<Partial<PipelineMetrics>>({});

  const mark = (stage: keyof PipelineMetrics) => {
    metrics.current[stage] = Date.now();
  };

  const report = () => {
    const m = metrics.current;
    if (!m.urlFetchStart || !m.connectionEstablished) return null;

    return {
      urlFetchLatency: (m.urlFetchEnd ?? 0) - m.urlFetchStart,
      connectionLatency:
        m.connectionEstablished - (m.urlFetchEnd ?? m.connectionStart ?? 0),
      timeToFirstAudio: m.firstAudioChunk
        ? m.firstAudioChunk - m.connectionEstablished
        : null,
      timeToFirstViseme: m.firstViseme
        ? m.firstViseme - m.connectionEstablished
        : null,
      totalPipeline: m.firstViseme
        ? m.firstViseme - m.urlFetchStart
        : null,
    };
  };

  return { mark, report, reset: () => { metrics.current = {}; } };
}

Instrument each pipeline stage: mark urlFetchStart before minting the provider credential, connectionEstablished once the session connects, firstAudioChunk when the provider's first audio arrives, and firstViseme on the first frame useLipsyncStream's onFrame emits. Because the licensed model runs on-device once it has loaded, timeToFirstViseme measures the on-device inference cost directly — it should be sub-10ms after the audio starts playing.

Optimization Techniques

The highest-impact optimization is credential pre-fetching. Without it, the user clicks "Start" and waits 200-500ms for the credential mint before the provider session even begins. Mint the short-lived credential on page load, refresh it before it expires, and the connection starts instantly:

// Pre-fetch on page load
useEffect(() => {
  fetchAndCacheCredential();
  const interval = setInterval(() => fetchAndCacheCredential(), 9 * 60 * 1000);
  return () => clearInterval(interval);
}, [fetchAndCacheCredential]);

// On start: use the cached credential, fall back to a fresh mint
const start = async () => {
  const credential = cachedCredential ?? (await mintCredential());
  await connectProvider(credential); // OpenAI clientSecret / Gemini token / ElevenLabs signedUrl
};

Additional avatar latency optimization techniques ranked by impact:

1. Pre-fetch the provider credential on page load -- eliminates 200-500ms (Priority 1)
2. Preload Rive assets before the conversation starts -- eliminates 100-300ms of Rive initialization (Priority 1)
3. Use flash models -- gpt-realtime for OpenAI, gemini-3.1-flash-live-preview with thinkingBudget: 0 for Gemini, eleven_flash_v2_5 for ElevenLabs TTS (Priority 1)
4. Create the audio tap inside the click, before any await -- so its AudioContext is born running instead of suspended (Priority 2)
5. Tune the provider's audio chunk size -- smaller chunks reduce latency but increase overhead (Priority 2)

For transparency: our benchmark measures latency from the moment the user stops speaking to the first frame of avatar response animation, using the Performance API in Chrome DevTools. P50 is 340ms. P95 is 480ms. These numbers reflect warm connections -- the first response takes 1.5-3 seconds due to cold start, which the pre-fetching strategy addresses. Note that none of this latency comes from the avatar: once the licensed model has loaded, lip sync is computed on-device and adds only sub-10ms of inference and render.

According to MDN Web Performance APIs documentation, Date.now() provides millisecond precision sufficient for pipeline-stage measurement.

Why 2D Avatars Outperform 3D for Real-Time Use Cases

Every competitor in the real-time avatar space focuses on 3D photorealistic faces: D-ID, HeyGen, Synthesia, Beyond Presence. The 2D avatar SDK and chatbot SDK space is completely unoccupied, and for real-time use cases, there is a strong technical argument for 2D. The AI avatar API approach we use here sidesteps the GPU bottleneck entirely.

The architectural advantage is clear: 2D Rive animations eliminate the server-rendering bottleneck entirely. There is no video generation step, no video encoding, no video streaming, and no video decoding. The avatar animation computes locally in the browser at 120fps with sub-10ms per frame. This is what makes sub-500ms end-to-end latency achievable without expensive GPU infrastructure.

For applications requiring photorealistic human likeness -- personalized video messages, digital twins -- HeyGen and D-ID remain the right choice. But for a brand mascot animated in real time, stylized agents, and character-driven experiences where response speed matters, the 2D approach is faster, cheaper, and more customizable.

Mascotbot's lip sync engine is a deliberate black box: audio goes in, a stream of mouth-shape frames comes out, and the SDK drives them onto the Rive mouth inputs. It is a trained ML model that Mascotbot licenses and delivers as a WebAssembly runtime — you do not hand-map phonemes to mouth shapes, and you do not build or train the inference engine yourself; that production-grade work is what the license buys. The Rive WASM runtime then renders the animation at near-native speed, and vector-based rendering scales to any resolution without quality loss. The one knob you tune is how smoothly the engine merges frames (the natural lip-sync config), which is why it stays a small, stable object.

For the complete 2D Avatar SDK reference, see the 2D Avatar SDK complete guide.

Common Issues and Solutions

Audio Sync Drift Over Long Conversations

Symptom: Lip sync gradually falls out of sync with audio after 5-10 minutes of continuous conversation.

Why it happens: Clock drift between the audio playback timeline and the Rive animation timeline. Small timing differences accumulate over minutes.

Solution: Because the SDK taps the audio at the exact point it plays — capture point equals playback point — there is no second timeline to drift against; the visemes ride the same clock as the sound. Keep enableNaturalLipSync: true on useMascotPlayback for the smoothest merging, and make sure your naturalLipSyncConfig is a stable module constant. If you ever recreate that config object inside the render, playback reinitializes and lip sync breaks after the first chunk — the single most common integration bug.

A common pitfall we have encountered in real deployments is moving the config inline "just to tweak a value" and forgetting it is now a fresh object every render.

High Latency on First Response

Symptom: First avatar response takes 2-3 seconds, but subsequent responses are fast (~340ms).

Why it happens: Cold start. The provider session connection, LLM initialization, TTS model loading, the Mascotbot client reaching status === "ready", and Rive asset loading all happen on the first request.

Solution: Preload everything during page load, before the user interacts:

// Pre-mint the provider credential on mount (eliminates ~200-500ms)
useEffect(() => { fetchAndCacheCredential(); }, []);

// Rive .riv file loads from /public/ -- browser caches it after first visit.
// The MascotProvider initializes the inference client on mount, so it is
// already "ready" by the time the user clicks Start.

The pre-fetching pattern from Step 5 eliminates the credential mint latency. Rive asset files (50-200KB) are cached by the browser after first load, and the Mascotbot client warms up while the page idles.

Viseme Mapping Looks Unnatural for Certain Phonemes

Symptom: Mouth shapes don't match certain sounds, especially sibilants like "s", "z", "sh."

Why it happens: The default merging may smooth over distinctive shapes that matter for your character design, blending sibilants into neighboring mouth shapes.

Solution: Tune the natural lip-sync config. Raise keyVisemePreference so the engine favors distinctive shapes, lower similarityThreshold so it preserves subtle differences, and keep preserveCriticalVisemes: true. Define it as a stable module constant — never inline:

// Module-level constant, defined once outside the component
const CRISP_LIP_SYNC_CONFIG = {
  minVisemeInterval: 60,
  mergeWindow: 80,
  keyVisemePreference: 0.85,   // favor distinctive shapes
  preserveSilence: true,
  similarityThreshold: 0.4,    // preserve subtle differences
  preserveCriticalVisemes: true,
} as const;

// const playback = useMascotPlayback({
//   stream: true,
//   enableNaturalLipSync: true,
//   naturalLipSyncConfig: CRISP_LIP_SYNC_CONFIG,
// });

For the deeper viseme background, see the Lip Sync API guide.

Next Steps

You built a working real-time AI avatar with sub-500ms latency, 120fps lip-synced animation, and streaming voice AI integration. Deploy it as a live avatar for events, embed it as an interactive avatar in your app, or extend it further:

1. Add a talking avatar to ElevenLabs -- deep dive into ElevenLabs-specific voice agent configuration and optimization
2. Lip sync API tutorial -- understand the full viseme pipeline, phoneme mapping, and custom mouth shape configuration
3. 2D Avatar SDK complete guide -- explore the full Mascotbot SDK API, advanced components, and widget system
4. Create a custom brand mascot -- design and import your own character in Rive Editor, replacing the default avatar with your brand's mascot
5. Deploy to a live event kiosk -- take this tutorial's result offline with edge hosting, local fallback, and kiosk-mode configuration

Frequently Asked Questions

How to create a real-time avatar?

To create a real-time avatar, install an avatar SDK like Mascotbot, connect it to a voice AI provider (OpenAI Realtime, Google Gemini Live, or ElevenLabs) using that provider's own SDK, and tap the audio the provider plays into the lip sync engine. Mascotbot's engine is a licensed ML model delivered from the Mascotbot edge on first load, then run on-device, so audio never round-trips. The key is streaming at every pipeline stage -- LLM tokens stream to TTS, and the licensed model infers mouth shapes on-device from the playing audio -- achieving sub-500ms response latency.

How do real-time avatars work?

Real-time avatars work by streaming audio input through a pipeline: speech-to-text converts user voice, an LLM generates a response, text-to-speech creates audio, and a lip sync engine maps that audio to mouth-shape animations called visemes. With 2D Rive animation that engine is a licensed ML model that loads once from the Mascotbot edge and then runs on-device, inferring visemes in the browser from the same audio that plays through the speakers, so the whole pipeline completes in under 500 milliseconds -- every stage streams to the next, and the avatar adds no audio round-trip of its own.

What is an interactive avatar?

An interactive avatar is a digital character that responds to user input in real time through voice, text, or gesture. Unlike pre-recorded video avatars, interactive avatars generate responses dynamically during live conversation, with synchronized lip movement, facial expressions, and speech -- enabling two-way dialogue for support, education, and entertainment.

What is the best realistic avatar AI?

The best avatar AI depends on your use case. For photorealistic human avatars, HeyGen and D-ID lead the market. For real-time 2D animated characters with sub-500ms latency, Mascotbot is the only SDK offering 120fps Rive-powered lip sync. For enterprise video generation, Synthesia dominates. Choose based on latency requirements, character style, and budget.

What are real-time avatar applications?

Real-time avatar applications include AI customer support agents, live event kiosk characters, educational tutors, voice assistant interfaces, gaming NPCs, and brand mascot ambassadors. Any scenario requiring two-way conversation with a visual character benefits from real-time avatars, especially where response latency under one second is critical.