Skip to content
TanStackAI
Media

Realtime Voice Chat

TanStack AI provides a complete realtime voice chat system for building voice-to-voice AI interactions. The realtime API supports multiple providers (OpenAI, ElevenLabs), automatic tool execution, audio visualization, and multimodal input including images.

Overview

Realtime voice chat differs from text-based chat in several key ways:

  • Bidirectional audio - Users speak into a microphone, and the AI responds with synthesized voice
  • Voice Activity Detection (VAD) - Automatically detects when the user starts and stops speaking
  • Interruptions - Users can interrupt the AI mid-response
  • Low latency - Uses WebRTC or WebSocket connections for near-instant communication
  • Multimodal - Supports text input, image input, and tool calling alongside voice

The realtime system follows the same adapter architecture as the rest of TanStack AI:

  1. Server generates ephemeral tokens using realtimeToken() with a provider-specific token adapter
  2. Client connects using RealtimeClient (or useRealtimeChat in React) with a provider-specific connection adapter
  3. Provider adapters handle the protocol differences between OpenAI WebRTC, ElevenLabs WebSocket, etc.

Quick Start

1. Set Up the Server Token Endpoint

The server generates short-lived tokens so your API keys never reach the client:

ts
import { realtimeToken } from '@tanstack/ai'
import { openaiRealtimeToken } from '@tanstack/ai-openai'
import { createServerFn } from '@tanstack/react-start'

const getRealtimeToken = createServerFn({ method: 'POST' })
  .handler(async () => {
    return realtimeToken({
      adapter: openaiRealtimeToken({
        model: 'gpt-realtime-2.1',
      }),
    })
  })

Note: The realtimeToken() function works with any server framework. The example above uses TanStack Start, but you can use Express, Hono, Fastify, or any other framework that can handle HTTP requests.

2. Connect from the Client (React)

tsx
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'

function VoiceChat() {
  const {
    status,
    mode,
    messages,
    connect,
    disconnect,
    pendingUserTranscript,
    pendingAssistantTranscript,
    inputLevel,
    outputLevel,
  } = useRealtimeChat({
    getToken: () => fetch('/api/realtime-token', { method: 'POST' }).then(r => r.json()),
    adapter: openaiRealtime(),
    instructions: 'You are a helpful voice assistant.',
    voice: 'alloy',
  })

  return (
    <div>
      <p>Status: {status}</p>
      <p>Mode: {mode}</p>
      <button onClick={status === 'idle' ? connect : disconnect}>
        {status === 'idle' ? 'Start Conversation' : 'End Conversation'}
      </button>
      {pendingUserTranscript && <p>You: {pendingUserTranscript}...</p>}
      {pendingAssistantTranscript && <p>AI: {pendingAssistantTranscript}...</p>}
      {messages.map((msg) => (
        <div key={msg.id}>
          <strong>{msg.role}:</strong>
          {msg.parts.map((part, i) => (
            <span key={i}>
              {part.type === 'text' ? part.content : null}
              {part.type === 'audio' ? part.transcript : null}
            </span>
          ))}
        </div>
      ))}
    </div>
  )
}

Providers

OpenAI Realtime

OpenAI's realtime API uses WebRTC for low-latency voice communication.

Server (token generation):

ts
import { realtimeToken } from '@tanstack/ai'
import { openaiRealtimeToken } from '@tanstack/ai-openai'

const token = await realtimeToken({
  adapter: openaiRealtimeToken({
    model: 'gpt-realtime-2.1',
  }),
})

Client (connection):

ts
import { openaiRealtime } from '@tanstack/ai-openai'

const adapter = openaiRealtime()

Environment variables: OPENAI_API_KEY

Available models:

ModelDescription
gpt-realtime-2.1Latest full realtime model (default)
gpt-realtime-2.1-miniLatest smaller, faster realtime model
gpt-realtime-2Previous full realtime model
gpt-realtime-1.5Earlier full realtime model
gpt-realtimeOriginal full realtime model
gpt-realtime-miniOriginal smaller, faster realtime model

Available voices: alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, cedar

ElevenLabs Realtime

ElevenLabs uses WebSocket connections and requires an agent configured in their dashboard.

Server (token generation):

ts
import { realtimeToken } from '@tanstack/ai'
import { elevenlabsRealtimeToken } from '@tanstack/ai-elevenlabs'

const token = await realtimeToken({
  adapter: elevenlabsRealtimeToken({
    agentId: 'your-agent-id',
  }),
})

Client (connection):

ts
import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'

const adapter = elevenlabsRealtime()

Environment variables: ELEVENLABS_API_KEY, ELEVENLABS_AGENT_ID (optional)

Voice Activity Detection (VAD)

VAD controls when the system detects that you've started and stopped speaking. Three modes are available:

ModeHow it worksBest for
serverThe provider detects speech server-side using audio energy levelsDefault — simple, low client complexity
semanticDetects end-of-utterance using semantic cues like pauses and sentence completionNatural conversation — avoids cutting you off mid-sentence
manualYou call startListening() / stopListening() explicitlyPush-to-talk interfaces

Set the VAD mode when creating the hook:

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'

const { startListening, stopListening, updateSession } = useRealtimeChat({
  getToken,
  adapter: openaiRealtime(),
  vadMode: 'manual', // or 'server' or 'semantic'
})

With manual VAD mode, use push-to-talk style interactions:

tsx
<button onMouseDown={startListening} onMouseUp={stopListening}>
  Hold to talk
</button>

You can switch VAD mode at runtime without reconnecting, via updateSession:

ts
updateSession({ vadMode: 'semantic' })

For semantic VAD, configure eagerness to control how long the model waits before deciding you've finished speaking:

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'

const chat = useRealtimeChat({
  getToken,
  adapter: openaiRealtime(),
  vadMode: 'semantic',
  semanticEagerness: 'low', // waits longer before detecting end-of-speech
})

Tools

Realtime sessions support client-side tools. Define tools using the standard toolDefinition() API and pass their client implementations:

ts
import { toolDefinition } from '@tanstack/ai'
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'
import { z } from 'zod'

const getWeatherDef = toolDefinition({
  name: 'getWeather',
  description: 'Get weather for a location',
  inputSchema: z.object({
    location: z.string().meta({ description: 'City name' }),
  }),
  outputSchema: z.object({
    temperature: z.number(),
    conditions: z.string(),
  }),
})

const getWeather = getWeatherDef.client(async ({ location }) => {
  const res = await fetch(`/api/weather?location=${location}`)
  return res.json()
})

// Pass tools to the hook
const chat = useRealtimeChat({
  getToken,
  adapter: openaiRealtime(),
  tools: [getWeather],
})

The realtime client automatically executes tool calls and sends results back to the provider. Tool calls appear as tool-call and tool-result parts in messages.

Text and Image Input

In addition to voice, you can send text messages and images:

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'
import { base64ImageData } from './assets'

const { sendText, sendImage } = useRealtimeChat({ getToken, adapter: openaiRealtime() })

// Send a text message
sendText('What is the weather like today?')

// Send an image (base64 data or URL)
sendImage(base64ImageData, 'image/png')

Audio Visualization

useRealtimeChat exposes audio analysis data for building level meters, waveforms, and spectrum analyzers.

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'

const {
  inputLevel,    // 0–1 normalized microphone level
  outputLevel,   // 0–1 normalized speaker level
  getInputFrequencyData,   // Uint8Array — FFT bins for spectrum analyzer
  getOutputFrequencyData,
  getInputTimeDomainData,  // Uint8Array — waveform samples for oscilloscope
  getOutputTimeDomainData,
} = useRealtimeChat({ getToken, adapter: openaiRealtime() })

The inputLevel and outputLevel values update on every animation frame while connected, making them suitable for driving CSS animations or canvas visualizations.

Simple level meter:

tsx
<div style={{ width: `${inputLevel * 100}%`, height: 4, background: 'green' }} />

Pulsing audio indicator:

tsx
function AudioIndicator({ level }: { level: number }) {
  return (
    <div
      style={{
        width: 40,
        height: 40,
        borderRadius: '50%',
        transform: `scale(${1 + level * 0.5})`,
        backgroundColor: `rgba(59, 130, 246, ${0.3 + level * 0.7})`,
        transition: 'transform 0.1s ease',
      }}
    />
  )
}

Spectrum analyzer using canvas:

ts
function drawSpectrum(canvas: HTMLCanvasElement) {
  const ctx = canvas.getContext('2d')!
  const draw = () => {
    const data = getInputFrequencyData()
    ctx.clearRect(0, 0, canvas.width, canvas.height)
    const barWidth = canvas.width / data.length
    data.forEach((value, i) => {
      const height = (value / 255) * canvas.height
      ctx.fillRect(i * barWidth, canvas.height - height, barWidth - 1, height)
    })
    requestAnimationFrame(draw)
  }
  draw()
}

Interruptions

Users can interrupt the AI while it's speaking:

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'

const { interrupt, mode } = useRealtimeChat({ getToken, adapter: openaiRealtime() })

// Programmatically interrupt
if (mode === 'speaking') {
  interrupt()
}

With server or semantic VAD, interruptions happen automatically when the user starts speaking. Interrupted messages are marked with interrupted: true in the messages array.

Using RealtimeClient Directly

For non-React applications or more control, use RealtimeClient directly:

ts
import { RealtimeClient } from '@tanstack/ai-client'
import { openaiRealtime } from '@tanstack/ai-openai'

const client = new RealtimeClient({
  getToken: () => fetch('/api/realtime-token', { method: 'POST' }).then(r => r.json()),
  adapter: openaiRealtime(),
  instructions: 'You are a helpful assistant.',
  voice: 'alloy',
  onMessage: (message) => {
    console.log(`${message.role}:`, message.parts)
  },
  onStatusChange: (status) => {
    console.log('Status:', status)
  },
  onModeChange: (mode) => {
    console.log('Mode:', mode)
  },
})

// Connect
await client.connect()

// Send text
client.sendText('Hello!')

// Subscribe to state changes
const unsub = client.onStateChange((state) => {
  console.log('Messages:', state.messages.length)
})

// Disconnect when done
await client.disconnect()

// Clean up
client.destroy()

Using ElevenLabs

TanStack AI supports ElevenLabs as an alternative realtime voice provider. The client API is identical — swap the adapter and token function:

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { elevenlabsRealtime } from '@tanstack/ai-elevenlabs'

const { status, messages, connect, disconnect } = useRealtimeChat({
  getToken: () => fetch('/api/elevenlabs-token').then(r => r.json()),
  adapter: elevenlabsRealtime(),
})

Note: ElevenLabs uses agent-based configuration — voice and system prompt are set in the ElevenLabs dashboard or via token overrides. See the ElevenLabs adapter page for setup details.

Next Steps

Advanced

Reference detail you do not need to get this working.

Session Configuration

Configure the realtime session through the hook options:

OptionTypeDefaultDescription
getToken() => Promise<RealtimeToken>requiredFunction to fetch a token from the server
adapterRealtimeAdapterrequiredProvider adapter (openaiRealtime(), elevenlabsRealtime())
instructionsstring—System instructions for the assistant
voicestring—Voice to use for audio output
toolsAnyClientTool[]—Client-side tools with execution logic
vadMode'server' | 'semantic' | 'manual''server'Voice activity detection mode
semanticEagerness'low' | 'medium' | 'high'—Eagerness for semantic VAD
autoPlaybackbooleantrueAuto-play assistant audio
autoCapturebooleantrueRequest microphone on connect
outputModalitiesArray<'audio' | 'text'>—Response modalities
temperaturenumber—Generation temperature
maxOutputTokensnumber | 'inf'—Max tokens in a response

Connection Lifecycle

The realtime client manages a connection lifecycle with these statuses:

StatusDescription
idleNot connected
connectingEstablishing connection
connectedActive session
reconnectingReconnecting after interruption
errorConnection error occurred

And these modes while connected:

ModeDescription
idleConnected but not actively interacting
listeningCapturing user audio input
thinkingProcessing user input
speakingAI is generating a response
ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'
import { useEffect } from 'react'

const { status, mode, error, connect, disconnect } = useRealtimeChat({ getToken, adapter: openaiRealtime() })

// Handle connection
useEffect(() => {
  if (status === 'error' && error) {
    console.error('Connection error:', error.message)
  }
}, [status, error])

Message Structure

Realtime messages use a parts-based structure similar to UIMessage:

ts
import type { RealtimeMessagePart } from '@tanstack/ai'

interface RealtimeMessage {
  id: string
  role: 'user' | 'assistant'
  timestamp: number
  parts: Array<RealtimeMessagePart>
  interrupted?: boolean
}

Each part can be one of:

Part TypeFieldsDescription
textcontentText content from sendText()
audiotranscript, durationMsTranscribed voice content
tool-callid, name, arguments, input, outputTool invocation
tool-resulttoolCallId, contentTool execution result
imagedata, mimeTypeImage sent via sendImage()

Error Handling

Handle errors through the onError callback or the error state:

ts
import { useRealtimeChat } from '@tanstack/ai-react'
import { openaiRealtime } from '@tanstack/ai-openai'
import { getToken } from './token'

const { error } = useRealtimeChat({
  getToken,
  adapter: openaiRealtime(),
  onError: (err: Error) => {
    if (err.message.includes('Permission denied')) {
      alert('Microphone access is required for voice chat.')
    } else {
      console.error('Realtime error:', err)
    }
  },
})

Best Practices

  1. Token security - Always generate tokens server-side. Never expose API keys to the client.
  2. Microphone permissions - Handle the case where the user denies microphone access gracefully.
  3. Cleanup - Always disconnect when unmounting components. The useRealtimeChat hook handles this automatically.
  4. Instructions - Keep voice assistant instructions concise. Remind the model it's in a voice interface so responses stay conversational.
  5. Tool design - Keep tool descriptions clear and tool outputs small, since results are processed in real time.
  6. Error recovery - Implement retry logic for transient connection failures.
Edit on GitHub
Generations
Text-to-Speech