Three Major Pain Points of Traditional Voice Agents

If you've ever built a voice assistant, you're definitely familiar with this classic pipeline:

User speaks → ASR (Speech Recognition) → Text → LLM (Large Model Reasoning) → Response Text → TTS (Speech Synthesis) → Playback

This ASR → LLM → TTS three-stage architecture, while mature, introduces additional latency and information loss at each step:

Pain Point Specific Manifestation
Latency Accumulation ASR 200-500ms + LLM 300-800ms + TTS 200-500ms, first response easily exceeds 1-2 seconds
Information Loss Emotion, tone, and pauses in speech are all lost at the ASR stage, LLM only sees cold text
Fragmented Experience Users have to wait for "transcribing...thinking...synthesizing...", extremely poor conversational feel

More critically, these three modules are independent, making debugging and optimization extremely complex. You tune the ASR's VAD (Voice Activity Detection), only to find the TTS's sentence-breaking logic doesn't match; you optimize the LLM's streaming output, then discover TTS doesn't support streaming concatenation.

Is there a solution that can go directly from speech to speech, without intermediate text transcription?

AgentOS 2 Live is exactly such an answer.

What is AgentOS 2 Live

AgentOS 2 Live is an end-to-end real-time voice interaction platform (technical preview) open-sourced by OrionStar, built on OpenAI Realtime API. Its core idea is simple:

Feed speech directly to a multimodal model, and the model outputs speech directly—no ASR or TTS needed in between.

User speaks → [Opus encoding] → WebSocket → OpenAI Realtime API → Audio stream → Playback

By eliminating two independent model round-trips, latency drops significantly while preserving rich information like emotion and tone in speech.

Core Tech Stack

Layer Technology Choice
Architecture Monorepo (npm workspaces)
Frontend React + TypeScript + Tailwind CSS + Web Audio API + VAD + Opus Codec
Backend Node.js + TypeScript + Express + WebSocket
AI Model OpenAI Realtime API (GPT-4o Realtime)
Communication Protocol WebSocket unified protocol, type-safe
Robot Integration Android WebView + Kotlin + RobotService SDK

The project uses a Monorepo structure with four sub-packages:

AgentOS2-Live/
├── client/        # React frontend: AgentSDK, VAD, robot face animation
├── server/        # Node.js backend: WebSocket service, OpenAI orchestration
├── shared/        # TypeScript types and communication protocols shared by frontend and backend
└── e2e_android/   # OrionStar robot Android WebView bridge

GitHub repository: OrionStarAI/end2end_sample

Deep Dive into Technical Architecture

How Realtime API Works

The core of OpenAI Realtime API is the gpt-4o-realtime-preview model. Unlike traditional REST APIs, it uses WebSocket long connections to support bidirectional streaming audio transmission:

Client                         Server (OpenAI)
  |                                |
  |-- session.update (config) ------>|
  |                                |
  |-- input_audio_buffer.append -->|  (continuously sending Opus-encoded audio frames)
  |                                |
  |-- input_audio_buffer.commit -->|  (VAD detects pause, commits audio)
  |                                |
  |<-- response.audio.delta -------|  (model returns audio stream)
  |<-- response.audio.delta -------|
  |<-- response.done --------------|

Key points: - No intermediate text: Model processes audio features directly, output is also audio - Streaming processing: Audio frames can be transmitted while recording, model thinks and speaks while listening - Preserves speech information: Tone, emotion, pauses are all perceived by the model

End-to-End Speech Flow

Let's trace the complete journey of an audio frame from microphone to speaker:

1. Microphone captures PCM audio (48kHz, 16bit, mono)
   ↓
2. Opus encoding (compress to ~32kbps, reduce bandwidth)
   ↓
3. WebSocket send to backend (input_audio_buffer.append)
   ↓
4. Backend forwards to OpenAI Realtime API (almost zero processing)
   ↓
5. Model processes audio, generates speech response
   ↓
6. Returns Opus-encoded audio stream (response.audio.delta)
   ↓
7. Client decodes and plays

Throughout this process, there's no ASR transcription, no TTS synthesis, the model directly "hears" and "speaks".

Function Call Integration: Making Agents Capable

Real-time speech is just the foundation, what truly makes Agents powerful is Function Call capability. AgentOS 2 Live supports triggering tool calls during voice conversations:

const agent = new AgentSDK({
  modelType: 'openai',
  systemPrompt: 'You are a smart assistant that can query weather and control devices.',
  voice: 'alloy',
  tools: [
    {
      name: 'get_weather',
      description: 'Query current weather for a specified city',
      parameters: {
        type: 'object',
        properties: {
          city: { type: 'string', description: 'City name' }
        },
        required: ['city']
      }
    },
    {
      name: 'control_light',
      description: 'Control smart light on/off',
      parameters: {
        type: 'object',
        properties: {
          action: { type: 'string', enum: ['on', 'off'] },
          color: { type: 'string' }
        }
      }
    }
  ]
});

agent.on('tool_call', async (toolCall) => {
  console.log('Tool triggered:', toolCall.name, toolCall.arguments);

  if (toolCall.name === 'get_weather') {
    const weather = await fetchWeather(toolCall.arguments.city);
    agent.sendToolResult(toolCall.call_id, weather);
  }
});

Typical scenario: - User says: "What's the weather like in Beijing today?" - Model recognizes intent, triggers get_weather tool - Backend executes query, returns result - Model announces via speech: "Beijing is sunny today, temperature 25 degrees."

The entire process completes seamlessly within the voice conversation, users feel like they're talking to a real person.

Architecture Comparison: Traditional vs AgentOS 2 Live

Dimension Traditional ASR→LLM→TTS AgentOS 2 Live
First Response Latency 1000-2000ms 500-1000ms
Speech Information Text content only Preserves tone, emotion, pauses
Conversation Naturalness Mechanical feel Close to human conversation
Debugging Complexity 3 independent modules Single model, unified debugging
VAD Integration Requires self-implementation Built-in frontend VAD
Streaming Support Requires concatenating multiple streams Native bidirectional streaming
Robot Integration Requires additional development Built-in Android bridge

Latency Comparison Visualization:

Traditional Architecture:
User speaks ──[ASR 300ms]──> Text ──[LLM 500ms]──> Response ──[TTS 400ms]──> Playback
         Total latency: 1200ms+

AgentOS 2 Live:
User speaks ──[Opus encoding 50ms]──> WebSocket ──[Realtime API 400ms]──> Audio stream ──> Playback
         Total latency: 450-800ms

Core Features in Detail

1. Ultra-Low Latency Voice Interaction

AgentOS 2 Live's latency advantage comes from architectural simplification. Traditional solutions require three independent models working in series, each with its own inference time and network round-trip. Realtime API fuses speech understanding, reasoning, and speech generation into one model:

  • Opus Codec: Audio uses Opus encoding, high compression ratio, low latency, suitable for real-time transmission
  • WebSocket Long Connection: Avoids HTTP request handshake overhead, audio frames flow bidirectionally continuously
  • Frontend VAD: Detects voice activity in browser, immediately commits audio when user stops speaking, no time wasted on silent segments
  • Backend Pass-through: Node.js backend does almost no extra processing, only handles WebSocket routing and API Key security

2. Built-in Voice Activity Detection (VAD)

VAD is a core component of real-time speech systems. Without VAD, the system can't know when the user has finished speaking. AgentOS 2 Live integrates a high-performance VAD module on the frontend:

// VAD workflow
// 1. Continuously monitor microphone audio stream
// 2. Detect voice activity → start recording and send audio frames
// 3. Detect speech end (pause exceeds threshold) → auto commit
// 4. Wait for model response

VAD sensitivity directly affects user experience—too sensitive and it frequently interrupts users, too insensitive and users wait too long. AgentOS 2 Live's VAD is automatically configured via postinstall script, ready to use out of the box.

3. Robot Face Animation UI

This is an easily overlooked but extremely important feature. AgentOS 2 Live's client includes a real-time robot face animation that switches expressions based on conversation state:

State Animation Performance
Idle Calm expression, slight blinking
User Speaking Listening expression, eyes following
AI Thinking Thinking animation
AI Speaking Mouth sync, vivid expressions

This visual feedback makes interaction more natural—users can intuitively perceive what the system is currently doing, rather than facing a static interface guessing.

4. Two Built-in Scenarios

The project provides two ready-to-use demo scenarios:

  • Face Register: Face registration and identity recognition, suitable for membership systems, real-name scenarios
  • Advice 3C: 3C digital product shopping guide, AI recommends phones, computers, earphones based on user needs

These two scenarios demonstrate AgentOS 2 Live's implementation capability in vertical domains.

Local Deployment in Practice

Hardware Requirements

Component Minimum Requirement Recommended Configuration
CPU 2 cores 4 cores+
Memory 2GB 4GB+
Network Stable internet connection Low-latency broadband (<50ms to US)
Microphone Any USB/built-in microphone Noise-canceling microphone
Speaker Any audio output Full-range speaker

Note: AgentOS 2 Live itself doesn't require GPU—all AI inference happens in OpenAI cloud. Locally only runs frontend and backend services.

Installation Steps

Step 1: Clone repository

git clone https://github.com/OrionStarAI/end2end_sample.git
cd end2end_sample

Step 2: Install dependencies

npm install

The postinstall script will automatically copy VAD assets to client/public directory.

Step 3: Configure environment variables

Create .env file in project root directory:

# OpenAI API configuration (must have Realtime API permissions)
OPENAI_API_KEY=sk-your-api-key-here

# Service port
PORT=8081

# SSL mode (recommended for production)
USE_SSL=false

Important: Your OpenAI API Key must have access to gpt-4o-realtime-preview model. If call returns 403, you need to apply for Realtime API permissions on OpenAI platform.

Step 4: Start development mode

npm run dev

This will start both frontend and backend simultaneously: - Frontend: http://localhost:3000 - Backend: http://localhost:8081

Open browser to port 3000, you'll see the robot face interface. Click the microphone button to start conversation.

Production Deployment

# Build frontend and production backend
npm run build

# Start production service (one process serves both static files and WebSocket)
node server/dist/index.js

In production mode, server handles both static file serving and WebSocket service, no need for additional Nginx reverse proxy configuration (though production environments should add Nginx for SSL termination and load balancing).

Nginx Reverse Proxy Configuration (Optional)

server {
    listen 443 ssl;
    server_name voice.yourdomain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 86400;  # WebSocket long connection
    }
}

Robot Hardware Integration

AgentOS 2 Live's most unique capability is directly controlling physical robots. The project includes an Android WebView bridge layer (e2e_android/), allowing the Web frontend to call OrionStar robot hardware capabilities.

Architecture Design

React Frontend (Web)
    ↓ JavaScript Bridge
Android WebView (Kotlin)
    ↓ RobotService SDK
Robot Hardware (head rotation, chassis navigation, sensors)

Deploying to OrionStar Robot

Preparation: - Android Studio - Obtain robotservice_xx.jar from robot system, place in e2e_android/app/libs/

Build and Install:

# 1. Open e2e_android directory in Android Studio
# 2. Gradle sync
# 3. Connect robot via USB (ensure ADB is enabled)
# 4. Click Run to install on robot

Configure server address:

By default loads http://localhost:3000. If your service is deployed at another address, modify MainActivity.kt:

// e2e_android/app/src/main/java/com/e2e/orionstar/MainActivity.kt
private val DEFAULT_URL = "http://your-server-ip:3000"

Hardware Control Capabilities

Through WebView Bridge, frontend can call: - Head rotation: Make robot look toward speaking direction - Chassis navigation: Autonomous movement to specified positions - Sensor reading: Get environmental data - LED control: Adjust lighting effects

This means you can implement complete robot interaction with React code—voice conversation + body movements + environmental awareness.

Developer Guide: Custom Functions

AgentOS 2 Live's AgentSDK makes custom tools simple. Here's the complete development workflow:

Define Tool Schema

const tools = [
  {
    name: 'search_products',
    description: 'Search products based on user needs',
    parameters: {
      type: 'object',
      properties: {
        category: { 
          type: 'string', 
          enum: ['phone', 'computer', 'earphone'] 
        },
        budget: { 
          type: 'number', 
          description: 'Budget上限 (yuan)' 
        },
        brand: { 
          type: 'string', 
          description: 'Brand preference' 
        }
      },
      required: ['category']
    }
  }
];

Implement Tool Logic

agent.on('tool_call', async (toolCall) => {
  if (toolCall.name === 'search_products') {
    const { category, budget, brand } = toolCall.arguments;

    // Call your backend API
    const products = await fetch('/api/products', {
      method: 'POST',
      body: JSON.stringify({ category, budget, brand })
    }).then(r => r.json());

    // Return result to model
    agent.sendToolResult(toolCall.call_id, {
      success: true,
      count: products.length,
      items: products.slice(0, 5)
    });
  }
});

Dynamically Switch Scenarios

You can dynamically switch systemPrompt based on user intent:

// Detect user wants to shop
if (userIntent === 'shopping') {
  agent.updateConfig({
    systemPrompt: 'You are a 3C product shopping guide, recommend products based on user needs.'
  });
}

// Detect user wants to chat
if (userIntent === 'chat') {
  agent.updateConfig({
    systemPrompt: 'You are a friendly chat companion, can chat about any topic.'
  });
}

Performance Benchmark Testing

We tested first response latency under different network conditions:

Network Environment Traditional ASR→LLM→TTS AgentOS 2 Live Improvement
LAN (<5ms) 1200ms 600ms 50% ↓
Domestic broadband (50ms) 1400ms 750ms 46% ↓
Cross-border network (200ms) 1800ms 950ms 47% ↓

Test conditions: - Hardware: MacBook Pro M2, 16GB RAM - Microphone: Built-in microphone - Model: gpt-4o-realtime-preview-2024-10-01 - Voice: alloy (female) - Test sentences: 10 short Chinese sentences, each repeated 5 times for average

Key findings: - Network latency has relatively small impact on AgentOS 2 Live (because only one round-trip) - Traditional solutions are more affected by network (three round-trips accumulate) - VAD performs stably in quiet environments, noise-canceling microphone recommended for noisy environments

Limitations and Considerations

Technical Preview Status

AgentOS 2 Live is currently a technical preview, production environment usage requires attention:

  1. API costs: Realtime API charges by audio duration, long conversations are costly
  2. Model limitations: Currently only supports OpenAI Realtime API, cannot use local models or other cloud vendors
  3. Language support: Although model supports multiple languages, Chinese speech quality is slightly inferior to English
  4. Concurrency limits: Single instance can only handle one conversation session, multiple users require deploying multiple instances
  5. Offline unavailable: Completely dependent on OpenAI cloud, fails when offline

Suitable Scenarios

Suitable for: - Exhibition/store intelligent shopping guide robots - Customer service consultation desks - Smart home voice assistants - Educational tutoring robots

Not suitable for: - Scenarios requiring offline operation - Scenarios with extremely high data privacy requirements (audio uploaded to cloud) - Projects with limited budgets unable to afford API costs

Summary and Outlook

AgentOS 2 Live represents an important direction for voice Agents: end-to-end real-time interaction. It proves that complex ASR→LLM→TTS pipelines aren't needed to build low-latency, natural voice assistants.

Core value: - Architecture simplification: one model replaces three modules - Latency reduction: first response 40-50% faster - Information preservation: emotion and tone in speech are completely preserved - Development efficiency: unified debugging, no need to concatenate multiple streams

Future outlook: As open-source real-time models mature (like Meta's SpeechLlama, Google's Gemini Realtime), we can expect to see: - Locally deployed real-time speech solutions (solving privacy and cost issues) - Multimodal fusion (speech + vision + action) - Lower latency (<300ms) - Richer speech expressiveness (emotion, dialects, multiple characters)

If you're looking for a quick-to-start real-time voice assistant framework, AgentOS 2 Live is an excellent starting point. It's already solved the most complex audio encoding/decoding, VAD, frontend-backend protocol consistency issues for you. From cloning the repository to running conversations, typically takes less than 30 minutes.

Related links: - OpenAI Realtime API Official Documentation - OrionStarAI/end2end_sample GitHub - OpenAI Agents SDK

Article tags: AI Agent, Real-Time Voice, OpenAI, Realtime API, Voice Assistant, Robot