The Forefront of Voice AI: The Future of Voice Agents Pioneered by OpenAI

In recent years, applications and services utilizing large language models (LLMs) such as ChatGPT have appeared one after another, and text-based conversational agents have become widely used. However, for many people, communication where they actually speak with their "voice" and listen to the other party with their "ears" is the most natural method. Agents capable of voice input and output, so-called "voice agents," have the potential to significantly enhance the presence and real-time nature of conversations, and are an important area for creating new user experiences.

OpenAI has provided powerful text-based models (the GPT series) until now, but is now aiming for a similarly high-quality conversational experience with voice. In this livestream themed "Audio Models in the API," new models for speech recognition (Speech to Text: STT) and speech synthesis (Text to Speech: TTS) were introduced, along with SDK updates for combining them to build smooth voice agents. In this article, we will organize the content in an easy-to-understand manner and explain it with concrete demos, implementation examples, and use cases.

1. Overview of New Audio Models

1-1. Introduction of GPT-40 Transcribe and GPT-4 Mini Transcribe

First introduced were the new generation speech recognition models, "GPT-40 Transcribe" and "GPT-4 Mini Transcribe." These were announced as a group of models that achieve higher precision speech recognition than the previous Whisper series.

• GPT-40 Transcribe

  • Improved recognition accuracy in all languages compared to conventional Whisper

  • Price is "$0.06 (6 cents) per minute," equivalent to existing Whisper

• GPT-4 Mini Transcribe

  • Achieves smaller size and higher speed through distillation of larger models

  • Price is "$0.03 (3 cents) per minute," half the price of GPT-40 Transcribe

These are trained on large-scale audio data and are characterized by high recognition performance even in noisy environments. Also, the fact that they support multiple languages is a very major appeal.

1-2. Additional Features: Streaming Support and Noise Removal

With the provision of these models, the following new features and enhancements have also been implemented.

• Support for streaming input
  Developers can have continuous audio data recognized in real-time. As long as the user continues to speak, the model can immediately convert it to text and return it, minimizing response wait times.

• Noise cancellation and voice activity detection
  The model side is equipped with noise cancellation functionality, making it difficult for recognition performance to degrade even in noisy environments. Also, the model automatically detects when the user is speaking and performs processing (Voice Activity Detection) to return text in appropriate units.

This allows developers to build smooth, high-precision voice recognition systems without having to write complex processing logic themselves.

2. Possibilities of Voice Agents

2-1. What is a voice agent?

A voice agent is a general term for a system that takes voice as input and has an agent (AI) return a response according to the content of that input. It can be said to be a voice version of the "chatbots" often seen in text-based chat. For example, it is a mechanism where when a user makes an inquiry through a phone or voice UI device, the AI immediately understands the content and responds or handles it.

In the livestream, OpenAI's Jeff stated, "There are already many text-based agents, but by making them voice-enabled, even more natural communication can be realized." Furthermore, the outlook was presented that "current voice models are evolving rapidly, and utilization at a level comparable to text-based ones can be expected."

2-2. Benefits of introducing voice agents

1. Improvement of user experience
   Interaction via voice is intuitive, and users who are not accustomed to reading or typing can use it easily.

2. Multilingual support
   It is possible to utilize the results of multilingual learning cultivated in the Whisper series and GPT-40 Transcribe, and it can smoothly support an international user base.

3. Operational Efficiency
   Automating customer support and reducing the labor required for phone responses can significantly reduce costs and human burden for companies.

3. New Text to Speech Model: GPT-4 Mini TTS

3-1. How to Specify Speech Styles

During the livestream, Yaroslav demonstrated the "GPT-4 Mini TTS." By accessing the website "openai.fm," you can not only try out several pre-prepared voice styles but also specify your own text and style (tone, emotion, speaking manner) to generate audio.
For example, if you instruct it to "speak like an excited scientist," it will generate an audio output with a unique character like the following.

Conversely, if you instruct it to "speak in a calm tone, gently," you will get audio with a completely different tone.

3-2. TTS Pricing and Developer Guide

The usage fee for GPT-4 Mini TTS has been announced as "$0.01 (1 cent) per minute." The sample code provided by OpenAI introduces how to easily call the API using Python or JavaScript, and it is possible to perform speech synthesis by simply following steps like the ones below.

1. Setting up the API key

2. Specifying text input (the string you want to synthesize) and speech style (voice tone, etc.)

3. Receiving the synthesized audio file or streaming playback

4. Agents SDK Update: Turning Text Agents into Voice Agents

4-1. Flexible Architecture via Chaining Methods

According to the livestream, there are two main methods for building voice agents.

1. An integrated voice model that outputs audio directly from voice input

   • While fast responses can be expected, fine-tuning is somewhat more difficult compared to text processing.

2. A chain method of Speech to Text → LLM → Text to Speech

   • STT converts user audio to text, LLM (e.g., GPT-4) analyzes intent and generates a response, and TTS converts the response back into audio

   • If you already have a text-based agent, this method can be easily extended

   • High reusability and can leverage high-precision language processing

This time, OpenAI updated the "Agents SDK" to strongly support the latter chain method, providing a mechanism to convert text agents into voice agents with minimal code modifications.

4-2. Code Implementation Example: Voice Support with Just a Few Lines of Changes

In the example Yaroslav showed as a demo, for an existing customer support text agent, voice input and output were realized by simply adding or changing just a few lines as follows.

1. Adding logic to receive and buffer voice input data

2. Send the received audio to the "Voice Pipeline" and pass it through the STT→LLM→TTS pipeline

3. Return the generated audio to the client side and play it automatically

The agent's tracing (execution log analysis) function has also been expanded, allowing voice input, response audio, and tool call timing to be visualized and played back on the UI. This is a major advancement, as it makes debugging easier for developers and enables sophisticated, smooth voice interaction for users.

5. Use Cases and Prospects

5-1. Education and Learning Field

Voice agents are easily applicable to foreign language learning and pronunciation training; for example, a system can be envisioned where if a user speaks in English, the AI provides corrections and feedback via voice. The livestream emphasized that "you can learn a language through interactions close to real conversation," which has the potential to significantly lower the hurdles for multilingual learning in the future.

5-2. Customer Support and Call Centers

By combining OpenAI's new model groups with existing support chat and call center automated response systems, a more sophisticated conversational experience can be expected. In particular, because noise cancellation and multi-speaker support are well-developed, it will be easy to use as a replacement for or supplement to telephone help desks.

5-3. Creative Uses

Like the contest project on "openai.fm," entertainment and creative audio productions that utilize speech synthesis are also attracting attention. It is also impressive that many developers are experimenting with and playing with unique voice styles, such as the event featuring a "special edition radio" introduced at the end of the livestream.

The new speech recognition models (GPT-4o Transcribe, GPT-4 Mini Transcribe) and speech synthesis model (GPT-4 Mini TTS) introduced this time are both highly accurate and scalable, making the implementation of voice interfaces significantly easier. The ability to switch existing text chat agents to voice support while maintaining their original logic is a huge benefit for developers as well.

OpenAI's Agents SDK provides a workflow that incorporates STT and TTS with minimal code changes and supports real-time streaming. It is also equipped with robust tracing functions for debugging and log verification, making it easy to visualize agent behavior and effective for enhancing the value provided to users.

In the future, these voice models are expected not only to become even higher-performing but also to integrate with various tools and platforms. Widespread adoption is expected in diverse fields such as education, healthcare, public services, and the entertainment industry. Why not seize this opportunity as a developer to explore the possibilities of new voice agents?

Related Articles