OpenAI Unveils Realtime API and New Features at DevDay, Secures $6.6 Billion in Funding

OpenAI hosted its annual DevDay conference yesterday, where it introduced the new Realtime API, along with features like prompt caching, vision fine-tuning, and model distillation.

The Realtime API, now in public beta, is built for creating low-latency, multimodal applications. OpenAI showcased how companies like the fitness coaching app Healthify are using it for more natural interactions with its AI coach, and how the language learning app Speak is leveraging it to enable conversational practice in different languages.

Additionally, OpenAI’s Chat Completions API now supports audio input and output, making it possible to build voice-enabled applications that don’t need the low-latency performance of the Realtime API. This allows developers to send text or audio to GPT-4o and receive responses in text, audio, or both.

By combining the Realtime API and the new audio capabilities of the Chat Completions API, developers can create more seamless, natural interactions using a single API call instead of integrating multiple models.

OpenAI also announced new plans to enhance the Realtime API with support for more modalities like vision and video, increased rate limits, official SDKs, prompt caching, and expanded model support.

Another notable update is the launch of prompt caching, which speeds up processing times and reduces costs by 50% when using cached input tokens. This feature is now standard in the latest versions of GPT-4o, GPT-4o mini, o1-preview, and o1-mini, as well as their fine-tuned variants.

The company also introduced vision fine-tuning for GPT-4o, allowing for better image understanding, which can be applied to tasks like advanced visual search, autonomous vehicle detection, and medical image analysis. Until the end of the month, OpenAI is offering 1 million free training tokens per day for vision fine-tuning on GPT-4o.

Lastly, OpenAI announced Model Distillation, enabling developers to use outputs from larger models like GPT-4o to fine-tune smaller, cost-effective models. This feature includes tools to capture input-output pairs, run evaluations, and integrate with fine-tuning capabilities. OpenAI is offering 2 million free training tokens per day on GPT-4o mini and 1 million tokens per day on GPT-4o through the end of the month for developers to explore this feature.

In related news, OpenAI announced today that it has raised $6.6 billion in funding, bringing its valuation to $157 billion. According to CNBC, Thrive Capital led the round with participation from Microsoft, NVIDIA, SoftBank, and others.

“The new funding will allow us to strengthen our leadership in frontier AI research, expand compute capacity, and continue developing tools that help people solve complex problems. Our goal is to make advanced AI a broadly accessible resource, and we’re grateful to our investors for their support. By collaborating with partners, including U.S. and allied governments, we hope to unlock this technology’s full potential,” OpenAI stated in its release.