From voice-controlled browser to anime assistant: 6 incredible OpenAI Realtime API examples

Here are some wild examples shared on X by developers. 

Speech to Picasso

This incredible use case brings forth a voice-controlled painting app. Jordan Singer, who as per his X bio is the founder of Mainframe, a generative computing company, shared his new creation with OpenAI’s realtime voice API on X. Singer calls it Teledraw, an experimental drawing app that is a fusion of real-time voice and image models. It explores innovative interfaces by using the latest latent consistency models which allows users to create art through voice commands. Singer showed the unique UI, which mimics a phone call, pushing the boundaries of interactive technology.  

🎨 new technologies necessitate new interfaces

with real-time latent consistency models, here’s a different kind of drawing app: pic.twitter.com/XwNKzt2vF0

— Jordan Singer (@jsngr) December 3, 2023

PDF mind reader

Another X user, Marcus Schiesser, who calls himself a tech enthusiast, has created a voice chat for documents. Known as Voice Chat PDF, the tool is built using OpenAI Realtime API, Llama Index, and Next.js. The app allows users to chat with their own documents. The demo shared by Schiesser shows the feature using a document on physical mailing standards, highlighting how a user can interact with content using voice in real-time.

Want to chat over your own documents using the new @OpenAI Realtime API?

You can do so now using Voice Chat PDF, built using @llama_index and @nextjs.

The video below shows an example using a document about physical mailing standards.

📄 https://t.co/Oq6GCdvIrM pic.twitter.com/GmAaLbSo7L

— Marcus Schiesser (@MarcusSchiesser) October 4, 2024

Assistant for mock interviews

Kenn Ejima, former head of Japan Quora, shared an AI interviewer who conducts mock interviews, essentially quizzing people on their resume. The new mock interview app lets users practice interview skills by uploading their CVs or resumes for AI-driven questions. It currently supports Stanford MBA applications and allows one free trial every 24 hours. It is built with Remix, Render, Quadrant, and Cloudflare R2. 

🚀 Just launched! 🚀

Practice your interview skills with our 2-minute mock interview app using @OpenAI’s new Realtime API.

🎤 Upload your CV, and let the AI interviewer ask about your experience.

👉 https://t.co/aNRWBcIc2e

Try it for FREE! pic.twitter.com/5fcPG5UfhJ

— Kenn Ejima (@kenn) October 11, 2024

Voice-controlled browser

Software engineer Sawyer Hood shared a voice-controlled browser on X. With this browser, one simply needs to open and say out loud what they are searching for. The browser is built using OpenAI’s Realtime API and lets users navigate the internet through voice commands. The system deploys a custom DOM format for reliable page understanding, avoiding the intricacies of raw HTML. The browser is currently in development and according to Hood, the browser aims to offer seamless voice-based web interactions. 

The open ai realtime api is sick! I hooked it up to control my browser so I could browse the web with my voice 🤯 pic.twitter.com/sCsNOz1OXr

— Sawyer Hood (@sawyerhood) October 4, 2024

Your trading assistant

Wily Douhard, a developer, has made a voice assistant that can track the price of multiple stocks using your voice. Douhard has created something known as Chainlit Realtime which supports WebSockets for real-time audio interactions by integrating OpenAI’s Realtime Voice API. This app shows how developers can build responsive assistants that stream audio commands and responses seamlessly. 

🎙️Chainlit Realtime is here! 🎙️

Featuring first-class WebSocket support for realtime audio interactions in Chainlit applications.

We’ve added support for @OpenAI real-time API to unlock a whole new UX for devs building intelligent, responsive assistants. pic.twitter.com/RxEUtqOGyI

— willy douhard (@willy_douhard) October 4, 2024

Your realtime-anime friend

Bryan Pratte, founder of Hallway.AI, showed how OpenAI’s Realtime API when combined with ExpressionEngine, can bring anime characters to life. Based on the demo, this integration seems to enable real-time voice interactions with animated characters. It offers an immersive experience as seen in the demo below.

OpenAI Realtime API + ExpressionEngine opens up a whole new world. Chat with @join_hallway characters coming in hot! pic.twitter.com/oYckyuEilu

— bryan pratte (@btp4z7) October 1, 2024

On October 1, OpenAI introduced the Realtime API that allows developers to build applications with live interactions. This API supports speech-to-text, text-to-speech, and real-time conversation abilities which makes it possible to create dynamic assistants and voice experiences. With audio and text being streamed back and forth, the Realtime API allows for highly responsive applications. 

According to OpenAI, this API has been designed for use cases like virtual assistants, live collaboration tools, and interactive educational apps. The Realtime API uses OpenAI’s powerful language models which offer seamless real-time conversations that enhance user engagement and interaction across a wide range of use cases.