:speaking_head: ᴺᴱᵂ ᴾᴸᵁᴳᴵᴺ OpenAI GPT - Realtime Voice Chat On Your Data (incl. RAG, Tools & Conversation Audio Recording) [Keeps your key secure]

redvivi · 2024-10-18T19:30:50.183Z

Drayma:

Do I need to do anything to make different voices work? For some reason I can only get it to speak back to me if it’s on the default alloy voice. If I try to change it to anything else, I don’t hear any responses.

Supported voices for OpenAI Realtime API are alloy, echo or shimmer. I have no problem with those voices - please confirm @Drayma

Drayma · 2024-10-19T05:07:53.986Z

Thanks for the explanation. I’ve updated the app but it still shows as mandatory.

Drayma · 2024-10-19T05:09:21.247Z

Is there a reason all six are not supported? This is what they mention within the OpenAI docs and it shows it within the documentation you have as well.

I was really hoping that fable would be available.

redvivi · 2024-10-19T06:24:12.053Z

Drayma:

Is there a reason all six are not supported? This is what they mention within the OpenAI docs and it shows it within the documentation you have as well.

I was misled by OpenAI streaming documentation too which differs from Realtime API. Better to contact OpenAI directly.

See Advanced Mode in Realtime API - API - OpenAI Developer Forum

Drayma · 2024-10-19T09:36:52.552Z

Got it! Thank you for the response.

Drayma · 2024-10-19T09:37:24.989Z

ᴺᴱᵂ ᴾᴸᵁᴳᴵᴺ OpenAI GPT - Realtime Voice Chat (inc. Tools) [Keeps your key secure] Plugins

Also I can’t seem to get it to work on mobile. When I press detect mic, it opens up the microphone permission pop up in the browser and if you click Allow it doesn’t go away. If you click on it a few times and then off the pop up, you’ll eventually give it permissions. However, when you click on Start Voice, it doesn’t seem to be hearing me as I can’t hear any responses.

Have you tested the plugin on mobile across different browsers? I would be interested to know which ones were looked into, if at all, just so I know what to expect.

redvivi · 2024-10-19T09:39:45.327Z

Drayma:

Have you tested the plugin on mobile across different browsers?

Haven’t found one not working (yet!). Tested Chrome, Safari, Firefox on both desktop and mobile.
If you have an issue let me know via DM, I will investigate.

ron2 · 2024-11-05T19:53:28.028Z

Hi again! I set up the plugin on a test page and replicated the workflows and so on of the demo. Have confirmed the API key and credentials are working by sending a text message and getting an instantaneous voice response.

But, even when confirming that the plugin is listening, I cannot get any response. Here’s a screenshot of what I’m seeing after sending a text message (what’s up), showing the microphone, and the listening state. Demonstrating that everything but the voice input and response seems to be working. Any clue as to what I might look at to get it running?
Thanks!

Gets text message but no response to spoken messages550×556 55.2 KB

redvivi · 2024-11-05T20:18:29.875Z

First you send a text message, you get a voice response and then when you try with voice input you don’t get any voice response.

If you start fresh directly with the voice input, you get a voice response without issues.

It is what you notice?

ron2 · 2024-11-05T21:28:32.277Z

Thanks. No, in fact I do not get a voice response at all from voice input. This is the result of selecting the microphone, activating the plugin, talking to the app, hearing no response. Have not sent a text message.

IMG_12861138×1086 126 KB

redvivi · 2024-11-05T21:56:28.574Z

Please send me in DM the page of your app I may test.

ron2 · 2024-11-06T14:52:22.196Z

Checking to see if you got the DM.

redvivi · 2024-11-06T15:24:24.470Z

Yes I did. Will follow up privately.

redvivi · 2024-11-07T21:29:58.005Z

Hey Bubblers!

Informing you that I have released an interruption improvement. Now the model stops immediately if one speaks over it, like on OpenAI Playground.

Enjoy!

redvivi · 2024-12-06T16:00:49.982Z

Good news! OpenAI cuts its prices for Realtime API (as expected ).
Plus new voices.

x.com

OpenAI Developers

@OpenAIDevs

Two Realtime API updates: - You can now build speech-to-speech experiences with five new voices—which are much more expressive and steerable. 🤣🤫🤪 - We're lowering the price by using prompt caching. Cached text inputs are discounted 50% and cached audio inputs are discounted… x.com/i/web/status/1… pic.x.com/jLzZDBrR7l

4:51 PM - 30 Oct 2024 1.8K 230

redvivi · 2024-12-12T18:07:54.439Z

Hey,

Following OpenAI Realtime API’s changes for interrupts, a fix has been released today.
For those who had an error after interrupt, this fix adresses this issue.

redvivi · 2024-12-21T14:15:38.515Z

Hey guys,

I’m pleased to share that the plugin has undergone performance improvements, resulting in notably faster response times for more reactive conversations and improved accuracy in remembering past history.

Enjoy!

redvivi · 2024-12-21T14:19:02.678Z

Also, again, another price cut is coming up

Starting January 9, the Realtime API will point to our latest model snapshot gpt-4o-realtime-preview-2024-12-17 by default. This snapshot reduces the audio token price by 60% to $40/1M input tokens and $80/1M output tokens. Cached audio input costs are reduced by 87.5% to $2.50/1M input tokens. We’ve also improved voice quality and made inputs more reliable, especially for dictated numbers. gpt-4o-audio-preview will also be updated to point to the gpt-4o-audio-preview-2024-12-17.

redvivi · 2025-01-30T14:30:20.921Z

Hey Bubblers!

Informing you that this plugin has been updated with Conversation Audio Recording capabilities.

Enjoy!

redvivi · 2025-02-12T13:45:11.866Z

Hey guys,

The latest update supercharge your app by letting OpenAI Realtime API automatically pull relevant data from your app database with respect to privacy rules to enhance responses (RAG, a.k.a Retrieval-Augmented Generation). No more manual searching—just seamless AI-powered insights!

Screenshot 2025-02-12 at 14.43.26699×466 156 KB

Step 1: Set App Data Retrieval on the element

Screenshot 2025-02-12 at 14.30.34248×474 25.3 KB

Step 2: Enable your App Data API for the desired fields
!

That’s it! OpenAI Realtime API will now pull real-time data from your app whenever needed—making responses smarter, faster, and more accurate.

Remember: Set up your privacy rules to keep everything secure!

Let me know if you have any issues.

Enjoy!