:speaking_head: ᴺᴱᵂ ᴾᴸᵁᴳᴵᴺ OpenAI GPT - Realtime Voice Chat On Your Data (incl. RAG, Tools & Conversation Audio Recording) [Keeps your key secure]

stuart8 · 2024-10-17T07:45:19.836Z

I’d be willing to pay for such functionality or a private plugin if it is something that could be done. Feel free to DM me if you can help with that.

Drayma · 2024-10-18T18:53:02.889Z

Amazing work on the plugin! Really easy and inutitive to use.

Just a few comments/questions:

Do I need to do anything to make different voices work? For some reason I can only get it to speak back to me if it’s on the default alloy voice. If I try to change it to anything else, I don’t hear any responses.
What does this field do? It gives me an error if it’s empty and it’s the only one without a documentation tooltip.

image962×154 8.71 KB

Drayma · 2024-10-18T19:17:25.728Z

Also I can’t seem to get it to work on mobile. When I press detect mic, it opens up the microphone permission pop up in the browser and if you click Allow it doesn’t go away. If you click on it a few times and then off the pop up, you’ll eventually give it permissions.

However, when you click on Start Voice, it doesn’t seem to be hearing me as I can’t hear any responses.

redvivi · 2024-10-18T19:18:32.915Z

Drayma:

What does this field do? It gives me an error if it’s empty and it’s the only one without a documentation tooltip.

This field must be set if you need to resume an existing conversation that has been interrupted by the Stop action.
You must pass one of the existing conversationId of ALL CONVERSATIONS state.
However there was a bug that I have fixed, this field should not be mandatory - please update.

Drayma:

Do I need to do anything to make different voices work?

Looking into this.

redvivi · 2024-10-18T19:30:50.183Z

Drayma:

Do I need to do anything to make different voices work? For some reason I can only get it to speak back to me if it’s on the default alloy voice. If I try to change it to anything else, I don’t hear any responses.

Supported voices for OpenAI Realtime API are alloy, echo or shimmer. I have no problem with those voices - please confirm @Drayma

Drayma · 2024-10-19T05:07:53.986Z

Thanks for the explanation. I’ve updated the app but it still shows as mandatory.

Drayma · 2024-10-19T05:09:21.247Z

Is there a reason all six are not supported? This is what they mention within the OpenAI docs and it shows it within the documentation you have as well.

I was really hoping that fable would be available.

redvivi · 2024-10-19T06:24:12.053Z

Drayma:

Is there a reason all six are not supported? This is what they mention within the OpenAI docs and it shows it within the documentation you have as well.

I was misled by OpenAI streaming documentation too which differs from Realtime API. Better to contact OpenAI directly.

See Advanced Mode in Realtime API - API - OpenAI Developer Forum

Drayma · 2024-10-19T09:36:52.552Z

Got it! Thank you for the response.

Drayma · 2024-10-19T09:37:24.989Z

ᴺᴱᵂ ᴾᴸᵁᴳᴵᴺ OpenAI GPT - Realtime Voice Chat (inc. Tools) [Keeps your key secure] Plugins

Also I can’t seem to get it to work on mobile. When I press detect mic, it opens up the microphone permission pop up in the browser and if you click Allow it doesn’t go away. If you click on it a few times and then off the pop up, you’ll eventually give it permissions. However, when you click on Start Voice, it doesn’t seem to be hearing me as I can’t hear any responses.

Have you tested the plugin on mobile across different browsers? I would be interested to know which ones were looked into, if at all, just so I know what to expect.

redvivi · 2024-10-19T09:39:45.327Z

Drayma:

Have you tested the plugin on mobile across different browsers?

Haven’t found one not working (yet!). Tested Chrome, Safari, Firefox on both desktop and mobile.
If you have an issue let me know via DM, I will investigate.

ron2 · 2024-11-05T19:53:28.028Z

Hi again! I set up the plugin on a test page and replicated the workflows and so on of the demo. Have confirmed the API key and credentials are working by sending a text message and getting an instantaneous voice response.

But, even when confirming that the plugin is listening, I cannot get any response. Here’s a screenshot of what I’m seeing after sending a text message (what’s up), showing the microphone, and the listening state. Demonstrating that everything but the voice input and response seems to be working. Any clue as to what I might look at to get it running?
Thanks!

Gets text message but no response to spoken messages550×556 55.2 KB

redvivi · 2024-11-05T20:18:29.875Z

First you send a text message, you get a voice response and then when you try with voice input you don’t get any voice response.

If you start fresh directly with the voice input, you get a voice response without issues.

It is what you notice?

ron2 · 2024-11-05T21:28:32.277Z

Thanks. No, in fact I do not get a voice response at all from voice input. This is the result of selecting the microphone, activating the plugin, talking to the app, hearing no response. Have not sent a text message.

IMG_12861138×1086 126 KB

redvivi · 2024-11-05T21:56:28.574Z

Please send me in DM the page of your app I may test.

ron2 · 2024-11-06T14:52:22.196Z

Checking to see if you got the DM.

redvivi · 2024-11-06T15:24:24.470Z

Yes I did. Will follow up privately.

redvivi · 2024-11-07T21:29:58.005Z

Hey Bubblers!

Informing you that I have released an interruption improvement. Now the model stops immediately if one speaks over it, like on OpenAI Playground.

Enjoy!

redvivi · 2024-12-06T16:00:49.982Z

Good news! OpenAI cuts its prices for Realtime API (as expected ).
Plus new voices.

x.com

OpenAI Developers

@OpenAIDevs

Two Realtime API updates: - You can now build speech-to-speech experiences with five new voices—which are much more expressive and steerable. 🤣🤫🤪 - We're lowering the price by using prompt caching. Cached text inputs are discounted 50% and cached audio inputs are discounted… x.com/i/web/status/1… pic.x.com/jLzZDBrR7l

4:51 PM - 30 Oct 2024 1.8K 230

redvivi · 2024-12-12T18:07:54.439Z

Hey,

Following OpenAI Realtime API’s changes for interrupts, a fix has been released today.
For those who had an error after interrupt, this fix adresses this issue.