🗣 ᴺᴱᵂ ᴾᴸᵁᴳᴵᴺ OpenAI GPT - Realtime Voice Chat (inc. Tools & Conversation Audio Recording) [Keeps your key secure]

Hey Bubblers!

Now, OpenAI Realtime pushes the boundaries by integrating language generation with seamless voice interaction, giving app builders the tools they need to craft more natural and conversational AI experiences.

From creating virtual assistants to powering real-time customer support, this new model opens a vast array of possibilities for voice-driven applications.

This plugin uses an external service to provide streaming capabilities.

You can test out our OpenAI GPT - Realtime Voice Chat Plugin with the live demo.

20230607PHT95601_original

Enjoy !
Made with :black_heart: by wise:able
Discover our other Artificial Intelligence-based Plugins.

8 Likes

:fire: great timing

1 Like

Any chance for a VAPI plugin on the roadmap?

That’s to be considered if no current plugin satisfies the user’s need.

Hey Bubblers!

Informing you that this plugin has been updated to support Conversations (Threads).

Enjoy!

1 Like

Is there a way to push in a user key or a BYOK functionality to this? We’d like to use with our users but wouldn’t want to run from our own key. If we are using a BYOK function then having that exposed prior to firing up the chat isn’t the biggest of issues (as the user will have just entered it)

Nope.

I’d be willing to pay for such functionality or a private plugin if it is something that could be done. Feel free to DM me if you can help with that.

Amazing work on the plugin! Really easy and inutitive to use.

Just a few comments/questions:

  • Do I need to do anything to make different voices work? For some reason I can only get it to speak back to me if it’s on the default alloy voice. If I try to change it to anything else, I don’t hear any responses.

  • What does this field do? It gives me an error if it’s empty and it’s the only one without a documentation tooltip.

Also I can’t seem to get it to work on mobile. When I press detect mic, it opens up the microphone permission pop up in the browser and if you click Allow it doesn’t go away. If you click on it a few times and then off the pop up, you’ll eventually give it permissions.

However, when you click on Start Voice, it doesn’t seem to be hearing me as I can’t hear any responses.

This field must be set if you need to resume an existing conversation that has been interrupted by the Stop action.
You must pass one of the existing conversationId of ALL CONVERSATIONS state.
However there was a bug that I have fixed, this field should not be mandatory - please update.

Looking into this.

Supported voices for OpenAI Realtime API are alloy, echo or shimmer. I have no problem with those voices - please confirm @Drayma

Thanks for the explanation. I’ve updated the app but it still shows as mandatory.

Is there a reason all six are not supported? This is what they mention within the OpenAI docs and it shows it within the documentation you have as well.

I was really hoping that fable would be available.

image

I was misled by OpenAI streaming documentation too which differs from Realtime API. Better to contact OpenAI directly.

See Advanced Mode in Realtime API - API - OpenAI Developer Forum

Got it! Thank you for the response.

1 Like

Have you tested the plugin on mobile across different browsers? I would be interested to know which ones were looked into, if at all, just so I know what to expect.

Haven’t found one not working (yet!). Tested Chrome, Safari, Firefox on both desktop and mobile.
If you have an issue let me know via DM, I will investigate.

Hi again! I set up the plugin on a test page and replicated the workflows and so on of the demo. Have confirmed the API key and credentials are working by sending a text message and getting an instantaneous voice response.

But, even when confirming that the plugin is listening, I cannot get any response. Here’s a screenshot of what I’m seeing after sending a text message (what’s up), showing the microphone, and the listening state. Demonstrating that everything but the voice input and response seems to be working. Any clue as to what I might look at to get it running?
Thanks!

First you send a text message, you get a voice response and then when you try with voice input you don’t get any voice response.

If you start fresh directly with the voice input, you get a voice response without issues.

It is what you notice?