New TTS Audio/Speech AI Plugin using Vapi

michael.david.marsac · 2024-09-05T04:59:18.695Z

Howdy ya’ll super excited to introduce our very first public plugin!

If you haven’t used any smart Audio AI tools before, you’re missing out on a whole world of functionality.

This plugin let’s you get a smart assistant like Siri directly in your app that can speak, communicate, and transcribe conversations in real time. These transcriptions can then be used to make decisions based on the conversation or start follow on actions. A ton of both localized and digital first use cases to explore here.

We are actually using it for our own app SARA AI, which answers the phone for local restaurants, and would love to share in the fun with ya’ll.

Below is a link to:

how to set up vapi on the vapi side of things (get keys and assistant ID’s)
how the plugin works and
the plugin itself.

If you have any questions hit us up here or join our discord!

Setting up Vapi

Using the plugin

The plugin itself

Bubble

Templates for Bubble

Buy and sell Bubble templates to speed up app development. Bubble is a visual programing language that lets you build apps without code.

mattblake · 2024-09-12T09:57:36.839Z

Very cool, GLWS!

I’ve been watching Vapi since the start of the year. Exciting to see how the plugin makes audio AI so easy to implement into Bubble.

Would a Hume.ai plugin be just as easy for you and the team to build? Per / minute cost is slightly lower than Vapi and Hume have a very interesting ethos behind their LLM development.

michael.david.marsac · 2024-09-12T16:07:28.474Z

Hey Matt, thanks for positive feedback, we appreciate you

Most certainly can, would you be willing to be our first customer? If you pre-paid I’d drop it to $100 for you instead of $125 and we could have it done by sometime next week. Would also let you have some input on how it works if there’s any more advanced features you’d need.

Let me know and feel free to DM me!

bexnoel · 2024-10-03T15:21:45.290Z

I would also be extremely interested in a Hume.ai plugin. Is this on your roadmap? I would purchase it ASAP! Thank you so much!

ron2 · 2024-12-01T20:19:43.646Z

UPDATE:
Figured this out, see next post of mine. The plugin encountered a comma in my variable text and apparently attempted to interpret it as a delimiter before a second variable, and thus failed. Best to find/replace commas with a space or something like that. Also: Dynamic Variables are really cool and powerful. Glad to be able to get back to spinning them up.

Hi there, @michael.david.marsac. I’ve been experimenting with a few VAPI plugins, and I really like yours for a number of reasons. One is the dynamic variables you can enter when you make a call. I am having some difficulty though. I can make the variables work if I directly enter plain text in the element inspector, like:

specials:Today we have hamburgers on special.

This works.

But if I have a multiline input and enter the specials in that, in the same way you have in the demo video, I get an error.

Here are a couple of screenshots. One is from your demo video, to make clear what I’m referring to, the second is a screenshot of the error, the third is how the plugin is set up to get the value of a multiline input.

The variable {{theCaper}} is set up in the Assistant and as mentioned, works if the Vapi plugin variable is just the plain text, and not a dynamic reference to the data.

Appreciate your assistance. This is a really valuable piece of functionality and I’m hoping to make it work. (Would also be cool to have dynamically referenced files, but first things first.)

Thanks!

Ron

image862×1192 75.8 KB

image888×650 58.2 KB

image1462×621 74.7 KB

ron2 · 2024-12-02T10:51:04.079Z

I figured it out! There was a comma in my variable content. You can see it in the last image. Removed that and all worked as expected. I will henceforward do a find/replace on anything that goes up there. Might want to add this to your docs as a tip or something, or sanity check for commas on your end.

Why is it always one little character that eats your day?

Thanks anyway! Will leave the original post up in case anyone else runs into this.

Ron

michael.david.marsac · 2024-12-02T16:05:38.056Z

Hey Ron, glad you found the issue and it’s working great for you

Let me know if you ever need anything, feel free to ping me or join our discord!

ron2 · 2024-12-03T14:03:03.233Z

Thanks, Michael. Much appreciated.

In fact, I have run into some anomalous behavior, and I wonder if you might check it there to confirm. When I use an 11Labs voice, the conversation text returned from the plugin includes only the “user” prompts. The Assistant responses do not appear, so only getting half the conversation.

When I switch to a different provider, say OpenAI, Azure, or Cartesia, both sides of the conversation are returned.

I have tested this by having a text box that simply displays the VAPI element’s transcription of the current call. And to test with different voices, I am selecting different providers in the VAPI dashboard, and publishing the changed voice settings.

Can you possibly try this and confirm whether it is happening or not in your tests? Kind of stumped as to why a voice provider would make a difference in the conversation communicated back to the client, but, of course. software.

Best,
Ron

ron2 · 2024-12-04T15:33:02.481Z

@michael.david.marsac Bumping this.

michael.david.marsac · 2024-12-04T16:48:19.721Z

Hey Ron, sorry for the delay, here’s a quick loom: Understanding Voice Transcription Tools | Loom

ron2 · 2024-12-09T16:15:37.393Z

Michael, thanks! And sorry for the delay on my end. Some traveling within EU and have dropped a couple of threads.

I still seem to have this problem, though I believe I am set up properly. I’ll test again with a vanilla implementation set up exactly as your demo is, and hopefully that will allow me to home in on the culprit.

Thanks again, Ron

michael.david.marsac · 2025-01-06T21:46:52.768Z

@ron2 did it ever work for you?

ron2 · 2025-01-31T22:08:26.084Z

Hi, sorry for the delay! Some traveling.
I ended up taking another tack, implementing Get Call from which I can obtain a fairly well-formatted transcript as well as other useful information, like cal cost, etc.

By the way, I noticed the JSON assistant field in the recent update. Looks like you’ve got support for transient assistants, then, with bespoke per-call data. Good move!

Ron

michael.david.marsac · 2025-02-01T17:59:36.706Z

Yeah Dynamic data is a huge power play with this stuff. Hopefully we can find some powerful use cases for it! Thanks for the continued support.