Streaming with assistants

benoit.devilliers · 2024-09-11T09:38:54.656Z

A client of mine is building an app where you can chat with an AI (in this case using Open AI assistants). In their first version they did this by scheduling a workflow to run every 1 sec to check if they got an answer from open AI which is not ideal.

I initially suggested replacing this piece by a Buildship workflow which I know works properly, learned it from Gregory John. There is also this guide: Buildship guide: Create entries in your database using Ai by Sebastian Avelar → .

Note: my client wants to have his own LLM hosted on Azure in the future.

The other day I happened to be on a call with @georgecollier and he suggested that a plugin from @gaimed could achieve the same thing. The plugin isn’t available anymore but george mentioned it could be run on with Google Cloud Console Solutions, I’ve also seen ab mention it could be run using Cloudfare workers. I’m curious to hear more about this. If someone has the code of this plugin available somewhere I would love to take a look

[FREE PLUGIN] AI Proxy - ChatGPT, OpenAI, Claude, LlaMA, PaLM Real-Time Streaming

By running this through Claude he generated shared this script with me:

CleanShot 2024-09-11 at 05.25.53688×1028 111 KB

I would love to understand the pros and cons to use Buildship versus running serverless code (apart from cost).

I’d also be curious to understand why you would pick Google Cloud Console Solutions versus Cloudfare workfers.

Thank you for your help,
Have a good day!

Benoît

georgecollier · 2024-09-11T09:45:04.087Z

The streaming limitation to overcome is that Bubble’s API connector will not allow streamed responses. Aside from using webhooks (please don’t do it) to get a streamed response, there’s no other way aside from using a plugin.

So, now that’s out of the way:

Existing plugins are varied in their approach but accomplish the same thing, which is that you send off a prompt, and it streams back the response that you can access from a state on the plugin element.
Some plugins expose API keys - choose one that doesn’t
Some plugins route via the plugin owner’s server, which may have data privacy implications and it can in theory be shut down at any time

So, a serverless function alone won’t get you the streamed response, because you still need to get over the Bubble limitation of not being able to have streamed responses from the API Connector (which is why we use a plugin).

benoit.devilliers:

Google Cloud Console Solutions versus Cloudfare workfers.

Much the same… I just go with Google Cloud Functions as I find it pretty easy to set up and I keep all my clients on the same provider for ease of management.

benoit.devilliers:

Note: my client wants to have his own LLM hosted on Azure in the future.

Does he really though, or does he think he does without knowing what that actually means?

benoit.devilliers:

I would love to understand the pros and cons to use Buildship versus running serverless code (apart from cost).

No difference - whether you use Buildship or a custom code, you still need to use a plugin to overcome the streaming limitation. Bubble backend, BuildShip, custom code - none of them alone will allow you to stream text. Bubble backend definitely won’t allow you to. Buildship and custom code probably will.

benoit.devilliers · 2024-09-11T10:19:54.299Z

Thank you @georgecollier for your answer, if I got it correctly you are saying that a streaming set up that involves API connector isn’t a real streaming.

I recall I drew this on the call while you gave explanations

CleanShot 2024-09-11 at 06.07.351677×910 29.4 KB

When you say

No difference - whether you use Buildship or a custom code, you still need to use a plugin to overcome the streaming limitation. Bubble backend, BuildShip, custom code - none of them alone will allow you to stream text. Bubble backend definitely won’t allow you to. Buildship and custom code probably will.

Are you saying you have your own plugin to call Google Cloud solutions/Cloudfare workers (haven’t used that in Bubble yet)?

Then it would mean I either need:

plugin + buildship
plugin + custom code (either cloudfare or Google Cloud Solutions)

Is that correct?

I believe @jamesdevonport will be talking about running custom code on Cloudfare in his newsletter this week so it sounds like great timing

About this

Does he really though, or does he think he does without knowing what that actually means?

On this matter I may need to dig a bit more but I believe it is privacy related and they want to have their own thing. Started working on the UX first and now getting more into technical!

georgecollier · 2024-09-11T10:33:07.299Z

benoit.devilliers:

Are you saying you have your own plugin to call Google Cloud solutions/Cloudfare workers (haven’t used that in Bubble yet)?

Nah, I don’t use streaming. It’s overrated as LLMs are responding quicker and quicker. Not worth the hassle, and using backend allows me to make really modular AI flows.

benoit.devilliers:

plugin + buildship

plugin + custom code (either cloudfare or Google Cloud Solutions)

or just plugin, if the plugin is already connected to the plugin owner’s custom code

But if someone insisted they needed streaming in their app, I’d be using an existing plugin that doesn’t require me to connect any external providers and works out of the box. I haven’t been following the LLM plugin marketplace on Bubble closely as I don’t use them anymore so can’t recommend any off the top of my head, sorry

benoit.devilliers · 2024-09-11T10:45:22.107Z

Nah, I don’t use streaming. It’s overrated as LLMs are responding quicker and quicker. Not worth the hassle, and using backend allows me to make really modular AI flows.

Oh ok so you are just using API connector to call your Google Cloud Solutions?

But if someone insisted they needed streaming in their app, I’d be using an existing plugin that doesn’t require me to connect any external providers and works out of the box.

I have trouble understanding what that would look like. By no external provider do you mean something that would directly connect to OpenAI or Claude for example or are you referring to something that could be used with anything?

I haven’t been following the LLM plugin marketplace on Bubble closely as I don’t use them anymore so can’t recommend any off the top of my head, sorry

Got it, thank you!

georgecollier · 2024-09-11T10:47:12.886Z

benoit.devilliers:

Oh ok so you are just using API connector to call your Google Cloud Solutions?

I don’t use the Cloud Functions for streaming / AI model calling, but I use them for other parts of the app

benoit.devilliers:

y no external provider do you mean something that would directly connect to OpenAI or Claude for example or are you referring to something that could be used with anything?

No I mean if I did use a plugin, I wouldn’t want to have to set up and deploy my own custom code solution to work with it as well. I’d choose one that works out of the box without having to do any other technical stuff

benoit.devilliers · 2024-09-11T10:59:26.478Z

Got it thank you