A client of mine is building an app where you can chat with an AI (in this case using Open AI assistants). In their first version they did this by scheduling a workflow to run every 1 sec to check if they got an answer from open AI which is not ideal.
Note: my client wants to have his own LLM hosted on Azure in the future.
The other day I happened to be on a call with @georgecollier and he suggested that a plugin from @gaimed could achieve the same thing. The plugin isn’t available anymore but george mentioned it could be run on with Google Cloud Console Solutions, I’ve also seen ab mention it could be run using Cloudfare workers. I’m curious to hear more about this. If someone has the code of this plugin available somewhere I would love to take a look
The streaming limitation to overcome is that Bubble’s API connector will not allow streamed responses. Aside from using webhooks (please don’t do it) to get a streamed response, there’s no other way aside from using a plugin.
So, now that’s out of the way:
Existing plugins are varied in their approach but accomplish the same thing, which is that you send off a prompt, and it streams back the response that you can access from a state on the plugin element.
Some plugins expose API keys - choose one that doesn’t
Some plugins route via the plugin owner’s server, which may have data privacy implications and it can in theory be shut down at any time
So, a serverless function alone won’t get you the streamed response, because you still need to get over the Bubble limitation of not being able to have streamed responses from the API Connector (which is why we use a plugin).
Much the same… I just go with Google Cloud Functions as I find it pretty easy to set up and I keep all my clients on the same provider for ease of management.
Does he really though, or does he think he does without knowing what that actually means?
No difference - whether you use Buildship or a custom code, you still need to use a plugin to overcome the streaming limitation. Bubble backend, BuildShip, custom code - none of them alone will allow you to stream text. Bubble backend definitely won’t allow you to. Buildship and custom code probably will.
Thank you @georgecollier for your answer, if I got it correctly you are saying that a streaming set up that involves API connector isn’t a real streaming.
I recall I drew this on the call while you gave explanations
No difference - whether you use Buildship or a custom code, you still need to use a plugin to overcome the streaming limitation. Bubble backend, BuildShip, custom code - none of them alone will allow you to stream text. Bubble backend definitely won’t allow you to. Buildship and custom code probably will.
Are you saying you have your own plugin to call Google Cloud solutions/Cloudfare workers (haven’t used that in Bubble yet)?
Then it would mean I either need:
plugin + buildship
plugin + custom code (either cloudfare or Google Cloud Solutions)
Is that correct?
I believe @jamesdevonport will be talking about running custom code on Cloudfare in his newsletter this week so it sounds like great timing
About this
Does he really though, or does he think he does without knowing what that actually means?
On this matter I may need to dig a bit more but I believe it is privacy related and they want to have their own thing. Started working on the UX first and now getting more into technical!
Nah, I don’t use streaming. It’s overrated as LLMs are responding quicker and quicker. Not worth the hassle, and using backend allows me to make really modular AI flows.
or just plugin, if the plugin is already connected to the plugin owner’s custom code
But if someone insisted they needed streaming in their app, I’d be using an existing plugin that doesn’t require me to connect any external providers and works out of the box. I haven’t been following the LLM plugin marketplace on Bubble closely as I don’t use them anymore so can’t recommend any off the top of my head, sorry
Nah, I don’t use streaming. It’s overrated as LLMs are responding quicker and quicker. Not worth the hassle, and using backend allows me to make really modular AI flows.
Oh ok so you are just using API connector to call your Google Cloud Solutions?
But if someone insisted they needed streaming in their app, I’d be using an existing plugin that doesn’t require me to connect any external providers and works out of the box.
I have trouble understanding what that would look like. By no external provider do you mean something that would directly connect to OpenAI or Claude for example or are you referring to something that could be used with anything?
I haven’t been following the LLM plugin marketplace on Bubble closely as I don’t use them anymore so can’t recommend any off the top of my head, sorry
I don’t use the Cloud Functions for streaming / AI model calling, but I use them for other parts of the app
No I mean if I did use a plugin, I wouldn’t want to have to set up and deploy my own custom code solution to work with it as well. I’d choose one that works out of the box without having to do any other technical stuff
