Forum Academy Marketplace Showcase Pricing Features

[Updated Plugin] ChatGPT with Real-Time Streaming (No more timeouts!)

Showcase
#1

Hey Bubblers,

I’ve just released a new ChatGPT plugin that supports real-time streaming! You can now get responses streamed into your application immediately, as in the official ChatGPT interface, so no more waiting 20 seconds for a response, and no more timeouts with GPT-4!

Features:

  • Immediate, real-time streaming, so that responses come through without long waits
  • Easy message history, allows for multi-turn conversations. Memory that works out of the box.
  • Token counting - see how many tokens each request uses, in case you want to monitor usage.
  • Stream interruption - stop ChatGPT’s generation if it seems to be going in the wrong direction.

(I’m still actively developing the plugin, so see below for features in the works: [Updated Plugin] ChatGPT with Real-Time Streaming (No more timeouts!) - #49 by launchable)

The plugin page is here (ChatGPT with Real-Time Streaming Plugin | Bubble), or you could watch the setup video below. (this is slightly outdated now, and will be updated with the version 6 release, coming soon)

There are 2 demo pages to see in action:

  1. A ChatGPT style interface - chatgpt-demo
  2. A simpler interface with setup instructions - GPT Streaming

Let me know if you have any questions!

Korey @ Launchable AI

7 Likes
ChatGPT "Stream": True and Chunking
The Beginner's Guide to AI + Bubble
#2

shouldn’t you disclose that this works with a third party server?
Also Math.random is not secure, it’s safer to use web crypto api.

nice idea anyway.

2 Likes
#3

@dorilama - thanks for the feedback, that’s a great point re: disclosure of the third party server. I’ve added a note to both the description and usage page.

re: math.random being insecure, for transient keys, what do you see as the risk?

#4

Math.random is unsafe and predictable, web crypto use unpredictable hardware random generator, plus using a random % alphabet does not generate an even distribution. So a brute force attack will need less number of tries.
It doesn’t look like there is any auth for your ws endpoint (maybe I am mistaken), so I guess if someone wants to try there are more chanches than what you think to connect to someone else’s websocket and receive the messages.

#5

Thanks for the pointers. I’ll look into swapping in web.crypto instead of math.random. Do you have any other suggestions regarding security? I’ll continue to work on making it as robust and secure as possible.

Regarding the chance that someone can guess a websocket id, I believe its pretty slim. The generation space is about 10^53 combinations. ChatGPT’s take on the size of the space is the following:

To put this in perspective, the age of the universe is estimated to be around 13.8 billion years, or approximately 4.35 x 10^17 seconds. Therefore, even if every atom in the universe were to attempt to create a WebSocket connection simultaneously and continuously, it would still take many, many orders of magnitude longer than the age of the universe to reach 10^54 attempts.

#6

that applies only if you use a secure unpredictable random generator (which Math.random is not) and an even distribution in the alphabet used. That’s the kind of thing a brute force attack relies on.

#7

Okay, you’ve convinced me lol. I’ve adapted it to use the crypto library, and a much larger search space. Thanks for the quick feedback!

#8

Very cool. Been looking for a way to do this. What is the 3rd party server that it’s talking about?

#9

Thanks!

The server is one that we’ve built and are running, to handle the connection between the Bubble app and the OpenAI API. So it’s a “third party” server in the sense that its not directly Bubble → OpenAI, but we’re the third party. (I’ve updated the notes to make this more clear; thanks for pointing out that it wasn’t clear!)

1 Like
#10

Made several updates today, including max tokens, better message history handling, and a ChatGPT-style interface.

#12

Using assistant or system functionality seems to break the plugin. @launchable

image

#13

@Noqode - Can you check which version of the plugin you’re using? Another user also
found this bug this morning, and it should be fixed now, on 4.9.0.

p.s. - thanks for trying the plugin, and for posting your find here!

2 Likes
#14

Hey ChatGPT builders. Just published a new version (v 5.0.4), that simplifies setup and use.

Setup tutorial coming soon - “Build a ChatGPT clone in 10 minutes”. :robot: :rocket:

1 Like
#15

Awesome plugin, very good to use :blush:

1 Like
#16

Thanks @sonlovin! Glad you’re enjoying it!

#17

I don’t understand why you need to include your server in between. Can you please explain and can we avoid it or use our own server?

1 Like
#18
#19

Ahh… thanks!

#20

Then it will be great to know how to implement streaming service witnin our app so it doesn’t need another hop…

#21

Hey all. Just added a new action - “Stop Generating”, so you can cancel a request if GPT is going off the rails. Also a new state, “Data Container Currently Generating?”, so you can run workflows conditionally, based on whether text is currently streaming.

Let me know if you find this useful!

2 Likes
Extending timeout period for API calls to GPT-4