[Updated Plugin] ChatGPT with Real-Time Streaming (No more timeouts!)

Hi all,

Just published some important stability updates with 5.7.1 :shield:

  • :computer: Connection handling is now more robust, which helps if your app is getting a lot of traffic
  • :white_check_mark: New Workflow Action - “Ensure Connected”. You can run this before running “Send Message”, to ensure the connection is open. Not strictly necessary, but adds stability for long user sessions.

If you find any bugs or issues, please let me know!

1 Like

Hi all - two more notes about recent updates:

  1. First, apologies to all for the service interruptions. The traffic continues to grow faster than I’ve been able to keep up. Which is of course fantastic, but has brought some growing pains :grimacing:

  2. Second, please update to 5.7.1 if you’re using this plugin in production, and please use the “Ensure Connected” event before the “Send Message” event. This will result in a much more stable user experience.

I’ve just upgraded the servers again to handle the traffic spikes, so I’m hoping we’ll see things become a lot more stable over the next day or so.

Thanks!

3 Likes

Thank you!

Hi all,

For those following the stability/scaling issue, there have been some significant improvements made over the last 2 days, and more coming over the weekend. The plugin is becoming more performant and reliable every day :smiley:

Thanks to everyone for your patience this week, and your help in finding and fixing the bottlenecks! :rocket:

5 Likes

Hey all,
For anyone who may have experienced the service interruption earlier today, the issue has been fixed. Sorry! I’m hoping we’ve just about rooted out all the possible ways the system can break. If you’re curious, we’ve hit all of the following failure modes in the past 2 weeks:

  1. Redis/cache overload
  2. Network I/O limits
  3. CPU throttling
  4. RAM limitations
  5. File descriptor limits

All have now been addressed, and we’ll keep patching/improving as we find new scaling pains.

1 Like

A Short-Term Roadmap

Here are the improvements coming over the next few days:

  1. :female_detective: Run without 3rd-party server - connect directly to OpenAI, so you don’t have to worry about data privacy issues. This has been built already, and I will release it tomorrow (May 8) after more thorough testing.
  2. :bar_chart: Health Check - easily check if service is connected, so that you can use a fallback option, like …
  3. :hammer_and_wrench: Non-streaming response Fallback - In case streaming is unavailable, fallback to a standard wait-for-response API call. The goal for the plugin is to never have this event trigger, but also to never have a request not complete because of some connectivity issue.
  4. :desktop_computer: Custom OpenAI endpoints - Support for OpenAI on Azure and other APIs exposing the ChatGPT models. This will be experimental.

Other features in the works:

  1. :eye: Code Syntax Highlighting - Auto-detect and format code snippets, as in the official interface
  2. :robot: System Message Improvements - Checkbox to always send most recent system message, so you don’t always need to set it.
  3. :bookmark: Better Message History Handling - Send more complete message history context, with fewer tokens, using conversation summarization from LangChain.

Let me know what else you’d like to see in this list!

4 Likes

Hey,

Thanks for the plugin - it’s great!

However, I am sporadically facing bugs when I send it the message history and add a prompt on top of that. It seems to give a garbage response in return.

I have also recorded a loom video to showcase the fact that I am sending the message history correctly and it is still returning a garbage response.

Loom link: Loom | Free Screen & Video Recording Software | Loom

I had shared this with my team and am facing some heat because of these errors. I tried pretty hard to debug this - can you please help out here?

Hi Hrishikesh,

Thanks a lot for letting me know about this. I’ve just sent you a follow-up email. I’ll take a look at this this morning and get back to you shortly once it’s resolved. Stay tuned!

1 Like

Thanks Korey!

1 Like

If anyone else is currently seeing this issue, check to see whether you are running “Set Message History” in the same workflow as you are calling the “Send Message” action. Currently, this sometimes cause the message history not to be sent, because the generation starts asynchronously, before the “Set Message History” action has had time to run. This appears to be the case especially in “Debug Mode”, as Bubble will pause the workflow steps (ie the Set Message History action), but not the plugin action.

I’m looking for a solution to this, but in the meantime, you can do the following:

Create a custom workflow that calls “Clear Message History”, and call that custom workflow from your larger workflow. This will force it to be called when you want it to.

You can also try setting it in a different workflow than in your “Send Message”, for example, when a user reloads a previous conversation, or on message generation completion, etc…

Thanks again for the pointer & help debugging this Hrishikesh!

3 Likes

Hi all,

There is currently a bug in the plugin causing all uses of “Send Message”, after the first, to fail with the error below.

I’m looking into this issue now, but don’t think it’s directly an issue with the plugin, as nothing had changed before the break, so I have opened a support ticket with Bubble.

In the meantime, you can upgrade to 5.9.0, which has a new action “Send Message to OpenAI Directly”, which talks directly to OpenAI, instead of the backend server, and has most of the same functionality.

This was a little rushed to release, given this current break, so please let me know if you encounter any errors.

Thanks for your patience everyone!

Sample error message:

Plugin action ChatGPT - Send Message error: 
Error: Cases weren't exhaustive: {"role":"user","content":"Tell me about Bubble.io"} }
    at exhaustive (/var/task/util/util_lambda.js:142:9)
    at get_wrapped_pw (/var/task/util/util_lambda.js:154:12)
    at /var/task/util/util_lambda.js:181:47
    at Array.map ()
    at Object.get_wrapped_list (/var/task/util/util_lambda.js:181:35)
    at LambdaListWrapper.get (/var/task/plugin_api_v3.js:54:40)
    at eval (eval at build_function (/var/task/util/util_harness.js:38:12), :19:61)
    at eval (eval at build_function (/var/task/util/util_harness.js:38:12), :135:8)
    at /var/task/plugin_api_v3.js:250:27
    at run_fn (/var/task/u.js:550:18)

Error: Outer Error (see above for inner error)
    at Block.wait (/var/task/u.js:399:33)
    at await_promise_on_fiber (/var/task/plugin_api_v3.js:206:25)
    at LambdaListWrapper.get (/var/task/plugin_api_v3.js:54:12)
    at eval (eval at build_function (/var/task/util/util_harness.js:38:12), :19:61)
    at eval (eval at build_function (/var/task/util/util_harness.js:38:12), :135:8)
    at /var/task/plugin_api_v3.js:250:27
    at run_fn (/var/task/u.js:550:18)

Hi again all,

Confirmation from Bubble that the same error is being experience with other plugins. Fix incoming.

This has seemingly been resolved now, and both modes of calling the API (via the websocket server, or client-side, direct to OpenAI) are working as intended.

Thanks for your pointers and patience everyone!

1 Like

Thanks Korey!

If we hit the OpenAI API directly, I presume streaming won’t be possible in that case?

@hrishikesh - Yep streaming is still possible.

@launchable I noticed you added the API key to a workflow.

I believe this will be visible to any user who knows how to find it.

Shouldn’t that be in the backend?

Thanks @launchable fantastic plugin! Looking at the demo editor I only see space for 3 messages. Does this mean conversations are limited to only 3 messages in total?

Hey @Buddha - there are two ways you can call the API, one server-side and one client side. The server side one protects the key, and the client side one doesn’t, so the latter (Send Message Direct to OpenAI) should only be used if users are providing their own keys.

I’ll make this more clear in the docs and tutorial coming later today. The update with this new action was just released yesterday, so I haven’t finished documenting everything. Good catch though!

1 Like

Thanks @mattblake!

Conversations aren’t limited to only 3 messages. They can be infinitely long. But you can only send three new messages at a time, ie., per conversation turn. The plugin will also send the last 8 messages by default. You can limit this to fewer messages if you wish, and I’ll be adding support for it to be more (with automatic length control so you don’t go over the token input limit).

Hi all,

Setup tutorial video has been updated (https://youtu.be/vcHjAxI2n8o).

Let me know if you run into any issues!