🤖 BubbleGPT - An AI powered Bubble at what cost?

[keith]

I totally get where you’re at, @zelus_pudding. I’ve been goofing around with transformer/AI stuff for a while ever since OpenAI Jukebox piqued my interest and have trained several of my own weird models (mostly audio stuff but a lot of messing around with the GPT and Dall-E models as well).

Total aside here, not entirely apropos of what you’re saying:

Even up to GPT 3.5 I was pretty skeptical of “safety” concerns around transformer models like GPT. “All they are really good for is creative writing/making disinformation anyway” (these activities are basically identical) was basically my take and my feeling was that anyone who thought otherwise was just being bamboozled (which itself is a safety risk but just a minor one that boils down to “well, people be dumb and I deal with this all the time”).

And because of what is now generally called “hallucination” (these models’ proclivity to make up authoritative-sounding bullshit), the hallucination is all they are good for except for very specific domains like text summarization (which includes tasks like “here’s some code, describe what it does” which are tasks at which these models excel).

It’s a little early to tell, but GPT-4 seems rather different. The only way to interact with it at present is to upgrade to the paid version of ChatGPT (“ChatGPT Plus”), which I was able to do yesterday. There you can select the GPT-4 model and get a limited number of responses in a given timeframe.

I did some (non-Bubble related) text summarization tasks with it (here’s a list of ~200 business requirements, categorize them and tell me what are the most common concepts/keywords and also point out some of the more unique ones) and GPT-4 performs similarly to GPT-3.5 (and in fact I liked some of 3.5’s summarizations a bit better than 4’s, which seems to be very “buttoned up” and to-the-point in its response style).

The thing that really stood out for me in the GPT-4 announcement stream was the part where they were pair programming with GPT-4 on a Discord chatbot. I found it very interesting that, simply by sending it more current documentation on the API, the model was able to generate corrected code. (3.5 will of course write code and “knows” JavaScript quite well, but the quality of its output is questionable and often its suggestions are outright wrong, especially for naive prompts.)

And of course, GPT (all versions) doesn’t really know much about very specific APIs such as the Bubble plugin APIs, the documentation for which is (1) hidden behind a login (any actual helpful information is inside the plugin builder interface itself) and (2) not particularly robust, correct, or helpful anyway.

But, can you talk it through the specifics of a quirky API like the Bubble plugin API and teach it to write correct code? It seems that you can, as evidenced by the transcript of this chat I had with GPT-4 last night:

Not-exactly-live Bubbling with @keith: Transcript - @keith teaches GPT-4 to write Bubble element plugins

NOTE: AIs like (even) GPT-4 are unreliable and you can’t just throw random shit at them and get reliable answers. However, if you know what you’re doing, you can (in the case of GPT-4, here represented by “Chat GPT Plus” [the only way to get access to GPT-4 just now]) teach it new tricks. In the following transcript, I talk GPT-4 through the some of the issues involved in building an element plugin for Bubble. You’ll note that its answers are actually accurate… because I’m teaching it to do th…

A couple of things to note here:

1. Does it really know about List Shifter, or is its answer about that just a good guess or a “hallucination”? I did, in chats with previous versions of GPT, talk about List Shifter and even showed it the entire minified code. Did that then get included in training data for version 4? You’ll also note that its reply about this is pretty complimentary to me personally… is it just trying to butter me up? To what end, I wonder?

2. After I briefly mention the 3 types of Bubble plugin APIs, GPT-4 claims to understand their differences and gives some info that is mostly right. But is that coincidence or a hallucination? Again, it might just be a good guess.

3. Once I actually start teaching it, without actually providing code examples, but just describing some peculiarities of the API in natural language, GPT-4 becomes really amazing. For example, it understands my explanation of the non-standard JavaScript opening line for the initialize function and remembers it later.

4. You’ll note that it comments its code. I hadn’t seen previous versions do that in the same way. Also, I had this chat on mobile and my frequent typos don’t really confuse it at all.

5. This section of code is particularly endearing (I asked it to do this but I feel like it went above-and-beyond… again, what does it want from me?):

   // Create a string expressing how awesome Keith is
   const awesomeMessage = ‘Keith is incredibly awesome!’;

6. Its code is actually correct.

I’m going to continue that conversation and see how long it takes until it loses attention and also to see if it can adopt my coding style. I trust that it will, but I wonder how long it takes before it loses important info from earlier in the thread…