Forum Academy Marketplace Showcase Pricing Features

Why are API Data Types not Exposed in the Database?

Hi K, what you were seeing is dates that came from my API. They were not produced at that moment by the API call. They were dates STORED IN MY DATABASE. The dates came from a PREVIOUSLY EXECUTED API call. I was fetching the values DIRECTLY from the Calendar object being visualized on that page. I was NOT showing return values that came back JUST THEN by an API call. (See what I mean?)

Those dates were dates from the “Start” field of that Calendar’s “List of iCalfromURL Events” stored on the object. That list was populated by an API data call like this:

In the image above, iCalFromURL’s Event is the array of complex objects my API returns to represent iCalendar events. iCalFromURL’s DaysBlocked is the array of date objects my API returns to represent all individual booked days that are part of the Events.

For a little more detail, here’s the setup for the API call itself in Bubble (the dialog that pops up when you configure the parameters for the API call – my API takes a URL to the iCalendar .ics file in question as well as a time zone [so that the date math is done correctly]):

When this “Make changes to Calendar” step is complete, Calendar’s iCal Events is a list of those “weird” iCalFromURL Events and Calendar’s Booked Dates is a list of dates.

So what I was showing you is that YOU CAN fetch a list of complex objects from an API and store them directly in a field IN THE APP DATABASE.

Once the values are stored in that way, you can READ them back, as in my example, show them in a list. But apparently the only linkage between one’s app database and those objects is the field in which those objects are “stored”.

It’s EXACTLY THE SAME as if you had an API that, say, creates and returns a made up book title (let’s call it MakeASillyBookTitle). If you set that API up as a “DATA” type call, you might do something like:

  • Create a New Made Up Book and, in that step, you set the value of a field called “Title” to be the result of the API call to the MakeASillyBookTitle API. The string returned by the API call gets pushed into the Title field of the newly created “Made Up Book” object, right?

This is how my iCalFromURL API is used. At either scheduled times, or as an initialization step in various places, I fire off a workflow like what you see above. Asynchronously, Bubble goes out, pings the API and brings back a fresh version of the events represented by the iCalendar at that URL.

The point being to insure that events I might be displaying in my app are as up-to-date as one can expect.

This all works swimmingly. The only downside, like I’ve been saying, is that the ways that one can USE those previously stored event things are limited because they are (apparently) in another table or something and are not actually “in” my app’s database in the same way that OTHER complex data types are.

Does that help you understand?

Is there a TLDR version? :blush:

If I’m skimming your post correctly. You’re saying you did store the data from the API in one of your tables. If so, then you need to display records from that table,not from the API’s data type thing – which should be something like Calendar;s “Future iCal Events”, not “iCalFromURL Events”

image

This is where I think you would select “Calendar” or “Future iCalEvents” as the type of thing and the data source would be the Calendar tables list of…, not “iCalFromURL Event”

Not like this… you would only select “iCalFromURL Event” if you were going to make the “Data Source” field a “Get data from API”.

Hi K,

You mostly understand me, but I’m not having any “problems” of a simple nature with it.

I don’t have a problem displaying the data. Never have. Indeed, the way one does it (and the reason one doesn’t realize that there are limitations to data stored in this way), if just via Calendar’s (or whatever the list is attached to) whatever_field_the_data_is_stored_in.

(As you pointed out. I’m totally aware of that. That’s how I displayed it in the video.)

My point about ALL of this is simply that it makes no sense that this data exists in a separate table (or the API Ghetto as I prefer to call it). Just seems that there should be automatic “flatten and promote” for data such as this.

It sucks that it’s not searchable. You cannot “Do a search for…” this other-table-data-type or whatever one wants to call it. Neither can you “Create a new thing…” of this data type. Neither can you “Make changes…” to such things. Because the things (while being stored in Bubble somewhere) are apparently not in the APP DATABASE table.

As you have very correctly pointed out, you can iterate over a list of such things that comes back from an API and thereby put them INTO an app database data type, but my problem with that is that it adds a bunch of pointless compute and TIME overhead.

It’s not pointless, and doesn’t use any more “compute time” then if it automatically created all of the records at the creation of the API initialization.

Just ignore those API data type fields in the menu for the time being, until the Bubble team verifies that they’re supposed to be there.

And you can modify the lists. You just have to go through the parent table (i.e. Calendar) to set list, add list, delete list, etc.

Hmm… Am I missing something essential about your technique for flatten/promote such items?

I’m reviewing very carefully what you first proposed. This sort of step:

image

… iterates iterates over a list of things returned by the API, right? I’m trying to AVOID that. In my case, if I want to “convert” the iCalFromURL Events to some parallel data type that’s in my app, I’d have to loop over all of them in this way. This takes time.

Like, we’re reading this one item from the one table and then we’re writing to the app database table and then Bubble imposes a delay before it will process the next one, etc. and so it can take lots of minutes.

This defeats the point of my API having put all the stuff into a nice list in the first place. Bubble is bad at making lists via iteration (I guess not BAD, but SLOW).

Hey K, YES, you are 100% correct about this. One can deal with the LIST. But one cannot deal with the individual objects inside it. This is what I think one should be able to do.

The individual objects are primitives or for-all-intents-and-purposes LIKE primitives.

To put a finer point on it: My API returns a list of JSON things. That list (basically a bundle of those individual things) gets associated with a field. And yeah, you can set/add/delete the ENTIRE LIST. You can also READ an individual item in the list (like Calendar’s iCal Events:item# n).

However, that’s where it ends. Let’s say I wanna modify item #n in that list. It can’t be done:

Calendar’s iCal Events:item #n’s Start (for example – and all other fields in that individual item) are READ ONLY. Because the entire object – keys AND values – is a primitive.

If you try to do this: Change item #n’s Start to item #n’s Start+days 2 (for example), the Issue Checker will tell you that this cannot be done because iCalFromURL is “not saveable”.

This is exactly the same thing that Bubble would tell you if you tried to change the integer 2 to the integer 4. A object in the app database is “saveable”. Saveable things can be modified. New saveable things can be created.

But the things in these lists returned by API calls are not saveable things. They are constants/primitives.

What you’ve pointed out to me is the likely reason WHY they appear this way. (That being, those objects are in some other table.)

Make sense?

1 Like

Again, stop using iCalFromURL Event unless your calling your API. You need to change the Fields in your database like “Future iCal Events”.

The names of the fields don’t make any difference. THAT IS HOW I ACCESS THESE THINGS. They can ONLY be accessed that way. You can’t search for 'em.

And that’s the whole point of my initial post.

I think what you and I are discussing is you are saying, “Well, duh Keith, these values – the individual values themselves – are in some other table, not in your app database.” (Which is the part I was missing and I think your explanation must be the correct one.)

But then I’m saying, "But that’s really silly, right? I’m reaching out to an API to get these things and put them in my app database. I don’t WANT them stuck over in some other table. The solutions for parsing that other table and representing these ‘events’ in my app database (app table I guess) as native things that can be fully manipulated involve iteration INSIDE OF BUBBLE.

But the whole point of passing a LIST (array) of these events in the first place was to AVOID iteration inside of Bubble. It’s a “you just can’t win” situation."

(And that’s a bummer.)

2 Likes

Here’s the question: WHERE does this “Activities” table you have come from?

It’s not some other table, it’s a data type (I know you call these “tables” and that nomenclature isn’t incorrect in a sense) that is IN YOUR APP.

You created it, right? By iterating over the response from your API to essentially (in a very slow Bubbly way) flatten those nested parameters. Right?

You went from this:

To this:

Like you said in your first detailed response (which I greatly appreciate, by the way): “One way to populate these values is to flatten them into one table in the database.”

But that’s not a single-step operation. You had to loop over those lists (of “toRecipients” and “replyTo” lists) using an “API Workflow on a List”, correct?

I know this can be done. It’s just horribly, horribly not performant.

Of course, sometimes that’s acceptable. But in general, since Bubble really, really discourages iteration anything that can be done to avoid it is a plus.

My point is that WE SHOULDN’T HAVE TO DO THIS in this way. Over time, the added time to do this becomes huge. When really, API Connector should have a flattening option that can execute in a real server-side loop and just flatten required stuff in ms, rather than minutes.

The other thing I think you’re missing is that YOU DON’T have to do what you’re doing – IF YOU’RE WILLING TO LIVE WITH or work around certain problems.

For example, if you just took the return values for your API such as “toRecipients” list and “replyTo” list and shoved them into a field on Activity without flattening, you’d still be able to READ them just fine. You just wouldn’t be able to manipulate them directly “in situ” as it were.

An item in the “toRecipients” list would be like:

{“emailAddress name”: “Keith C.”, “emailAddress address”: “[email protected]”}

And that thing would be of type “toRecipients”. It would be one of those “API Ghetto” types. You wouldn’t be able to make new ones, etc. But you read 'em just fine and even flatten “on the page” should your interface need to add other recipients. (Basically, you promote them to full data type things as you need them, rather than in bulk via an API workflow on a list.)

The deal with MY complex data type that comes back is that is slightly different than name/email address pairs. There’s more fields (more data to flatten/promote in each iteration = more slow), and I ping that API pretty frequently. And I kinda depend/expect a rather quick response from the API and I have interface elements (live calendars) that need the data quickly.

So I don’t really have time to wait for an artificially throttled API Workflow on a List to complete.

Here’s the thing that’s driving me crazy:

The way these lists come back is ACTUALLY perfectly fine for me. With the NOTABLE EXCEPTION that the DATA TYPE ITSELF is not accessible in the same way as user-created data type. THAT’s what causes the problems for me.

Another less than perfect approach is to duplicate the API connector call, on the second one receive the data as text instead of interpreting JSON as fields.

Then store the JSON text blob or a subset of it along with the interpreted result, and this can be modified and passed to javascript more easily. Just a pain when working with other Bubble functionality, so that’s why I’d use it on a partial result.

2 Likes

Yes, the Activity is a table (data type) that I created in my bubble.

I wouldn’t say I had to loop over the list, since the list is already related to a “thing”. I would think it’s added just as quick as the time it takes for you to initialize the API call, plus a little extra time to save it to the database.

In the video you can see how fast it is to modify it in the database. I assume the same is true, or very close, in creating the Activity from the API call.

Here’s the API endpoint workflow that’s creating the Activity, the list is just one of many fields – Look ma! No looping required :slight_smile: .

There’s a lot of other steps for processing related items in that endpoint, but the first step is the creation of the Activity.

Yeaaah… well look at all actions on lists you’ve got launching:

There’s nothing quick about when those return usually.

I guess here’s my question about your use case: How soon after you hit your API are you expecting to be able to take action on the results?

Also, I do see this:

Do you or do you not ever decompose those into a native Bubble data type?

Because this is my issue in a nutshell. It takes way too long for me to decompose/flatten/promote all of my “events” things to full citizenship is too much overhead for the benefit.

All I really really really need at the end of the day is to be able to MERGE those things with a parallel data type at the app level. But this simply is not a thing that can be done.

Consider this: You create two data types. One is called Foo, the other is called Bar. They are identical but for those names. They are simple… in addition to the requisite Creator, Created date, and Modified date fields, they have but one user-created field:

“Number”: of type number

Data structure-wise they are identical.

Can you merge them? No. Why? Because even though they are structurally identical, they are different data types.

This is the problem with the API Ghetto lists. You can’t get at ‘em, can’t mimic them, all you can do is read them. Again: because the objects in them are treated as primitives even though we realize they are not that at all.

@mishav knows exactly what I’m talking about here. Help me out, bro, people don’t understand what I’m trying to explain here! :stuck_out_tongue_winking_eye::grimacing:

There’s just no good solution to this and all signs point to “broken.” There’s NO WAY this is what Bubble intended. Have you ever discussed this with support?

I’m like 99% sure that something fairly simple is either incomplete or bugged. It makes NO SENSE for the things on these lists to have a data type but that this data type is not available to the user (the editor user).

It can’t quite be explained.

Hi @Kfawcett thanks for all the screenshot and info. I spent the last 2 days trying to figure out how to make this all work for my use case and it wasn’t till I found this post and your approach that I was able to solve it.

One question I still have:
After you pull in the data from the API the first time, you created the items in the Data Types (Tables) in Bubble. How do you make sure that when you pull in the data again from the API on a later date you only pull in any changes, new items etc, rather than creating duplicates?

I am sure there is a straightforward filter or something but can’t figure it out.

Each entry in the end has a unique ID so ideally, it would update any associates fields that changed from that ID as well as add any new line items for IDs that were created and didn’t exist yet the first time around.

Hopefully, this make sense.

The only way forward to seriously address the list ghetto of API calls is for Bubble to add a Data Type tab to the plugin builder editor. This way plugin authors could create and edit custom data types that would then be available in whichever app installs the plugin. Until then data types sent to and received from API calls will always be second class citizens.

:rofl: :raised_hands:

I was literally having this same thought yesterday. There’s a lot of things about The Plugg and editor that could be improved and not having to use an API call to return an object would be great. Simply being able to define the data tape would make a lot more sense and make editing that data type easier

1 Like

It is a HUGE weakness of Bubble that you can’t created a typed object (a Thing) in the page. It limits the nifty stuff you could do with plugins, for sure.

2 Likes