API workflow on list of API connector objects

I am using an api connector to get data from a third party. The data that is returned contains a list. I can save the response into the DB no problem. I can access values out of it no problem.

What seems to be the challenge is “iterating” over the list. What i’m trying to do is basically Extract/Transform/Load the list into a new data type in bubble (which is a new custom list). I have accomplished this by using the Schedule Workflow on list. as @keith mentions in this post.

what is frustrating is that I cannot pass a single element of the list as a parameter to my endpoint. Instead I have to pass in every property of the list element. So basically I have to extract the values I want individually and then pass them into my endpoint. This becomes like 24 parameters on my endpoint. I would rather pass the single element from the list.

My Question Is there anyway to simply pass an API connector data type into an endpoint, so that I don’t have to break it into all of it’s primative values that I want to extract from it multiple times. When I get it from the API ----> passing it into my API workflow -----> saving it into my datatype in database.

I would take any advice for a simpler way to iterate over a list from an API connector response. Mutate / filter that list. And then save it into the database as a subset / reduction of the original.

For now the scheduled API workflow on list + backend workflow (endpoint) with 24 parameters works. Just tedious passing around 20+ primatives, instead of 1 object in the list.

1 Like

Does your data called in API are “Data” or “Action” ? This is an important thing because if you set “Data” you will be able to select this as list in List to run on and Type.

Also, this will be available to save it as a type in DB (are your saving the json actually as text?)

Thank you for quick reply!

I have it as an action because I trigger the api call on a button press, however I also tried it as data to see if I could select it as a endpoint parameter type. It seems to me the only values available as an endpoint parameter are primatives (number, text, yes/no) and custom database types (User, Events, etc …).

Side note . The response from the 3rd party API isn’t itself a list, however one of it’s properties is a list.

Example JSON Data)

{
    name: "test",
    age: 25,
    friends: [
        { name: "bob", age: 22},
        { name: "joe", age: 24}
    ]
}

imagine a single friend from the list above having 20+ properties. And I want to filter those friends and only save a few values from them.

If you set as data,
You can trigger “Make change to a thing” or “Create a new thing” and use the Get Data from API into a field set the the Type of the API call. This way, you will be able to Run the Workflow on Alist from the type of the API.

Welcome to one of my favorite topics in Bubble…

Indeed, @parker1, arrays (of objects) returned by the API connector are super-problematic. Which is why I’ve dubbed this, “The API Ghetto Problem”.

The array returned in this way is given a Bubble data type, tantalizingly called a, “List of [something]” (a “list of friends” in your example). But no scalar data type called friend is created.

And so, in Bubble, your obvious array of JavaScript objects (an array of friends) is just… a big blob of data.

And this blob, while Bubble understands that it is an array of something we can never quite get to a single something. A something is forever beyond our reach as there is no Bubble data type created for the something.

And yet, paradoxically, our app knows about a fucking LIST of something.

(Please note: The above description is the net effect of this problem. There is no reason for this problem to exist except that the Bubble team is seemingly incapable of grasping why this is a problem. I’ve explained it in many different ways, many different times, and they just don’t get it. It’s almost like Bubble is a low code platform created by a team that is itself… not that comfortable with computer science. Which I think is the problem here. BUT PERHAPS I DIGRESS…)

Anyway, though you can’t get an API_something from this List_of_API_somethings (what you desire to do @parker1 is simply to pass each API_something to a recursive workflow or to a workflow-on-a-list), you can (frustratingly) STILL ACCESS the fields of that API_something. It’s like there’s a phantom something (and, in fact, the problem is that there IS a phantom something, even though no there is no definition for the something).

So, as you’ve discovered, rather than passing a something to your workflow, you must destructure the something into its component primitive data type parts.

At least you can still do this, right?

That is, using your case as an example:

List of API friends :item *n* :name

Will in fact return the value stored in the “name” field for the (entirely virtual mind you) friend stored at index n. [Even though List of API friends :item n resolves to… well, not what we want.]

(Thank the Bubble gods for small mercies, I guess.)

So you have to just suck it up and design your API workflow you want to accept a something (a friend) to instead accept the individual fields of friend as primitives.

I could go on and on about why Bubble does not fix this. (The first response would be, “Well, where would we stop?” Which is a good question. Because of course a friend could have a field on it that is itself another list of objects and then, should Bubble create a data type for those objects? This is a reasonable objection, but the fact is that something like 80%+ of use cases would be solved by going just 1 level down. And something like 99.9% of cases would be solved by going 2 levels down.

[At three levels down (a list of objects returned by an API has field(s) on it that are lists of objects that themselves have fields of objects on them) this gets ridiculous and at that point, there is likely a DIFFERENT API call we can make. Simply put, well-designed RESTful API’s don’t return vastly nested arrays of objects so the first objection is just dumb on closer examination.)]

But anyway, just try and explain this to the Bubble crew. :man_shrugging:

image

Note that all we are asking for above is for the “data type” (object representation) of a scalar component of this list to be created. I won’t continue about the phallacy that to create and store individual objects of this new type would consume massive amounts of compute. (Which is what the Bubble team would have us believe.)

(In case you can’t tell, I’m entirely unconvinced that it would take more than 2X the CPU power to store this “blob” as it would take to store individual, properly-defined Bubble objects, without requiring us to resort to Bubble’s quite-costly recursion mechanisms. But I will save that discussion for another time. Just let it be known that the value Bubble puts on server compute is several orders out-of-whack with what we expect today. And there may be good reasons for this, but those good reasons don’t change the fact that the Bubble value equation is pretty warped. BUT AGAIN, I DIGRESS.)

10 Likes

Oh man! Thank you so much for making me feel sane! I’ve searched google and a lot of these forums for this. Your one response, I mentioned above, is the only one that seemed to answer my question. However, I wasn’t sure of a how so I went with “destructuring the primitive fields” as parameters to my backend…

It’s pretty sad that this is a limitation! when you have an object with LOTS of fields it’s a nightmare.

It gets better. I went with Scheduled workflow on list thinking this would work. It does however I found another fun “bubble” issue. on race conditions having the workflow modify a single list IE add to that list I would consistently get inconsistent results XD. Thus, recursive workflows allowing me to feel confident that only 1 “thread” will be touching my list at a time… Because sleeps (interval) never make me feel like it’s a real fix.

SO, now I’m not only having to destructure my object of 25 fields to pass into the recursive endpoint to start the recursion. I ALSO have to do it again to pass it to itself for the recursion. Quite frustrating…

Thanks again,
Parker

Ps. enjoyed the memes. I’m sure i’ll talk to you more about detailed issues of this platform. I feel like we will be comrades in sorrow. I will say, I do like the platform more than I hate it. And listshifter is :fire:

2 Likes

Note that my project (if I have one) is to fix these issues as best I can. I’m working on a server-side port of my PROCESS List action from List Shifter, which addresses some of the issues here (though only by making it easier to deal with parallel lists of values-that-should-be-fields).

If you’re into making Bubble more performant, powerful, and truly array-oriented, you’ll appreciate my “List Shifter” plugin and “List Popper and Friends SSAs” plugin.

Just search here for those and you’ll find these open up a world of possibilities (or at least ease certain annoyances…).

Noted! i’ve already played around with list shifter a decent bit and I like it, and the list popper I thought was interesting, but for me i’m just going to recursively count down list:count until it’s <= 0.

Love the idea of helping server side. LMK if I could help. I’ve been developing Node.js/Express and javascript for quite a while now. Chose bubble for easy / speed, sometimes I wonder if it was the right one XD.

you got any extra documentation / advice for JS in bubble? the JS toolbox has been really awesome, but the freaking undocumented get_object_from_id and get_object_from_ids discovery was huge for me. The bubble --> javascript --> bubble is kinda painful even with these tools. I don’t always want to build a plugin for simple 1 off tasks.

You seem like someone that would know if there are any pointers / hidden api documentation (like get_object_from_id.

Cheers

1 Like

Yeah, the problem with Run JavaScript is that all it does is throw a string (of our code) to eval().

This is often sufficient.

But sometimes, we want to actually get an OBJECT (custom data type/Thing) from a list of objects. We can only properly do this inside of a plugin.

So you might have noticed that List Shifter’s PROCESS List action has an eval(x) step you can select. THIS eval step preserves certain inside-of-plugin variables, such as “the_list” (the array we are currently iterating over), the various “Result of Step” values (as “rs1”, “rs2” etc.) and one doing that makes it possible (and FASTER) to do certain types of operations (that don’t rely on the “trick” of JavaScript type coercion).

I honestly thought I knew before I knew… I’ve switched to recursive workflow… everything is fine, except, how do I initially destructure? the Schedule API workflow on list allows me to access 1 member of the list This Val's thing. But since recursive workflow doesn’t use on list I don’t have any way to destructure it… How do you start the recursive workflow?

Now I really wish your list shifter tools worked on backend workflows…

My solution for each destructured primative on recursive workflow…

Result of step1 APIs reports: first items trades: item # Result of step1 APIs reports: first items trades: count's <theDesructuredPrimative> and I pass in the :countas `Recursive counter… I have to do that for every single field… :sob:

I honestly don’t understand why my suggestion doesn’t work for you
Instead of using an action, you use Data (or you can clone the call and change it to data).
You will be able to Run the On your Friends list. And each frieds, you will have access to name, age …) Without the need to specify them all in the Schedule WF on a list

And you don’t need to create or make a change to a thing to Create the list to run on, just use the Get Data from API in the Data source for the Scheduled list action.

1 Like

@Jici I’m not sure I 100% understand your answer. It’s possible it does solve my problem.

As far as I can tell. I cannot set the Backend API Workflows parameter type to anything regarding API anything. data or action. I can only set to primitives and database custom types.

You can do this (I think). Just pass the list of broken things to your workflow-on-a-list. In each step (iteration) reference the current workflow’s thing’s fields. (I think…)

Or, in a recursive workflow, do the same. (But use list popper to reduce the list one-by-one.)

See: List Popper and Friends: New SSA Plugin for Your Backend Workflow (API Workflow) Needs

Or maybe not… but I think you can do that?

@keith I had to move away from workflow-on-a-list because it was causing race conditions. So I moved to recursive. The problem I still don’t get is how do I pass the list list popper to the recursive call to begin the recursion? that list is of API Connector Ghetto type, So to start recursion I pass in all primatives for first element in the list. Then, in the recursive workflow, how would I access the list? All I was able to send was the primatives which do not include the list of API Connector Type.

List Popper doesn’t really help me if I can’t get access to the list inside the recursion. Here is another example to explain problem:

this data is return from api Step1 on backend workflow:

{
  "store": {
    "book": [ 
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      }, {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }, {
        "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      }, {
         "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

Now lets say I want to save this data in the database as ExpensiveBook (this is all example not practical or generic data type) .

ExpensiveBook custom data type has 3 properties on it. price, category, isbn.

So I have as Step2 in my backend workflow I schedule an API workflow (recursive) to “iterate” over that list and create a new ExpensiveBook and add it to relevent Stores list of ExpensiveBook.

The problem is, that to initially call the API workflow(recursive), I have to be able to send it either:

  • using List Popper) the list so I can pop an item and then send the next call a shorter list
  • or send it the list and an index that I decrement each time I call the recursive endpoint

Either way I don’t understand how i’m supposed to send the list because it isn’t yet mapped to a datatype inside of my database.

I’m sorry if i’m not explaining this well. LMK if there is anyway I can make this more clear. I cannot share the editor due to NDA stuff. However, I could potentially whip up an example demonstrating this even more obviously (idk if this is possible on hobby tier). I also would be willing to screen share privately.

Man recursive API calls seem REAL slow. I made one simply calling itself and nothing more. I “looped” I tested 5 times, 50 times, and 500 times. It seems that 5 times takes longer per loop cycle. But I seem to max out around 7-8 per second.

“Looping” / “recursioning” (lol) 500 times takes ~60 seconds. Seems pretty slow to be doing nothing, but looping. My best guess is 100 ms latency rescheduling itself.

Hi there!

I’m in the same trouble as @parker1 : The problem I still don’t get is how do I pass the list from my API Call to the recursive call to begin the recursion? ( API Connector Ghetto type)

Any idea on this ?
Regards,

Hello - any updates on how to resolve this issue? I’m trying to import a list of followers on twitter to database objects - running into an issue with working with the data from the API connector and backend workflows. Any suggestions bubble lords @parker1 @keith ?

Just replying to say “still a problem, fellow users I feel your pain, Bubble please sort this out” :slight_smile:

1 Like

Is this still an issue as of 2024? I have the same issue and can’t find a resolution.