Effective ways to speed up Recursive workflows?

[posting]

I am running a process which does this:

1. Get Paginated list of items from Shopify .
2. Save this list of items in a local list in my database (pictured below)
   
3. When the pagination is complete, a recursive workflow is called. This workflow has one job only: Traverse the list, for every item in the list, create a new “Product” and push this item into a data field of that “Product” (as pictured below).
   
4. When an item in the list has been processed, it is removed from the list. (pictured below)
   
5. The workflow then calls itself again, unless it has finished processing the list.
   

Now the problem:

The workflow is simply too slow. It’s quite fast when the list is small (25-40 items), it makes products quickly. About 3 products in a second. However, when the list is large (100 items) , it’s extremely slow.

Some guidance on what I’m doing wrong would be extremely appreciated. I have already experimented with having a pause before a recursive call, but the problem persists.

[boston85719]

Hard to tell exactly what is going on in the screen shots as it is not clear the database structure you are using.

Looks like you have a User with a data field that is labeled ‘temp products’ which is a list of things. I can’t tell if that list of things is a related field to a data type in your database or if it is coming from the API call.

It is not easy to figure out the root cause because I am unsure of what the ‘pagination is complete’ means (is this the API call paginating in someway through it’s entire list and adding each list of items from each page into a single list of items on your User data type or are you taking one API call result list and triggering the recursive workflow at that time.

My approach would be, if I were to be paginating through API results in my backend workflows, to incorporate the paginating API calls in a recursive backend workfow which will trigger the other recursive backend workflow you laid out in your question. This way I avoid using the User data type to store a temporary list value and get straight to sending the API results list into a backend recursive workflow to save Create the Product (for use).

That would speed it up as it would avoid making changes to the User in step 2.

Or in the least, just send the temp products as a parameter list value on the backend workflow trigger and just operate on the parameter list for your recursive settings, and if you need to have the temp products data field on the User data type, then have two parameters as list values both on triggering receiving the same list and the one used for your recursive settings and the second to keep the list as it was when triggered so after the recursive actions are complete, you can run the action to make changes to the User and remove from the temp products the list in one action, instead of doing a single item from the list in each iteration of the recursive workflow.

Learn More

Boston85719 is an expert Bubbler with a decade experience as an educator. Real name Matthew, he has been actively building SaaS apps, marketplace apps, scheduling apps and more for clients, himself and for sale as templates.

As an official Bubble Bootcamp Instructor, he leads Bubble Bootcamps on a regular basis.

Always willing to offer advice via the Bubble Forum, Matthew also offers Private Personal and Group Training Sessions.

Through his site, NoCodeTrainer, Matthew provides a range of tutorials with editor access to help you jumpstart your Bubble development.

Always accessible you can send Matthew a private message via the forum or send an email directly with your requests.

Be sure to checkout the Stripe Integration Course when you are ready to integrate Stripe payments to start monetizing your application via product sales or subscriptions.

[posting]

Hi boston, thanks for replying.
I’m sorry if it came off as confusing, let me elaborate a little bit more.

The “temp collections” is a list that I get from an API (Shopify). It’s a list of products. It has many fields inside, including it’s own list of variants (which is an object with it’s own data fields).

How my workflow works is:

This first workflow, paginates through all the products in a shopify shop, bringing in 50 products per API call.

As you can see, the temp products is a list that the API sends me, which I then save to the user.

After this workflow has finished running (pagination has stopped), it calls the next workflow (Called make_product_item).

This workflow accesses each individual item through the “temp products” field in the User data type, and starting from the first item on the list, it creates a new type “Product (for use)” , pushes the item into this new type, and adds this new type to the User. (pictured below)

It then calls itself.

Now, the reason why I’m saving this list inside the “temp products” field in the User data type, is because it makes it easy to access individual items in this list.
I cannot send this list as a parameter because I cannot define the type of the parameter to be equal to the type that an external API gives. (For example, the list that I’m getting from an API is called “Get Products-products”, however I cannot define a parameter with that type, hence I have to save the list inside the User, and then access it by accessing “Current User” inside the workflow.

Thank you for your patience. Hope that clears things up.

[boston85719]

Okay that makes sense. That really is a limiting feature of the backend workflows Key type that it is not possible to have it be the same as the type returned by the API call…that really limits the possibilities of how to set this up and bypassing the temp_products setup.

Seems like you probably have it setup as best as can be considering that limitation of the parameter key type.