Populating my database via Recursive Workflow?

Need help APIs
1

I’ve spent nearly half the day going through every possible piece of content on the topic of recurring workflows and still can’t figure out a solution and would love some help.

I’ve got external APIs that will be sending me information on anywhere from 150-70,000 products. I want to save that data to my database and then run recursive workflows to keep the pricing information on these products up-to-date.

The initial problem with Bubble is I’m limited to 100 items per call. As I researched how I can paginate my way around that, it seems “recursive workflows” are the answer. Every example I see almost always talks about how up update existing things, not create them, especially when using data from an existing API.

The documentation on this is so vague it’s frustrating. I get the logic of it all, I just can’t seem to find a way to implement this and it seems like it should be such a simple thing. Any ideas or visual resources anybody can point me to that talk specifically about creating a bulk amount of data via an external API?

1 Like
2

Understand your pain. Bubble is weak in handling lots of data entries at once or updating lots of things.

Can you also keep those datapoints outside of Bubble and access them via API calls only when you need them?

3

I have implemented a workflow for a similar task. I have iterated around 5M records via API and copied some of them into the Bubble DB based on some search constraints.

There are several workflows involved. The major are two of them:
WF1 : Recursive WF that gets the list of 100 records via API and schedule API workflow on the list (WF2). Then WF increases the cursor (aka offset) by 1 and schedules itself again.
WF2: Accepts a list of things from the API. It takes an API thing and creates a thing in the DB. Here you need to experiment with the interval because you can max out the app capacity and the WF1 and WF2 will stop.

Actually the capacity in Bubble apps seems to be not constant. Even if you select a safe interval between the workflows the app still can max out randomly. It can be fixed with an autorestarter WF that will relaunch your parsing WF if it stops.

Also, keep in mind that the parsing is pretty slow. It will take 2-5 sec per record. So you can calculate how long it will take for your use case.

2 Likes
4

Thanks for the reply @shpak.serhiy! This is comforting to know that what I’m needing to accomplish is at least possible.

Do you by chance have any screenshots or visuals of what you described? Is your method suggesting Bubble runs though 100 items at once and then paginates through, or will this still handle each object one by one?

5

This week I am writing a guide on parsing data with Bubble. It was on my list. I’ll post the link here.

Can’t share screenshots from my current setup.

Both.
The app loads 100 records and passes these records the action “Run API workflow on the list”. And then the app schedules loading the next 100 records in (let’s say) 30 sec - the time frame sufficient to process 100 records.
What is happening when the app launches “Run API workflow on the list” - it just schedules 100 tasks spaced by the time period indicated in the action settings.

I assume that my explanation might sound complicated or not clear without screenshots. I’ll describe it in details in the blog post.

1 Like
6

Hey Brandon, Sem has a valid point here. Are those pricing updates coming from / retrieved from an external system? If so, is there anyway to have relevant price updates pushed to Bubble? To run recursive workflows on 70.000 items might not be very resource (incl. time) efficient. Of course, it is always an option to test it first and change your approach based on the results.

7

It’s a very valid concern. My data is coming from multiple API endpoints (different retailers) so one could argue that the best structure would be to point all of those retailer APIs to an external DB like Firebase, and then send that data (via a single API instead of multiple) to Bubble’s DB.

Yes, I’d still be saving the data on Bubble’s DB but with Firebase I could setup a webhook where data is automatically sent from Firebase to Bubble ONLY when Firebase’s DB detects a change in data (price update).

I’m still in MVP mode and need to move quickly so at the moment handling everything in Bubble, even if it’s not as effecient, is my best route. Also, I’ve done the math on how many products my site will have and it’s in the 40-50k range.

@shpak.serhiy I’m looking forward to this guide! If you can remember to tag me when it’s live, please do!!