So I’ve got some data from Best Buy’s Product API and the call I’m trying to make has more than 100 items, which means I’ll need to make multiple calls. I’ve never done this type of pagination and I’m a bit stuck on what to do next.
Looking at Best Buy’s API documention, it appears they support a “cursorMark” parameter to accomplish this:
Via the Bubble API Connector, I went ahead and added a “cursorMark” parameter with the value of “*”:
After initializing the call, it appears I’ve (maybe) done something right as you can see the “nextCursorMark” field appears now:
With that set, going back to my test page where the API data is being pulled into an RG, I’m not quite sure what needs to be done next to make the next call(s) to populate beyond the 100 items. I have it set as an infinite scroll, so hypothetically I’d like it to just continue to populate until it runs out of items.
The RG group is pulling from the API:
Along with the Data source being the same API, specifically the “products” one as that is the only one that doesn’t give me errors:
Obviously, something is missing here. I don’t know how to make that second “call” I suppose and when I do, I don’t know what settings I would set in order to listen/look for that “markCursor” stuff. I’ve scoured the web for ANY kind of visual on how something has shown more than 100 datapoints from an API using pagniation/cursor and haven’t found ANYTHING. Everybody talks about the logic, which makes total sense, I just can’t see WHERE and HOW to implement this. Any help would be appreciated!
cursor is usually a number to indicate what is the ‘next’ item to request. In Bubble for a Bubble API you would see a call with a cursor of 0 and a limit of 100 pulling in items 1-100…the next request would be cursor of 100 and limit of 100 pulling in items 101-200.
Other APIs work differently…in some the cursor is the ID of the next item, and from your screen shot it looks like that is the case here. So, you would need to send in a parameter value of that ID to indicate which items to begin pulling from for the next call.
As to infinite scroll, I don’t believe that will be easy to accomplish as this is setup for pagination.
Appreciate the time you took to reply! That makes sense about the infinite scroll presenting issues. I was just looking for a quick and dirty way just to get 100+ to pass through. So perhaps I’ll need to build in some RG pagination for this test first.
If I were to save this data to my DB for other purposes, from my research is seems like taking a recursive approach (which is something I haven’t personally done yet) is the best approach as I’ll be pulling in many calls over time that will have hundreds or even thousands of products. Would the cursor functionality as you described it still play into this?
Yes, you’ll do your calls in the backend recursive workflow…have a parameter on that workflow to be the cursor value so you can on each recursive run have the call produce the correct results
Okay that is good to know! I’m struggling to see where I would place this command. Here is what I have setup to save entries from my API, based on @jacobgershkovich’s very helpful YouTube video:
I’ve got a button on this page that, when clicked, will fire the API call and set the “cursorMark” parameter as * since it appears that is what Best Buy says to do for the first call:
It then schedules an API WF on a List:
Moving over to the Backend workflows page, you can see the “save-monitors” workflow is setup with the fields I’m wanting to save to the DB:
The first thing the WF does is create a new Thing (Product) with the corresponding fields. (There is a condition that if a Product with a matching Model Number exists, skip it):
And, since the idea is for each “Thing” to have it’s own page on the site, I set a Slug for the new entries:
This of course all works great and fills my database up to the 100 results. What I can’t seem to find any guide or examples on WHERE to reference the cursor mark. My thinking is that if I make the next step “Schedule an API Workflow” and repeat this, is it going to still reference the front-end workflow that initially called the API? If so, where do I go about changing that cursor parameter so that it knows where the API “left off”?
If someone even had an article or video of someone else accomplishing this, I’d be so so grateful. It seems difficult to find information on saving API data to Bubble that involves multiple calls.
Need to setup a recursive backend workflow…not workflow on a list
Oh boy, I diferently did something wrong as when I went to fire a recursive workflow I think it sent off dozens of calls in less than a second and now my API access has been restricted and waiting for approval to resume 
So while I’m getting that worked out, I’m struggling to understand the basic theory here when it comes to saving these products one by one instead of “as a list”. Here is where my head is at:
-
I fire off a request to an API that will return say 500 items, but only 100 per page, which is also the limit of Bubble’s API.
-
In a normal “Schedule Workflow as a List” I would make the initial call pulling in 100 results. Then, after the 100 items are pulled, my workflow uses the “cursor” parameter to call page 2, and so forth, until all of the pages have been exhausted.
As I understand it, the cursor value that the initial API call gives you is referencing the first item on the following page. So in this case, item #101 on page 2.
In a recursive workflow only one item is being worked on at a time. So, as I understand it, I would need to do some type of “save first item’s value(s)” and and the end when I recall the same workflow, I’m adding a condition that subtracts the item we just worked on, and thefore Item #2 now becomes the “first” item in the repeated workflow and you continue until there are none left.
That makes sense when dealing with internal data, but in the case of my API data, it will just go item by item until it hits item #100 and then be done since that is all the call brought me. At some point that second call to the API needs to happen using my cursor value but if I did that on each workflow I’m pretty sure
A) It will fire off an API call for each item (which would be terrible)
B) Each item would just be the first item of each page, since that is where the recursive value is pointing to.
There is a ton of great documention on recursive workflows when it comes to modifying existing data in your DB, but when it comes to pumping in 100+ items from an external source, I don’t recall seeing ANYTHING on this.
So if any Bubble YouTubers are out there who need a new video idea, I’ve definetely got one for you: Save more than 100 items to your DB via an external API 
1 Like
I use list popper and friends to make life easier in backend recursive workflows. Check out that plugin.
I also use parameters in the backend workflow to track the progress of the recursiveness to know when to perform another API call during the recursive process.
