How can I iterate on creating records from an external API

Need help APIs
1

Hi Bubblers,

I am going crazy trying to find a reliable way to import a large amount of data from an external API (in this case Google Drive). I have the Google Drive API authenticating perfectly and I have been able to write single page responses from the API to the database (scheduling a workflow on a list and bulk create using the Data API) but I cannot figure out how to create a loop that the process can be fed the the reduced list to do another iteration. Ideally I would like to add records individually. I would like to build a process where …

1.) I can know when the first batch of records is created (schedule workflow on a list works and I can pick up the NextPageToken but I am concerned that bad scheduling the next job could blow up the whole process. .

2.) I can be sure that all the records have been created -

I can’t see a way to do #2 this scheduling a list. I thought I had a make shift way of doing this with Bulk Load using the Data API. Again I have it working fine for single loads of 1000 records. The problem is that as the external API call is embedded in the Bulk Data API Post call, there seems to be no way to get the nextPageToken to continue pulling the data that has been requested.

Ideally, I want a way to iterate so that I am adding records individually so that I can do a control list to ensure that all records are completed. I thought I found a way to do this but keep coming up against the list. I can’t figure out how to remove one item from the list of items in the API response and then feed the reduced list back for the next iteration. I thought I had found a solution with Keith’s Crosley’s List Popper & Friends but I am unclear what the reduced list that comes out the other side looks like.

When I go to feed the list as a parameter, I can’t find a way of defining the data type as the API data type that originally started the workflow (it is not offered in the list of datatypes when I configure the schedule a workflow parameters. I tried to “format as text” the list on bringing it into List Popper & Friends so that it would look like the data type that it is supposed to be written (the same way that one does for the bulk create endpoint) but this ain’t working. It took me a long time to get the Bulk Create routine to work. A shout out to @tylerboodman for his post on formatting dynamic values as json safe.

The Json file that I am bringing in has no nested arrays in it. I will eventually want to understand how to deal with those to, but for now, I can get by with uploading the non nested data into Bubble.

Can anyone give me some advice or point me to where to read up on this?

Thanks in advance for your help!!

Cheers,

KH

2

Please post the API call example response body so we can get our head around what you’re trying to do and with what data :sweat_smile: There’s surely a value there which says how many total records there are. If there’s, say, 100 records per page, is it not a case of scheduling a recursive workflow (or schedule on list) and adding a condition so that when this record is the 100th record in the DB (or similarly check using an iteration number on each workflow), schedule a new API workflow that gets the next page of data and begins processing that on repeat until it runs out of records.

3

I know this is vague but this is the general idea of what you need to do:

  • You have to set up the initial API call
  • Then set up an API call assuming you have a “next” token.
  • Then you need a recursive workflow to “gather” all the Google Drive files
  • Then you do a recursive Data API Bulk Create so you can do > 1000 items

:grinning_face_with_smiling_eyes: