Hitting External API Rate Limit - Scheduled Workflow on a List

Hello all,

I’m using a self-built plugin to get data from an External API. Problem I’m running into is I’m hitting rate limits on importing of an individuals activities.

Here’s what I currently have setup:

  • The API will return a number of activities for a request, say up to 200 per request. Since I don’t know how many I’ll get, I run a subsequent get request (using a “Page 2” parameter) if the number of activities is > the 200.
  • Since the number of activities is unknown and I want to create a thing for each activity, I kickoff a Scheduled API Workflow on a List which reads the API call, creates a thing, populated based upon the List data.

This works and creates what I expect, but it’s slamming the server I’m hitting. For 137 activities, I’m hitting the API call over 1000 times. Presumably for each data field, it’s going out and fetching the data again, but why wouldn’t it simply be reading the JSON from the original GET request? Do I need the backend workflow to do something different?

Appreciate any help on this!

When you set an API Call do a data and not action, the API Call also become a type of thing in DB.
You can just store all information directly into a single field set to the type of the API Call amd you don’t need to se this for each field separately (and also, it can handle a list if needed).

You should create an item that will contain a list of the type from API Call and use this list instead of calling the API for each item in the list after.

Normally, in front end Bubble doesn’t do the request for each field, only once. But maybe it a backend WF it work differently. Inspect your logs to find where it’s making API Call and if this is done for each field

Thanks for the quick reply! The API call is set as data and JSON format.

So… let me just try to restate:

  1. Create a new type of thing that is a list.
  2. Call the API and create a new thing (of the type of step 1) and save the data there
  3. Run a command to process the list and create the individual things needed.

Is that right?

It really depend on how you use data.
If you really need to create one DB thing for each API Call Thing, You seem to have the correct process.
The new DB type will have a field of the API Call data list you currently run in Backend WF. You create a thing in this DB that contain the full list from Get Data from API. You run the WF in Backend based on that list

I need to do a number of calculations or comparisons between activities, based on various parameters. It made sense to me to break them up into individual items, but maybe I can do them a different way… My main concern would be getting/refreshing future activities results which might have the same activity id. If saved as a list in individual items, wouldn’t the database get filled with duplicate data?

When I create the new thing with the field type as the “List of Activities”, I need also say that this is “field is a list (multiple entries)” correct?

Thanks again for the quick response. I won’t be able to try things out till tomorrow as I’m rate limited at the moment…

Yes you need to select the list and set the type to the list (List Athlete Activities according to shedule API WF)

Thanks. I got it working. This thread helped: Need help writing to database with list called from api source - #7 by dustin

1 Like