Impossible to fetch more than 50k records via API?

While waiting for others to chime in, here’s an argument for why api data fetch limits should not exist (or should require a nominal charge).

We’re paying for our database records

According to recent pricing talks with @grace.hong (and as generally disclosed by Josh here), Bubble is going to start charging for the underlying memory usage of our records. I told Grace that was not only appropriate but “it’s about time” as, from their perspective, they previously had an unbound cost in how large our databases could be.

The charge will be a few dollars per Gig per month which affords several thousands of database records depending on structure… A-Okay

Fetching 50k records does not exhaust the API rate allowance

Bubble affords 1000 API calls per minute and we can boost that 3 times a month for free or pay a nominal amount to increase the rate. But fetching 50,000 records (at 100 records per fetch) requires only 500 api calls - not close to the limit even if placed in a minute.

The alternatives we’re left with are more expensive for everyone, Bubble included.

Most services we’d like to sync data to accept bulk inserts (some can handle 100 or 1000 records at a time). But if we can’t GET records past the first 50k, the only programmatic mechanisms available are;

  • One-at-a-time: We create a backend workflow that is able to export one record to our 3rd party db. The requirement of one-at-a-time is there because we’ll then have to run a “Bulk” job which runs that backend workflow for each record of interest. Now, instead of making only 500 HTTP connections to complete that sync, we’re forced to make no less than 50,000 connections - way more taxing on Bubble’s infrastructure and ours. And this isn’t even fully programmatic because a person has to click the “Bulk” button in the data tab to trigger said re-sync.
  • All-at-a-time: We schedule a search that loads all 50k+ (could be 700k) records into a backend workflow. We can’t pump that many records into our 3rd party DB at once so the workflow has to slice off the bit it can handle then recursively carry/schedule the remainder into a new instance of the same workflow. Carrying that remainder from workflow to workflow until it’s exhausted is also more taxing on Bubble’s infrastructure - both in memory and in CPU - than if we could just do the same via vanilla API.

None of those solutions can be baked into a plugin, which is something that needs to be possible to easily share such automations with Bubblers.

1 Like