Is there an import limit? Can't import 1000+ items using api connector

I am trying to import 1,000 item json url using the GET method in API connector but the most I can import at a time is 100 items. There are many pages, which means I have to create different calls to import a total of 10,000 items. When I try to upload the first page, 1,000 items, the console crashes. How can I upload large amounts of data?

I believe there may be limits on each GET or POST so the solution is to treat it like pagination and grab, say, 100 at a time. I’m not entirely sure of this - I just recall reading something about it in the forum a long time ago.

Also, how are you loading the data? Using the run a workflow on a list, or the new Data API Bulk Data Loader? I wouldn’t be surprised if the prior is rate limited and the latter is much easier and faster to load 10k entries.

Here’s a working example of the Data API Bulk Data Loader if that’s helpful: How to write data to BubbleDB using Data API Bulk Data Loader

Best of luck!
Scott

@nomads32n - There is a 1000 row limit f you’re using the Data API bulk loader.

“The maximum number of items that can be created via a single bulk request is currently 1000. There is also a limit of 4 minutes for the request to complete; if it takes longer than 4 minutes, items that have not yet been created will be marked as errors in the response. Creation speed is determined by the amount of available capacity your app has, as well as the size of the items that you are creating. Note that bulk creation can consume a lot of application capacity, so we recommend that if your app is being used in production and capacity is limited, that you test bulk creation in smaller chunks and work your way up.”

We’ve implemented this new feature with great success loading <1000 rows per call. We’d be curious to keep in touch if find a solution to do >1000 as we’d potentially like to bring click data back into our app on a regular basis. Currently, we’re exploring how to aggregate it before POSTing it back.

1 Like

Thank you. I’m not sure its not working, I am trying to import a json url file. Wouldn’t I use the GET method instead of POST?

It can work both ways but GET is probably what you want. We often use POST simply because we’re sending data to an API and then receiving a response that we then store.

Can you clarify what you’re currently doing? From the above, I understand you’re requesting data from another site using a GET command from the bubble connector. Is this correct? If so, are you bulk loading each row using the Data API?

How would I correctly bulk upload from an external json api url? When I try to Bulk import from external Api, I don’'t get 1000 items returnned. The most I can get into the database is 100.

If it’s working but cuts off at 100 and the job is <4mins, I’d recommend you submit a bug report from the editor. I haven’t been running jobs 100+ yet, so I cannot intelligently weigh in.