Uploading 50,000-100,000 rows CSV

Need help Database
1

Hi there!

I’ve created a backend workflow to let our users upload CSV files via our web app. It’s been working flawlessly for up to 4,000 rows CSV. It uses the 1T - CSV Uploader plugin, which creates smallers batches (100 rows) which are then uploaded through an API Workflow.

However, we are now trying to upload CSV files that contain up to 100,000 rows. We’ve tried changing the batch size, but the pluging either shows uploading file… perpetually or creates the batches and does nothing.

Anyone has had success in uploading these big CSV files?

1 Like
2

That’s quite a challenge.

May I ask what is the use-case for such large volumes? Does your CSV data require duplication to your Bubble app, or could the data be fetched/accessed in another way?

1 Like
3

Check out this post by Eli:

Bubble does not allow more than 50 files. So I’m not sure if it’s even possible to use that plugin, and unfortunately, I don’t have another solution.

3 Likes
4

I would try to find another place online which can accept your CSV then send them to your API endpoint.

Just a thought

2 Likes
5

Thanks for replying! Our users have access to monthly csv reports generated by music distribution platforms that are super complicated and very often exceed 50,000 rows. We’re processing these reports to create more digestible analytics for them

6

Those music distribution platforms don’t support webhooks?

7

Unfortunately, they don’t offer support for any kind of integration. Their customers usually have to manually export these reports from their dashboards :frowning:

8

When I looked into this, I found a bunch of online CSV API companies at a variety of price points. Most required you to contact them for that high of a row count however.

That being said, I’m sure you can automate it to slice them into 10k sections to send over to these systems.

Check out make.com and see what integrations they have.

1 Like
9

Not true! This is just lazy programming. My plugin Better Uploader allows you to upload more than 50 in one action. What he’s referring to is that Bubble does not allow plugins to send more than 50 files at a time. This is normal because Bubble wouldn’t want someone attempting to send hundreds or potentially thousands of files in one go.

What the plugin developer must do is send files in batches, wait until they are done uploading, then rinse and repeat until you reach the number of uploads (this is done programmatically in code). In other words, the plugin developer must use the callback parameter on the upload function.

So the answer to this question is: YES it can absolutely be done, but will probably require some code!

1 Like
10

Hey! Don’t call me a lazy programmer, big meech! :crazy_face:

He was asking about the 1T - CSV Uploader—which is limited to 50 files at a time. Cool to see you have an alternative.

1 Like
11

Sorry - I didn’t mean it that way! I should chill. But yeah, the callback function will let you circumvent that limitation