Hey Bubblers,
Before I head over to ask the Bubble team about this, I thought I would see if anyone had any insight into my issue. In a nutshell, uploading a CSV file takes much linger than it seems it should. I just did a test where I uploaded a 141kb CSV file that had 644 records in it. That is ALL it did. Upload it. No post processing, etc.
It took 2 minutes and 22 seconds.
I uploaded the same file to Google drive. It took 4 seconds. Andyeah, Google Drive didn’t have to parse it, but come on, it is a comma separated file.
I uploaded it into my desktop copy of Microsoft Access. It took 3 seconds and it parsed it.
I found a random website that lets you upload a CSV file and it parses it our into SQL for inserting into a database…(who knew) and the upload and conversion took…yep, less than 10 seconds.
Has anyone else run into this and found a way to speed it up?
Thanks,
Marc
Yes same! I uploaded a csv with 9000 records, 2 columns. It took close to an hour. Any way to speed it up?
I am intending to load much more data than that, and this speed worries me that i will not be able to accomplish my desired app.
I don’t remember the full technicalities but it is related to how bubble prioritise things. Uploading data is a lesser priority than other users using the editor or user using our apps. Hence, upload is done in the background when the server has time. The best way I know to upload is to break CSV files down into smaller files (fewer rows), then write an API Workflow (it’s just like writing a standard workflow) and schedule it to run perhaps overnight. Have it upload one file after another and schedule some time between each to prevent hitting your server capacity. It’s not ideal but we’ve all had to adapt to it and find solutions.
2 Likes
That’s sad to hear…How do we find out our server capacity?
The csv i am intending to upload is 10MB in size…
Your server capacity depends on the plan you are paying for. But uploading 10 MG is not a large file. That small. Just break it up into chunks of 1,000 rows at a time. Read this post for more information Best way to bulk upload?
We currently have to rate limit how many uploads we do at once to make sure this doesn’t consume to much capacity for a given app. As it’s in edit mode we feel it’s more acceptable than run mode speed. That’s something we can loosen on dedicated plans though.
3 Likes
Thanks, but it isn’t ME uploading the data, it is my customers and telling your customers to break up files and upload them at times that are better for the technology isn’t a good business decision.
Thanks Emmanuel, but this isn’t the edit mode upload, this is the run mode use of the upload csv file action. This is why I can’t say things like “chop your files up” or “do it in the middle of the night” because it is my customers who are ultimately doing the uploads.
Are you saying that regardless of how much extra server capacity I buy, the rate limiting is going to prevent me from getting uploads to be faster? But if I pay for a dedicated plan, you can adjust the rate limit?
Let me ask you this. The pricing seems to indicate that going from a team plan to a production plan increases collaborators, backup window, etc. but I don’t see anything about it increasing initial server capacity. Does a dedicated plan come with more base capacity? Right now I am paying for 9 additional units on a team plan, so $180 on top of my $165 a month, so $345. I think I read that dedicated plan is $1060 a month. The pricing indicates that CPU is priced at $100 per CPU per month (vs. $20 per month for server capacity in the shared plans) so a unit of server capacity must not be the same as a CPU. So can you give an equivalence? Is having my own two dedicated CPUs on a dedicated plan equivalent to… 10 server capacity units on a shared plan?
I am just trying to figure out if I would have to triple my costs ($345 to $1060) just to get faster uploads OR would I have to pay even more, $1060 PLUS extra Application/DB CPUs to get the same performance I am not getting from my 9 additional units.
Thanks for any insight you can provide.
Marc
2 Likes
Agreed @mguerrasio but it’s one of the costs of using bubble.
What I said earlier was about edit mode, in run mode capacity does help. The Logs Tab will show you how many minutes your app was rate limited for, and how much faster it could have been with more capacity.
It’s hard to estimate the number of capacity units that a dedicated plan offers, but we estimate it to be 40. Which turns out to be cheaper than if you were to buy 40 units on the main cluster, because you commit for a longer period of time, pay quarterly, etc.
2 Likes
Hi @emmanuel,
So my use case is a little different than the OP. I am trying to do a one time upload of a CSV file with 660,000 records in order to seed the database with information that the app uses in performing analyses. I started getting this going and at the rate it was uploading, it looked like this was going to take a couple days. Are the only options for speeding this process up to move to a dedicated plan? Or is there another option like paying to temporarily increase the prioritization of this process. Thanks for your help.
Adding capacity will make things a bit faster, but for a dramatic change, currently, yes, dedicated is the way to go.
Do you think it will be possible to share a dedicated server and price as part of Bubble plan in near future with the goal of faster response time and process? Amazon just announced
that their API can be reach at 15,000 requests per second.
Thanks - I decided to just go with the upload because it would not make sense for me to change plans at this point for a one time CSV upload, although I probably will be doing just that in the future. I noticed the first time I tried to upload the CSV file , I got disconnected from WIFI and the process stalled out. I cleared everything and restarted the upload. and I have about 130,000 records uploaded at this point. If I were to get disconnected from WIFI again and the upload stalled out as a result, is there anyway to get the upload process to continue from where it left off rather than having to delete everything and start over?
1 Like
Since it is you doing the upload and not your customer, the easiest solution is to break your file up into much smaller sections, maybe 50k per. Yeah, that means 13 files for you, but it is easier than having it time out and starting over.
I have been uploading thousands of records over the summer. I have found that if I prepare a csv file of around 150,000 records, I can leave it overnight and it is finished by morning.
I have also noticed that it does matter if your uploading has to create any relations in the data base. This is true especially for modifying existing records using the csv upload to modify option. Because bubble has to search for the existing record using the unique id to then modify it, the modification upload takes even LONGER.
1 Like
Interesting @bubbleguy. I had not considered that. So now I will upload and then later run API Workflows. We have been experiencing power outages so shortening upload times would be good. Nice tip.
So long as your computer will stay on through the night it should work. lol!
After the first night I found the sleep settings 
Thanks, that was the conclusion I came to as well. This may end up delaying launch for me a bit but two weeks is not that bad so long as the files get uploaded.
