Hi all,
We’ve been working on a new feature to allow you to import large CSVs of data into your app’s database. This feature works very similarly to the existing CSV upload feature, but it now accommodates much larger CSV files, and the upload will happen in the background once you kick it off.
(Note that this feature will be available to apps on the Personal plan and above, just like CSV export / import today)
If you’re interested in participating in this Alpha to help us test out this feature, please DM me with the ID of the app you want this feature turned on for. The ID is the name of the app that shows up on your dashboard (bubble.io/home), or, when you’re in the editor, look for id=<something> in the URL.
Thanks!
Allen
6 Likes
@allenyang Will this affect front end CSV uploads as well? Or is this targeted just at backend database uploads?
The plan is to eventually replace front end CSV uploads, but for the Alpha there’s an ‘escape hatch’ where participants can choose to use either method.
@allenyang will import limits also depend on the app’s plan? If yes, could you share the limits for every plan?
@allenyang I messaged you on being part of the Alpha for this, I have about 750K records to load across all my tables and it isn’t going to happen in the current upload system. My app is https://xcurio-needle-master.bubbleapps.io/
Thanks,
Mike
Not an explicit limit, but this does use capacity so the speed of the import may vary on your plan and what else your app is doing.
It just crapped out on the 396 number again.
I’ll move the file to my hard drive and restart and just leave my PC on. Maybe copying from my Google drive is the problem. Glad to move to the more expensive plan if that is the problem.
Sorry, I think the first time I tried turning the feature on, it didn’t “stick”. I tried it again and looks like it stuck this time - please give it another go
The indication that you have the feature is in the Upload popup, you should see a new checkbox that says something like “Use legacy behavior”
Will the upload also be smart so it can insert, update and remove from the DB?
It works by augmenting the existing Upload and Modify features in the App Data tab
Thanks @allenyang for swift reply. I’m not sure I understand how you mean. 
Hey Allen, I have exhausted my search for a CSV upload solution. I have a condition where a user would create things via CSV upload up to 800 items at a time. Without this I do not think I can offer this solution to my customer. I would love to take a look at this alpha.
To clarify - you can think of this feature as just like the existing bulk Upload and Modify features that you see on the Data > App Data tab, except, after this feature rolls out, it can handle a much larger CSV.
A major benefit of the new approach is that the upload / modify happens in the background, meaning you can leave that popup and go do other things while the process continues.
Could you please activate this feature for my app too? https://bubble.io/page?id=mystreetapp&tab=tabs-1
@allenyang I have a suggestion.
Every time there is an error with data on a row, the upload stops.
My suggestion would be to implement the option of allowing to skip those rows with issues and upload the rest of the data.
Then, the user would have the option to fix the issues with his .csv file and re-upload.
However, in this instance, your system should only look up what has been modified.
I think this would save a lot of computation power on your servers too.
Atomic writes are good, but given that we have files with millions of rows, the current system is not very officient, because the data can not be 100% clean.
For example, in a file with 250 categories, I had forgotten to input in the database the name of a category, but it was existing in the database. Because of it, the csv upload failed at row 1150 and I had to re-upload.
Thanks for the feedback; this (dealing with errors in mid-upload) is something we’re working on improving as part of the Alpha.
(Also, please use the group thread we have for Alpha testers for further feedback like this!)
Awesome, I had the same feedback. Work in progress! Still very good experience considering this is an alpha,
Could you please activate this feature for my app too? my app id is sirabox
Hi all - this feature rolled out in June: [New Feature] [Beta] CSV Upload Overhaul
It’s available to apps on the Personal plan and above.
To avoid further confusion, going to close this thread
1 Like
