[Recruiting for Alpha] Large CSV data imports

allenyang · 2020-05-28T17:02:06.741Z

Hi all,

We’ve been working on a new feature to allow you to import large CSVs of data into your app’s database. This feature works very similarly to the existing CSV upload feature, but it now accommodates much larger CSV files, and the upload will happen in the background once you kick it off.

(Note that this feature will be available to apps on the Personal plan and above, just like CSV export / import today)

If you’re interested in participating in this Alpha to help us test out this feature, please DM me with the ID of the app you want this feature turned on for. The ID is the name of the app that shows up on your dashboard (bubble.io/home), or, when you’re in the editor, look for id=<something> in the URL.

Thanks!
Allen

avern87 · 2020-05-28T18:09:42.266Z

@allenyang Will this affect front end CSV uploads as well? Or is this targeted just at backend database uploads?

allenyang · 2020-05-29T19:41:56.420Z

The plan is to eventually replace front end CSV uploads, but for the Alpha there’s an ‘escape hatch’ where participants can choose to use either method.

shpak.serhiy · 2020-06-08T08:53:51.410Z

@allenyang will import limits also depend on the app’s plan? If yes, could you share the limits for every plan?

mhclapperton · 2020-06-08T11:51:52.215Z

@allenyang I messaged you on being part of the Alpha for this, I have about 750K records to load across all my tables and it isn’t going to happen in the current upload system. My app is https://xcurio-needle-master.bubbleapps.io/

Thanks,
Mike

allenyang · 2020-06-08T19:58:28.965Z

Not an explicit limit, but this does use capacity so the speed of the import may vary on your plan and what else your app is doing.

mhclapperton · 2020-06-08T20:19:34.382Z

It just crapped out on the 396 number again.

I’ll move the file to my hard drive and restart and just leave my PC on. Maybe copying from my Google drive is the problem. Glad to move to the more expensive plan if that is the problem.

allenyang · 2020-06-08T20:38:54.029Z

Sorry, I think the first time I tried turning the feature on, it didn’t “stick”. I tried it again and looks like it stuck this time - please give it another go

The indication that you have the feature is in the Upload popup, you should see a new checkbox that says something like “Use legacy behavior”

philledille · 2020-06-09T06:36:16.828Z

Will the upload also be smart so it can insert, update and remove from the DB?

allenyang · 2020-06-09T13:15:39.407Z

It works by augmenting the existing Upload and Modify features in the App Data tab

philledille · 2020-06-09T17:50:37.724Z

Thanks @allenyang for swift reply. I’m not sure I understand how you mean.

eddie.sellers.jr · 2020-06-10T03:57:21.780Z

Hey Allen, I have exhausted my search for a CSV upload solution. I have a condition where a user would create things via CSV upload up to 800 items at a time. Without this I do not think I can offer this solution to my customer. I would love to take a look at this alpha.

allenyang · 2020-06-10T12:57:33.425Z

To clarify - you can think of this feature as just like the existing bulk Upload and Modify features that you see on the Data > App Data tab, except, after this feature rolls out, it can handle a much larger CSV.

A major benefit of the new approach is that the upload / modify happens in the background, meaning you can leave that popup and go do other things while the process continues.

mystreet · 2020-06-11T10:02:49.097Z

Could you please activate this feature for my app too? https://bubble.io/page?id=mystreetapp&tab=tabs-1

mystreet · 2020-06-14T14:18:27.864Z

@allenyang I have a suggestion.
Every time there is an error with data on a row, the upload stops.
My suggestion would be to implement the option of allowing to skip those rows with issues and upload the rest of the data.
Then, the user would have the option to fix the issues with his .csv file and re-upload.
However, in this instance, your system should only look up what has been modified.
I think this would save a lot of computation power on your servers too.
Atomic writes are good, but given that we have files with millions of rows, the current system is not very officient, because the data can not be 100% clean.
For example, in a file with 250 categories, I had forgotten to input in the database the name of a category, but it was existing in the database. Because of it, the csv upload failed at row 1150 and I had to re-upload.

allenyang · 2020-06-15T13:49:36.048Z

Thanks for the feedback; this (dealing with errors in mid-upload) is something we’re working on improving as part of the Alpha.

(Also, please use the group thread we have for Alpha testers for further feedback like this!)

mhclapperton · 2020-06-18T00:42:29.769Z

Awesome, I had the same feedback. Work in progress! Still very good experience considering this is an alpha,

mamprogr · 2020-10-25T10:26:31.670Z

Could you please activate this feature for my app too? my app id is sirabox

allenyang · 2020-10-25T23:46:21.347Z

Hi all - this feature rolled out in June: [New Feature] [Beta] CSV Upload Overhaul

It’s available to apps on the Personal plan and above.

To avoid further confusion, going to close this thread

allenyang · 2020-10-25T23:46:26.127Z