Most Efficient Way to Modify Data Through CSV

Hi Everyone,

I am trying to do a large modify date through CSV operation on the backend and it is taking a very long time to validate the data. As of the time writing, it has been about 13 hours and is still in the validation phase and I do not have an indication of how much progress has been made.

I am trying to modify ~150K rows in the db and one of the fields is a list of items in the db. Essentially, I am trying to use the bubble db as a traditional relational db with a one-to-many relationship for this data type.

I was wondering if there is a more efficient way to do this. Currently, each of the records has a list ID numbers, each of which is associated with the db record that should be attached to the item in question. The ID numbers are just integers generated by the program I wrote to create the csv in the first place. Would this be quicker if I used the bubble generated unique IDs? I also considered doing a bulk DB operation but wasn’t sure if that would be any faster. I would test these things out myself but don’t want to interrupt the current modify operation unless I have a better solution in hand since it is still running and I do not know if it is on the cusp of finishing.

Any help would be appreciated since this is the last thing I need to do to launch my app :grinning:

Hi Ryanellman!
Well I don’t have enough experience in Bubble to help you a lot, but I’ll try with an idea.
Why don’t you make all these changes within the database itself ?! You could do a procedure that does all the work and then you can save the file in a CLOB field or send it somewhere and retrieve it via the URL.

I hope it’s a light! Good luck !

Thank you for the tip! While I don’t think this will work for my case, I appreciate the help!

What I settled on was just connecting the things as they are used. If each record contains a reference to the database objects that are supposed to be connected to it, the first time that object is “used” by a user, it will do a search through the database for all the objects it should contain and add them to the field as a list on the original object. Then the next time the object is used, the app will just access the associated records directly.

I am still having problems with modifying through csv upload when trying to add the lists of references to my currently existing database objects, but once that goes through I should be good to go.

Hi :raised_hand_with_fingers_splayed:

To connect a record to the thing, you need a field with the same data between your csv and the “thing”. You can use the unique ID, or another field of the thing “User”.

For example when you have in the thing “User” and in excel, a common field called “Driver License”,

On the uploading settings, you can click on “Map Data”, and then select this “Driver License” field as the related field to the Thing, to let Bubble associate it to that thing.

For example, in this pictures I want to upload some persons from a csv, and I want to relate them to a thing “Company”.

I will use the field “Identification” in the thing “Company” and has the same data in my excel, in the column “Company Identification”.

Is that what you need?