Database Migration

I have an old .net application, a basic membership system for a local pool. Was doing some testing to see if Bubble would be a good replacement. So far it looks pretty good, probably can do almost everything without code. I am now testing some database migration code, just basic python scripts to read the old database and insert the stuff into Bubble. I have a couple questions that some guidance would be appreciated!

The database is not huge a thousand members with a number of tables that track various activities of the members, like fees and guest passes. The biggest is Sign Ins, maybe a couple thousand entries for each member. I was adding all the tables as lists of things to the main membership record. It all works just fine but it is really slow to add all the records for a given member. Would it be better to do a CVS file upload? Also given the size of things is it better to do references to the membership record instead of list of things, I hit a limit, “array length exceeds the limit of 200 entries”, it says contact support to increase but seems like maybe from a scalability perspective references might be better? Last one I am testing on a free account which is awesome, but performance with API calls is pretty slow will this improve greatly with the paid plan options, not sure it matters given data migration is a one time event but wondering, I eventually need to pick a plan!?

Thanks, Bubble is really cool!

Are you using the backend data uploader (ie. from the application builder’s Data tab) or are you using the Upload a CSV action as part of a workflow?

For larger amounts of data, I’ve found it best to use the backend data uploader.

i have used the csv uploader to build a dataset (of approx. 11,300 entries) and it works very well, albeit a little slow! FYI one of the things i have not been able to build/create is users through this method (if that is something that you need to build? @tbbacon)

Am currently attempting to run an api using the bulk upload to create users but have not been successful as yet, @dan1, any suggestions regarding the actual api that needs to be built to enable the ‘bulk’ data upload feature, as below?
37%20am

none of my api’s are actually selectable in the drop down list?
11%20am
probably just a simple fix, but would be great to get some input from those who have successfully bulk uploaded data!

Just to make sure we have terms straight. There’s the Upload feature and there’s the Bulk feature.

With the Upload feature, you can create new data points (users) in bulk. With the Bulk feature, this is not intended to create new users. Rather it is meant to run on existing entries in your database.

For the times I’ve used the Upload function, it’s worked smoothly enough for creating new users. However, you’ll encounter issues when you have users that have references to other users during the creation process. (Ex. created User B is created user A’s manager). It makes sense that this may cause issues, because the user B doesn’t exist as a reference point when User A is created. (That’s why I used a cleanup API workflow).

When I omitted those fields (created a placeholder field for that datapoint, which was used for mapping), it worked just fine.

Thanks for the info. I was just use the “data api” to create and update records (I wrote a python script that is doing data migration from a legacy database, it is doing a lot of adjustments on the data during the migration). Definitely don’t need to do it as a workflow. I think I will alter my script to output CSV file and then try the CSV Upload on the data tab.

I am using a python script with the data api and part of that is creating users, here is the python script if you find it useful… Thanks for the insight on the CSV upload my database is smaller then that I will give that a try instead of the API!

headers = {“Authorization”: “Bearer 03b9757715e006a2bb531138cc21907f”, ‘Content-type’: ‘application/json’}
url = “https://thpool.bubbleapps.io/version-test/api/1.1/obj/

#Create User
username = “test”
password = “testpassword”
body = {“email”: username, “password”: password}
response = requests.post(url + “user”, json=body, headers=headers)
print(“Created User:”, response.status_code)
if response.status_code != 201: print(response.text) # 201
results = response.json()
userid = results[‘id’]

1 Like

thanks @dan1 and @tbbacon,
have resolved the issue by adjusting my API endpoint to ONLY one parameter and then setting the correct TYPE of that parameter to match my dataset!

on another note, uploading the csv data to a new dataset (ie. WEB user data downloaded), then running the API (from the Bulk-operation-test function - in app data view) on that data set allowed me to create the user and modify a heap of details associated with that user.

i appreciate the input, cheers!

1 Like