I am currently hosting an application on Bubble which has a large table in the database. I currently have ~17k rows but I am currently going to increase the count to ~150k. However, I am having issues with inserting and deleting so many rows in a reasonable time. I am not sure if this is because I lack the knowledge in using Bubble or I am simply hitting the limits of what Bubble’s platform can do.
I have a couple of questions related to similar issues:
What is the most efficient way to delete a large amount (10k+) records? I am currently running API workflows from the Data tab to clean my current database and it is quite slow. Is there a more efficient and faster way to delete data in bulk?
Conversely, what is the most efficient way to upload a large amount of records (100k+) into the database? I am planning on using the CSV uploader (I used the CSV parser plugin and the application timed out) to upload everything. However, I expect this to take significant time and also potentially timeout in one single batch. I can also split into batches, but then it would be great to either set up parallel uploads or a prepared sequence. What is the fastest and most efficient way to do it?
Finally, moving forward, I plan on having 1mil+ records in this database in the near future. Is Bubble capable of supporting efficient I/O operations on a database of this size? Do I need to increase capacity and by how much? Is it more cost-effective just to migrate the database elsewhere and perform queries from Bubble’s SQL API?
Note that once the database is filled, I do not expect to perform queries that have more than 100 records maximum. My main issue is with deleting the entries and uploading large batches periodically. If anyone can give me some tips or pointers, I would highly appreciate it!
So far no major updates, I haven’t received any tips from anyone or found a solution that seemed particularly good.
I uploaded the 100k properties using the CSV Upload Option in the Database and it worked, albeit at the cost of me having to keep it going client-side as well as taking more than 15 hours.
Since it will be necessary to also do additional operations and modification on the database (and Bubble’s API does not seem to be that flexible for my needs) I am most likely going to transfer a majority of the app data to an external SQL Database and run queries there, before displaying the data in Bubble. There is then the problem of connecting Bubble with the database through the API, but with SQL I get the benefit of using an industry standard software and having a large body of knowledge for reference and support with my implementation.
How are you currently solving your problem @scharan?
My data base currently has 50.000 entries but I’m expecting 300.000 per month. I didn’t find any decent solution and I’m leaning towards the same path you are… having my DB outside of Bubble and interacting using an API. It won’t be as easy as I would like, but right now is the “best” way.
Has there been any progress on this? I read a great piece about how Bubble can expand CPU operations but this is the second and probably most important piece of the puzzle. After playing with this platform for more than a year and starting to get something ready for launch, my concern is the lack of data throughput should the app be successful. My guess is that I’ll need to move it until someone can identify how existing high volume sites are performing.
Honestly I don’t think the problem here is performance… in my opinion the problem is you managing 50k - 80 - 200k on a data base… if for some reason you need to delete those entries it will be a nightmare… Keep the most important tables/ data base outside Bubble.
