My app has about 10 different forms, each with an average of 10 text fields where users will enter lengthy answers to questions, some being many paragraphs long, others one word answers. It is also a place where users will come back and modify their answers frequently. I have set up the app to save each rendition of the change as a new thing, so if they make a small adjustment like a comma addition to text, Bubble will save that full text box of data as a new thing.
The benefit of this is there is audit trial, and timeline of how data was manipulated over time. This is somewhat important but its this feature that will multiple the ‘things’ in my database many many times over.
For example, if a user without any edits saves their data, that’s a 100 new things added to the database. Fine. However, if they were to on average make 200 changes per each of the 100 fields, that’s now 20,000 new things to store per project. There might be 100s of projects in the distant future, so millions of ‘things’ to store. And this doesn’t take into account other store of data that the app will have for other features which I"m planning.
I’m sure most will simply suggest ‘modifying’ a thing when a user needs to go back in and make a change instead of saving it as a new entry, which is fine. But if I do this, how will keep a track of data that was changed over time, for a particular field? Is there a way for the app to only record a new ‘thing’ where a major change has taken place which I can specify, perhaps X characters added etc?
You could save each form as a thing instead of each question.
Then you could have them submit their changes each time, right?
End of day, it shouldn’t really matter. Bubbles dB should be able to query against millions of records no problem.
Now, returning a million records is another story.
Makes me wonder how google docs keeps track of changes over time. It’s obviously not held in some local cache so… where is it and how many petabytes do they store or just past changes to data
I need to isolate each question as a separate thing because it will feed into document templates and so the API will need to know what question, or thing, needs to be place where in the template to generate the document. I haven’t considered storing the entire form as a thing, but I suspect this might be problematical for a couple of things down the line.
A lot of the volume will be made in the initial stage when users fill out their information. They will come in, put in some text, save it, come back again, add some more, save it - do this hundreds of times per field.
Thankfully I don’t need to return all the data to the user, maybe just the last few entries when the answers become something meaningful. At the moment I have a repeating group showing them all of the answers they’ve ever entered for a particular field. In future I’ll need to limit this to not put strain on returning these.
I also though about exporting or ‘archiving’ all of the ‘start-up’ data, so this would be all the incomplete data generated in forms when users are starting out, and export it to another MySQL database or something and call it via an API so at least the load isn’t on Bubble.
I wonder what the data limit of Bubble is too, and whether there is anyone managing millions of records.
Idk about on bubble, but you can definitely do this with firestore/Firebase
HMU for a quote on a custom built plugin for your app ID you take that route! You can literally query millions and millions of records in a second or less on average
I just looked into those two. So you’re saying that if Bubble can’t hack it, you’d suggest using Firestore/Firebase to store and query the data and delivery it via API to the app? Would the API be an bottleneck?
Would it be easy to migrate, I’d prefer to keep everything on Bubble for ease and simplicity, but if I need to reconsider in future, I’m hooping its not too much work to offload data from Bubble into one of those two
Well, I’d love to hear what others have done. It would be a task to transition but also possible. I transitioned an 3,000 row airtable sheet to firestore. It was quick enough.
Push comes to shove, it’s absolutely possible but planning ahead of time is always best.
Yeah, I have worked with clients that have millions of data records in their database. If you are trying to do searches in Bubble and want it quick, we found it faster to upsert the data to TypeSense so they can do searches really quickly. Most of the time, Bubble would suffice for databases, it just depends on what you are trying to do with it. I personally haven’t needed to go outside the Bubble database yet. Although, using AirTable was super easy to setup since Bubble made the plugin as well.
If you are going to eventually transition to another database, I would suggest just doing it now instead of later. Just makes things easier. Otherwise, you will need to figure out how to upload the data in bulk instead of updating one item at a time, which could take days/weeks.
Also, Bubble is working on optimizing things, so I am looking forward to seeing where they go with this.
Thanks it looks like the consensus is that the database can probably handle the millions of records. The speed isn’t that much of an importance to me and I’m sure by the time I make it to millions of records Bubble would have made further optimisations.
