Architecture for Bubble and Big Data 30 million records+

Need help Database
1

Hi,

I’m exploring backend hosting options for my Bubble app. My record base starts around 30 million and will grow quite quickly.

Has anyone experience hosting their large databases with Bubble versus external providers (Xano, Firebase/GBQ, Azure, AWS, BackendLess etc…)?

I’ve explored dedicated hosting with Bubble, but from what I’ve researched it may not be suitable for this volume of data. I also understand loading the data might take days.

If I host using one of the above cloud services and expose data via API, will it have a performance hit?

What’d be the best way to automate ETL for my data on a weekly basis?

Any help greatly appreciated.

Thank you

Chris

1 Like
2

A few months ago, I had experience working with an external database and linking it to Bubbles. Here are some tips I learned during this experience:

  1. The external database can only handle up to 5 requests sent simultaneously, and the 6th request will not start until one of the previous ones is finished.
  2. Writing queries in the SQL Connector plugin is possible, but it has some limitations.
  3. Regardless of whether requested data is used or not, or if there are duplicates, all data will be loaded.
  4. Request time is definitely longer when using an external database compared to using the Bubble database.

In conclusion, I do not recommend using an external database to store app essential data, such as data needed during page-loaded. You can store infrequently used data outside the app, but using an external database for essential data can be problematic.

I would like to thank the Bubble developers for creating an excellent way to interact with the backend. Although we may not have extensive knowledge to optimize queries, all queries are optimized before use, ensuring that app performance does not decrease.

One notable feature of Bubble is that it only loads the required data. For example, if we set a data type as a repeating group source but only use its name inside the repeating group, Bubble will only load the name fields, and other fields will remain ready to load.

1 Like