Forum Academy Marketplace Showcase Pricing Features

Database and scaling and Security

So I was talking about Bubble last night to some friends. They had questions about the database and how it would preform if there came to be hundreds of thousands of users on an app with multiple tables for each. SQL and other terms I am not familiar with were raised in this discussion. I am wondering and would like more information on how Bubble structures databases to deal with scale. What can we bubble programmers do to optimize development of an app so that it is fast and efficient even as it scales? Any information that would enhance user understanding about bubbles database strategies and scaling strategies would I think go a long way in convincing people to embrace it. Nothing worse then the fear in the back of your head that success will actually sink you if the databases can’t handle the workloads.

4 Likes

Hey there! Happy to give you a little insight into how we think about this. There are several main strategies for scaling databases:

-Horizontal scaling: Spreading data across multiple database servers.

-Vertical scaling: Building larger, more powerful database servers.

-Caching: taking load off of database servers via temporarily storing results in a different, lightweight system.

Our current setup is that we are using elasticsearch as our primary database store, and redis as our caching layer. We are likely going to migrate off of elasticsearch and onto a SQL-based server (probably Postgres) in the next few months, because we’ve been hitting some limitations of elasticsearch (it does not handle bulk write operations across thousands of items of data very efficiently, and it’s not as robust as we would like; we’ve had to build out custom backups and consistency layers to compensate for some weaknesses it has).

We start new apps on a shared database server. As an app grows, our strategy for managing the growth moves through several phases:

  1. Absorb load via our caching layer so that the primary database is only handling writes and the bare minimum of reads
  2. Move the app off our shared database onto a dedicated database server
  3. Vertically scale the database server via adding more CPU + RAM
  4. Horizontally scale the database via sharding

Steps 1 - 3 are completely transparent to you as a Bubble programmer. We do it behind the scenes as your app grows, and there’s nothing you need to do on your end.

Step 4 is more of a collaboration between you and us, since we need to work with you to figure out how to split your queries up across multiple servers. That said, you need to be really big before we get to step 4. Our cloud provider, AWS, has servers with 32 CPUs and over 244 GB of RAM… one of those servers, with a redis cluster handling 95% of the reads, can easily handle hundreds of thousands of users, possibly millions depending on your app. By the time we get to step 4, you’re almost certainly not an individual bubble programmer any more… you’re a successful team that we can work with as-needed to help you scale.

Does that help answer your questions about scaling?

13 Likes

This was an excellent explanation and I appreciate you taking the time to explain it so well.

I was just curious, as that blue bar on the top of the screen sometimes takes longer then other times, which led me to some worry on if I was not developing my database correctly. I am developing a project management tool which is data and field heavy. Some pages take longer then others to load. Have invested a lot of hours so am glad to hear you guys have really thought the database through.

I had a similar thought about security too. I’ve read a few forum posts where @emmanuel has mentioned issues with implementation of ideas due to security concerns. As someone whose Joomla website was hacked a few years ago (along with a huge number of other Joomla users) I was wondering how Bubble counters these issues.

3 Likes

It is a legitimate concern. Let’s see what they come back with on their security strategy.

3 Likes

@josh

I was wondering if it would be possible to provide an update on the progress with your database strategy, current state and future plans. Would be really interesting to know how this has evolved over the past 3 years.

Iain

1 Like

Well for a start I know Bubble has migrated to Postgres as its primary database storage.

However, I’m interested in knowing more about what the team has done in the background to improve the security since the OP. One of Bubble’s biggest apps was a fintech application so I presume it’s pretty secure, however, it will still be nice to hear an update.

2 Likes

Thank you for sharing the insights. Looking forward for more productive heads-up!
I came to visit this thread to note the obstacles and complexity for DB cache. For now, I’v just integrated PHP redis extension and things going smoothy when using it for my cloud hosting site.