Architecture question for a CRM (100's of thousands of records per user)

Need help Database
1

Hey all,

I’m sure this is a simple question, but I really want to get my head around this as we consider moving forward with another project.

Building a CRM for our users (MVP, before doing a full custom build in our main app) is an obvious product extension for us.

What I can’t get my head around is if Bubble can handle (without having to scale to a silly high enterprise price point, early on) so many DB records.

E.g. if we have:

Customer 1, who has 20,000 contacts in her CRM
Customer 2, who has 100,000 contacts in her CRM
Customer 3, who has 30,000 contacts in his CRM

With only 3 customers, we suddenly have a single Database ‘Thing’ called Contacts, with 150,000 records. Now, when we have 100 customers, that’s 1.5m records, 1000 customers, 15m records, and they are all pulling data from the same pool of records - as I understand how it works with Bubble.

Now, am I missing something here? Is this how it would be normally administered anyway with a custom solution and MySql DB? Or, do you create a new DB for each user, so they are only ever searching through their own 30,000 contacts, instead of 15m of contacts for their 30,000 contacts?

If someone has the expertise here to explain how this is typically handled, if it all goes into the same DB (e.g. contacts) or if I need a custom solution so each user gets their own DB I would appreciate it - this will help to ensure we don’t go to far down the rabbit hole.

Thanks!

Matt

2 Likes
2

Friendly bump!

3

I think you would have a data field on the user that is contact_list which is related to the contact data type, so you have on a user a list of all their contacts. That way when you want them to search you just have them search

current users contact list

instead of search contacts and then filtering

1 Like
4

Interesting, this is making me re-think a few things on the other flows too.

So, are we thinking then that normal architecture for such a scenario is that the DB would simply have millions of records with chunks just assigned to a user

5

I would consider storing those records on an external database and just use bubble to pull the required record.

Doing searches on bubble with so many records will be poor at best. Use the external database engine to do the hard work.

1 Like
6

Yes, and you could also on the contact data type have a field that is user which relates to the user who the contact is of…this could also be a list of users if you’d assume contacts could be shared between multiple users.

But I believe @StevenM Steven is correct to suggest an external database…maybe something to consider down the road as I personally have no idea how difficult that would be to implement, or what the costs are associated with it.

1 Like
7

Nobody really has that many contacts they need to process at one time.

As in “Hey I need to find Bill” and searching every single record.

In reality you will search for “Bill” in the US who works at Microsoft or something like that.

Large databases are not, per se, an issue for Bubble. What is an issue is understanding how to avoid going through several million records individually looking for something.

You can just as easily screw up a query using SQL and mySQL by making it do a tablespace scan. But in that case you have more diagnostics available.

3 Likes
8

Thank you Nigel, this makes sense to me.

I guess what i’m struggling with, is not only does Customer 2 have 100,000 contacts, as a ‘Contact’ thing, but if we had say 1,000 customers all with ‘Contact’ things - I can see that just being really difficult for Bubble to handle in just one single Database. I understand what you are saying, if we encourage the user to search using multiple parameters, but it would still be searching the ‘Contact’ database with millions of records.

I’m a complete amateur in this regard, so trying to get my head around best practises. It’s clear that the most optimal approach would be to get some support in setting up and connecting to an external DB.

9

Hi there
I’m definitely not a pro at CRM building, but had some experience with big data solutions. Our company was assisted and guided through a variety of industry-specific data-related business solutions by Avenga data service company. CRM is a valuable tool which handles huge data volumes, but it can only be utilized to its fullest with synchronization options including data storage. What I’ve learned so far big data implementation services make a real difference for businesses that plan to make use of big data.