I am creating an app similar to Apollo.io where users can search and select the leads that they want to purchase. This app currently has more than 1 million leads with over 10 million datapoints which has the potential to grow even more in the coming months/years. Even though the overall data weight is actually pretty low, I am storing the leads in a third-party database because I understand that the way Bubble handles very large sets of data (especially in searches) is not the most efficient.
While searching and displaying these using an API call works well, Bubble slows down after around 10,000 leads are returned (even though I am only returning 500-1000 leads per search and using pagination and only displaying 10-15 leads on a page).
Using the above expression to merge the next set of returned leads into the repeating group to display them as a single list so users can go through them as a single list instead of multiple fragmented ones.
Also creating a list of “selected leads” so users can dynamically update which leads they are actually going to purchase is very limited. Creating a list of leads in a custom state slows down exponentially when it surpasses 700 and crashes the app at around 1500 (even though the data returned doesn’t even touch 1mb, the RAM usage in the Chrome Task Manager reads as 1.3gb and crashes around 1.7gb).
I tried storing only the UUID of the leads in the custom state list and the RAM usage is kept very low but still massively slows the app down at 1000 list size. For reference I want to be able to select at least 10k leads at a time.
Currently I am trying to save the lists of selected leads on the third-party database and return it when referencing it but Bubble gives warning that the app will slow down when returning more than 1.5 mb in workflows (roughly 10k leads) and is noticeably slower when 20k leads are returned.
So my questions are:
- Is there a way to allow for more leads to be returned without slowing?
2a) Is there a way to have the list of “selected leads” done on Bubble without sacrificing performance?
2b) I have considered having a datatype for leads and saving only the UUID and having that as a list on the User. Will this potentially slow down performance or is this viable?
3a) The features themselves work, but is doing this on the previously mentioned scale even viable?
3b) Would something like this require a dedicated server?
I would really appreciate insight into this matter. Thank you in advance.