Hello everybody!
I would like to know your experience working with lots of records in Bubble. 
I have the following example:
I have a table that has fifty thousand records (50k). I have a ‘Do a search for’ with a simple filter of Company = Current User Company. The table has around 30 fields, I know it could be split into smaller ones, but I’m working with what I have
I have a Company that has 9k records. I use this search in a repeating group and then use :items until 100 in my repeating group. I want to implement pagination and in the UI I show the total count of the records, which in this case is in 9k.
This search takes a lot of time in my opinion, and I wanted to try other methods.
In Bubble docs, the operator**:items until** says it returns a sublist of the list, from the first item to the #th item.
Does this mean Bubble queries the 9k records in that single search and then displays only 100th? That’s my assumption.
The other approach I thought would be to use the Data API and query it using the API Connector.
I could use the limit parameter and also the pagination fields that the Response has. But is it the same as using items until ?
GPT says it’s better to use the API connector and query the Data API, but has anyone faced a similar situation that can validate this?
Thanks to everyone for your time!
Hey @design.agx - 50k records should be perfectly fine (millions should be fine).
What you want to avoid is loading too many records to the page, Bubble usually handles lazy loading (not loading items that are not shown), but if you’re using an advanced filter this might be a problem.
Do you want to share your search here, and maybe we can help?
5 Likes
Really depends a lot on what you consider best, so, is best, the fastest, cheapest, or perhaps, best being only show what is necessary out of the 9,000.
Also, depends on your goals of how those data entries are used in the app by the company and employees.
What is the use case, end goal and how are the users needing to access these values? Need a graph showing full history, or just a spreadsheet with last weeks new entries?
1 Like
Yes, the word ‘best’ is subjective to an usecase.
My usecase is just to show a list of the last 50 Contacts of a Company.
I want the option that loads the fastest and doe snot slow down the app. That why i thought looking for the 10k records to just show the last 50 is too much
Using the data api i can limit the server reponse. But i want to be sure its not the same as show items_until#
There is no advanced filter. its just a filter Company = Current User company
But maybe I should isolate this component and try to do some tests. It’s just that this RG is the one that takes the most time to load 
Hey @design.agx - it sounds like there could be a few checks to look at:
- do you have ‘load all items immediately checked’ on the RG?
- is the Repeating Group being referenced by something else?
- if you open up your network tab (ctrl+shift+i), you can view where there would be a loading bottleneck
If you share a few screenshots of your search/setup @boston85719 and I can have a race 
1 Like
Oh my bad 
I realised the RG is indeed referencing a custom state where the list of Contacts is set.
The problem was coming from this action, as it was doing it on every page load 
@DjackLowCode But besides this, do you have experience loading very big quantities of records in Bubble? You always stayed with the ‘Do a search’ ? Have you explored using the Data API instead with the parameter limit?
Just curious if what I initially was thinking made sense, or if there is no use case for it
Interested to hear your thoughts also @boston85719
@design.agx glad you found the issue XD
always use the ‘do a search’ action - there will be no upside to using the data API unless it’s an external application.
1 Like
From what I’ve experienced, bubble can handle searches pretty well and it doesn’t matter whether it’s 100 items or 100k items in the database. If the repeating group is set to only pull in/display X items it’s usually pretty fast - especially if X is kept bitesized, ie under 25.
What slows things down is the size of the actual data being downloaded along with those items. If you have a data-heavy field on the datatype bubble will pull it all in (privacy rules accounted for). Many people (including me) feel this is a flaw with bubble because not all data ‘needs’ to be pulled in on the initial search. To get around this many advocate using satalite tables for the heavier data. Keep the main searched-table as light as possible.
A light datatype along with a small number of items returned should always result in pretty fast searches. Yes it could be even faster … and on that note it was great to hear @emmanuel talk about the database work currently being implemented, which will no-doubt speed things up even further.
2 Likes
Try do a search for with sort by set on constraints area, the sort by being created date, descending set to yes, and then items until 50, if all you want is most recent 50 items. I believe that will return only most recent 50 and default RG lazy loading will allow first 12 or so to be visible pretty quickly.
1 Like
