Showing 10k rows in repeating group

Need help Database
1

Hello Bubblers,

I’d like to ask for advice on how to efficiently load many rows in a repeating group, as I’m currently facing significant lag.

I need to display all the tasks in the system for the app’s admin. At the moment, I have around 11k tasks that need to be shown, filtered, and searched depending on what the admin selects.

Currently, I’m only displaying 10 rows per page and allowing navigation between pages using “Next” and “Previous” buttons with the workflow actions Show Next / Show Previous for that repeating group.

I also tried setting a state to show items up to a certain number and then increasing that number whenever the user clicks “Next.” But in that case, my search gets restricted to only those 10 records.

2

Can you upload some screenshots about your repeating group and settings? The best way is that, repeating group with pages o infinite scroll, but send screenshot for I can help more you

3

Thanks for your reply. I’m sharing the settings of my tasks RG, which is displaying all the tasks currently available in the app. At the moment, I have 11k tasks, and the app becomes really laggy when I open the view where this RG is visible.

image
image908×659 73.3 KB

4

Do any of the 8 conditionals on the RG change the data source?

5
  1. Make sure you’re not storing images as base64 in rich text descriptions on the Task data type
  2. If any of your conditional data sources use advanced filters, that requires downloading ALL 11,000 records to filter over, which is of course, slow.
  3. Use ‘ignore empty constraints’ to ignore empty filters instead of advanced filters and conditional expressions, as 8 conditional expressions for searches is not maintainable.
6

Yes, it does. All 8 conditionals are working as filters whenever the state changes. For example, the project is set to a state, as shown in the picture below:

image
image623×497 52.4 KB

7

That is one reason for the issues you are experiencing.

From the screen shot, it looks like every single conditional is for a constraint data source ‘is not empty’. Instead of that, put all constraints on one search and check the little box at bottom of ‘ignore empty constraints’…that will at the very least make it not move through so many different searches and could help with the speed.

Make a custom state that is of type number. For the search use the operator ‘items until #’ and for the value use the custom state. This way when user presses ‘next’ it doesn’t move to a new page, it just adds to the existing list more rows.

If you really need all 11,000 items, consider checking out Data Jedi plugin. It will reduce the WU costs of the initial search and retrieval by 95% or more. It also has a paginator element that makes pagination simple and more flexible with event triggers. There are the two search and select elements, a single ‘search box’ and multiselect ‘multidropdown’. These are way more powerful than the built in bubble ones as you can use all 4 data types in bubble, not just custom data types, and can filter across multiple fields, not just one. And they can load and filter across thousands of records at once, and is really fast, especially if using the api objects for a hybrid data structuring approach that Data Jedi makes simple to implement and manage.

8

Thanks for your answer, it helped quite a lot.
But just to be sure - it’s still not a good setup to have the Search & Autocorrect element go through 11k tasks to check if a particular user completed their task? In other words, search without Data Jedi plugin in this kind of situation isn’t really possible at all, since it needs to process all 11k tasks and causes the app to freeze?

image
image706×799 90.6 KB

9

You’ll still have to return 11,000 data records with this Data Jedi idea, it’ll just be in a different / more WU efficient (but not necessarily a performant/good practice way). Search & autocorrect (like Data Jedi) runs client side on all of the possible results, so neither plug-in will let you only return the data you need.

Use a native search with an ‘Any field contains’ constraint! It’ll be fine for most use cases!

10

Thanks for the search advice - it works great since it’s running on the server side. Do you maybe have an idea of how to display 11,000 records for a user while avoiding lag?

11

You don’t display 11,000 records at once. You display them in pages. No user is going to scan 11,000 records!

12

Thank you George! Found a setup which works.

13

I am not sure which search & autocorrect element you are using, but in my experience they do causes some issues, especially with larger lists of data.

In terms of not being possible without Data Jedi, no, it is, my rationale for suggestion Data Jedi was under the condition of ‘need all 11,000 items’. If somebody needs 11,000 items, the Data Jedi plugin and the Hybrid Data Structuring approach reduces the WU costs of data fetch by 95%. If you do not need to have all 11,000 items on your page, you can just do a normal search of the DB with constraints to return only the number of things that you need (ie: those that match your search query constraints).

Basically the point of using the Data Jedi Plugin would be to return 11,000 items for 95% less WUs.

What kind of best practices are you thinking of?

Data Jedi is not a search & autocorrect plugin. It does have two elements for searching with typing, a searchbox and a multidropdown select element, but it is not a search & autocorrect plugin. I’m a bit confused on what the reason you believe it not possible to structure the data so that all of the possible results are the only data needed.

14

Yeah but nobody needs 11,000 item downloaded onto one page, and if they do, it’s likely indicative of one or more of bad UX, bad DB design, or mis-using server-side constraints.

15

No, but if somebody says they do, than they do. Different use cases exist. And if somebody needs to return 100 items 110 times, Data Jedi makes it 95% less WUs.

But you could rather than just throw shade on it, be helpful and let me know what best practices you think its usage goes against.

16
  • returning more data than is necessary to the client from the server (data usage, speed, security)
  • filtering heavily client side
  • reducing dependency on third parties
  • reducing usage of maintained native Bubble features which support things like privacy rules, native actions etc
17

Everything that Data Jedi does is the same as native bubble features with support for things like privacy rules and native features. I’ve explained that and demonstrated that to you in multiple ways. In fact, Hybrid Data Structuring can lead to greater level of security as you can have the same data as different objects leaving some fields off and stored in a data type only accessible to certain users.

Yes, as more people, services, or tools are added that are from different suppliers, things start to get a little bit harder to ensure everything works smoothly. In regards to Plugins, they are services, like any other service, so be it PDF, data backups or getting more features/functions that do not exist natively in Bubble, we need to pay attention to the service provider behind the plugin. Not all will stay around Bubble. Some service providers may leave Bubble for a vibe coding platform or something like that.

Sure, if doing complex filters it could create some kind of strain on browser resource during that process, but in my experience using the plugin and the hybrid data structuring approach, client side filtering on data sets is faster than using Bubble server once the data is already on the client device. So it is more about being capable to know when to filter client side or not.

This maintains a lack of understanding of how a Hybrid Data Structure can allow a developer competent in data structuring can have more options to ensure security, speed and data usage.

First of all, usage in your mind I am going to assume equates to WUs, and if we consider that a search of 1,000 things can be done for 95% less ‘usage’ to obtain the same amount of data, than the Hybrid Data Structure approach would be far better in terms of usage to implement and below is a little math example.

Let’s say database has 1,000 things, and the search constraints are such that we expect only to return 100 of those items. This is at a cost of 0.015 ‘usage/WUs’ per item returned, so a total of 1.5 ‘usage/WUs’ to fetch those 100 items ONE TIME.

Now, with a hybrid data structure approach we can instead return all 1,000 of those items, for only 0.015, so we can return all 1,000 items 100 times for the same exact usage/WUs as returning only 100 items Once.

For speed, the time for the data to be downloaded to the client device if using custom data types to return 1,000 out of 1,000 is longer than it is for the download from the server to the client device if returning all 1,000 objects as API Objects. This has been tested, verified, and posted for everybody to see. You can not lie about numbers. But yes, there may be a few milliseconds of savings if only returning 100 items.

In regards to the security, as mentioned, you can make your API Objects more secure by removing certain fields. Plus the same principles apply, use Privacy Rules to secure your data.

At the end of the day, the only thing in the list that is valid, is the reduction of dependency on third parties. But, if a tool provides somebody the functionality they need, that is not available natively, than we need to use a third party. That third party can be Stripe for payments, send grid for emails, or plugins that take advantage of Bubble’s best feature - it’s extendability. The Plugin marketplace and the thousands of available plugins are part of that extendability feature.

I applaud the effort though.