I’m building an app that displays user-generated content and now have over 10,000 entries in one data type.
Searches and filters are becoming slow. Is there any smart way to use privacy rules, backend workflows, or custom data structures to speed things up?
Hey @RickeyJ.Wren
For optimization, don’t use filters. Just use constraints. If you can’t use constraints, that means you need to adjust your database structure.
If you use filters, it pulls everything onto the page, then uses the filters.
Does that help? 
Bubble can handle millions of records just fine.
Aside from using Algolia, it’s probably worth restructuring the database like Jason said.
Also, as @nocodeventure mentioned Algolia makes things fast, correct me if I’m wrong, but only if all the data is supposed to be public. At least the way Bubble has it natively. As an alternative, there is also TypeSense that you can setup on your own that is an alternative to Algolia but can be setup privately.
I still think you should be able to do without it though. I have seen Bubble handle databases in the millions without an issue. Just needs to be set up properly.
Security in Bubble is pretty simple, as it all just boils down to Privacy Rules. One thing for speed of retrieval that may work with Privacy Rules is to set the Privacy Rule to say can not find in searches if the user is not supposed to see the data, and I believe that will help speed up the server processing of all data entries as I think those that have privacy rule of not find in search will not need to be ‘checked’ by the server and will not be returned to the client device (this has time associated with how much data is sent back).
You shouldn’t be using backend workflows to return data to the device if all you are doing is searching and filtering. If you have backend workflows involved, find the right way to not use them.
Using a Hybrid Data Structuring approach that my plugin Data Jedi helps make simple to implement, reduces the workload unit cost of retrieval and speeds up the content download time as it reduces the amount of content to return (it allows for a data structure and use of api objects that return less characters of data that can speed things up by 30-50% but keep in mind it is milliseconds). In some use cases it is very beneficial, and when dealing with large amounts of data that needs to be returned it is most useful, but if you are restricting heavily the amount of data accessible by Privacy Rules, it might not be as helpful.
One other thing is to consider how much data is loaded when page is loaded and whether you can load the rest when page is already loaded and some content is visible. For most apps where it is not expected that a user will load page and immediately start scrolling as fast as possible to see all things in a list, loading a smaller set of items first, and then loading rest after page is fully loaded can help speed things up. Additionally, make sure the checkbox on the RG to show all items immediately is not checked as when that is checked bubble will show no items until all are loaded.
