Best way to preload data?

Hi, all. This might have been answered, but I can’t find a clear answer.

If I have a list of things I want to perform many operations (filtering, sorting…), instead of making a search every time, I’d rather make it once and store it somewhere as a reference for all the further operations.

What’s the best way to store it? Custom States? Groups and Repeating Groups as holders?

I once read that Custom States are only a reference to the database. Also, the option “:make static” doesn’t make it offline, it just stops the real-time sync with the database.

I’ve use custom state lists as in list of Users or list of Orders on the page. You can load a lot of items into a list.

I’ve found that assigning types to a group is more trouble than its worth (more difficult to intuit the whats going on and to troubleshoot, less control, less flexibility). I believe they essentially do the same thing, just worse IMHO.

Hi @julienallard1

I have a structure similar to what you’ve mentioned, I use filters and rankings in almost every module, I got the best results with RGs.

holders

Best

1 Like

I got the best results with RGs

meaning?

I can’t argue about how valid this test can be, but when I edit it as a custom state instead of RG, the data comes pretty late.

pageisloaded

This has been my experience as well. In my case I need to merge several searches into one. However dealing with custom states slows things down noticeably.

I am now looking at serving a default/on page load search result. Then once users start filtering, use conditional logic to change the Data Source to the custom states. About to start this work so I’ll update when done.

Ok, so I re-factored my search page. Due to the complexity of my searches and filtering I need to employ states to store various steps in the search sequence. If I could handle everything without custom variables I would.

That being said… My general approach was to think about the DB heavy queries and minimize their frequency. Not having a computer science background I can’t definitively say this, but my hunch is that filtering is a lot lighter/faster than a DB query.

In my case, I need to do a large search for a set of objects. There is no way of avoiding it. So on page load, I load everything into a custom variable list. That’s my one heavy DB call. From there on in, I leave it untouched to prevent additional DB queries and move this variable to other variables with various filtering steps in between. Then merging searches as needed.

I’m guessing my page is 50% faster now. It was noticeably slow before. I’d say that now it’s OK.

Thanks @eren for replying. I think that for now I’ll avoid custom states.

@jon2, do you notice speed differences on your side?

@liran, thanks for sharing. I’d say that whenever possible, applying constraints on the query itself is best practice as it limits the amount of data to load. But in the case where you need to make different filterings on the same list, then yes, it might be better to make 1 big query than several smaller ones.

@keith, could you give us your thought on the matter?
Thanks!

It was recently suggested to me to use a repeating group that is “hidden”, so maybe stored in a popup or something, that will “house” the dataset that I am wanting to use as the basis for the datasource on multiple repeating groups on the page.

For the purposes of responsive design, my page has 14 different repeating groups ( this is because I have list and grid views, plus the ability to close the side menu and expand full page ) I haven’t noticed the page being slower because of the number of R.G.s on page but would like the initial page load to be quicker.

I experienced at first try with a custom state to “house” the list from a search on page load, that the data list never loaded as it was so large ( over 1700 items ).

Now what I am realizing is that for my filters and sorting to be easy to implement I need one R.G. that is “hidden” and have all other R.G.s that will be visible at some point use the “hidden” R.G. as their datasource. My hope is that also speeds up things instead of slows them down, but will see.

I think there are so many ways to set it up, that eventually you may find one “wins out” and has the seemingly faster load speed.

Totally agree on that. I made some performance testing on RG groups with preloaded data and with db direct query (search for) and the more data you load the bigger the difference :joy:

Some test results below:
(I measured time from changing the list of items from 10 items list to xxx items list and stopped when page rendered any result - so from user perspective something “finally” happened)

Repeating group row includes 3 text fields with Company name, number and person name & surname - so very simple.

Query type Items Test 1 Test 2 Test 3 Average Difference
Preloaded 100 2,55 2,88 2,88 2,77
DB 100 5,52 5,37 5,50 5,46 50,70%
Preloaded 200 5,52 5,37 5,50 5,46
DB 200 9,52 9,79 9,62 9,64 56,65%
Preloaded 300 8,40 8,34 8,45 8,40
DB 300 13,34 12,33 13,11 12,93 64,96%
Preloaded 400 11,23 11,18 11,43 11,28
DB 400 16,10 16,33 16,56 16,33 69,08%

image
image
(seems like each 100 items adds ~ 3 secs)

So my config is → Popup with 4 repeating groups currently (but there will be more) that do a search for full list of data (no filtering so there is still a lot of space here for optimization) and then I set a data source for any other user facing repeating group as those repeating groups.

I also tested the performance of RG page changing and here maybe the performance difference wasn’t as significant as with displaying all items at once but pre-loaded setup felt a bit more stable (less page data loads) when I changed pages really fast (it’s was a bit geeky test because nobody will change pages that fast but nevertheless :P)

Overall my users will not display more than 50 items per page so speeds are even better :smile:

Hope this will bring some more insights to our community or maybe someone can drop here even more RG performance tips.

Cheers!

6 Likes