Searches - When do they run? How do we optimise them?

[QIRO]

Hi all,

Looking at doing some optimisation in my app and I want to know if I’m wasting my time.

My application runs on one page, with elements shown and hidden as requied.

If I run a search in a Repeating Group, does the server perform all searches on page load, or are they performed when they are needed?

Can we optimise load times by only showing the RG when it is needed?

Is it more efficient to do multiple searches of the same database or one and filter?

For example:
Do a Search for Cars (that are Red)
Do a Search for Cars (that are Blue)
Do a Search for Cars (that are Yellow)

Or do one Search for Cars and multiple Filters to put the results into Red, Blue and Yellow?

The percieved performance of a website is more important than the actual performance, so as Filtering is done on the front-end, what is best practice here?

[georgecollier]

@QIRO

are they performed when they are needed?

Generally this, unless you apparently have the HeroIcons plugin being used inside the RG!

@QIRO

Do a Search for Cars (that are Red)
Do a Search for Cars (that are Blue)
Do a Search for Cars (that are Yellow)

Or do one Search for Cars and multiple Filters to put the results into Red, Blue and Yellow?

Probably the first - behind the scenes, Bubble does some magic to make things more abstracted.

There’s a general misconception (that I also had for a while) that using a hidden RG, and then referencing it with :filtered in multiple places, will result in the filtering happening client side and avoiding multiple searches. But if you look at the network logs, searches do still happen.

So, and you’ll need to test it if you want to make sure;

@QIRO

Or do one Search for Cars and multiple Filters to put the results into Red, Blue and Yellow?

This will
Search for Cars
Search for Red Cars
Search for Blue Cars
Search for Yellow Cars

as opposed to

Search for Cars
(filter everythign else on the client side)

[cmarchan]

Implying that they are charged $$$ @georgecollier ? Do you happen to know?

[georgecollier]

Yeah, here’s an example:

• repeating group var - products that shows all products in DB (4)
• repeating group with :filtered to identify products where name contains input’s value
• one search is done to load var - products on page load
• more searches are done to get the filtered list, even though we already have all possible products on the page
  

And yeah, WU is charged for that.

[georgecollier]

Oh, and before you ask - yes, changing it to an advanced filter does force it to process client side. So, if you want to use this method, you need to force usage of an advanced filter rather than just :filtered with constraints.

[cmarchan]

@georgecollier

Thanks man!

I use this approach all the time.

I wonder if $$$ charging happens when cascade-filtering involves advanced querying and thus … more server searching is needed.

May not be the case for simple filtering as all results have been loaded to the page …

[georgecollier]

@cmarchan

I wonder if $$$ charging happens when cascade-filtering involves advanced querying and thus … more server searching is needed.

So daisy-chain filtering with custom events is also optimised by Bubble. If you have List:filtered → Result of step 1:filtered → Result of step 2:filtered etc, then it will combine all of those into one query insofar as it is able to do so, rather than doing a search for each step.

[Lumyna]

Wow, I think a lot of guys use the same approach with a “Var”.

But this additional charge wasn’t supposed to happen, was it? After all, a search has already been done for the data in general, so the logic would be to use this already returned data to perform additional searches on them, as we have already paid for the initial search why we need to pay for others thar are just referencing the 1st one?

[georgecollier]

@Lumyna

as we have already paid for the initial search why we need to pay for others thar are just referencing the 1st one?

Behind the scenes (and I speculate here), Bubble substitutes values as appropriate.

The way we think about it is:

1. var - products
2. var - products:filtered

Actually, if var - products = Do a search for Products, Bubble sees:

1. Do a search for Products
2. Do a search for Products:filtered

var - products = Do a search for Products. var - products is, technically, an expression, not a list. That’s my best guess.

[Lumyna]

Interesting, thanks @georgecollier !

I’ve seen discussions about this many times on the forum, the most recent one involved using these “var” in popups, groups or floating groups, so I think a lot of people are using this approach of “prefetching” lists using RGs as “Var”.

I would like to know more details about this from other great contributors here like @Jici , @tylerboodman , @chris.williamson1996 , @boston85719 , @J805 , @dorilama , @adamhholmes , @gerbertdelangen and others, including any details provided by Bubble itself through @fede.bubble . So that the entire community can be updated on this.

[georgecollier]

Let’s not tag the entire community but yeah it’s just a myth that’s kind of become part of the mainstream knowledge.

It’s also just a logical consequence of how Bubble makes us think - when we see a RG data source, we assume we’re referencing the list itself, but in fact we’re referencing the expression.

[boston85719]

I can say to start, good on ya @Lumyna for tagging other experienced developers…the more developers testing WU charges the better, as it will lead to faster discovery of bugs that are costing us all money.

Before diving into things, to start, while testing, I’ve discovered another bug in WU charges. We are supposed to be charged 0.3 WUs for a search, 0.015 for each entry returned, and 0.000003 for each character returned. My first testing was with a data type that only the 4 built in fields, with 73 entries in the database, and checking the number of characters returned, each only returned 77 characters of data. Do note, that in the below screen shots, it represents a page with a single element, just one repeating group with a do search with no constraints.

73 entries returned

Each entry returned should have only _id (the unique id), created by (should only be the ID of user, and not inclusive of _lookup_admin_user_meta_live), created date, modified date.

As a whole, those characters returned should only be 77

I am generous in my assumptions and benefit of the doubt, so I also added characters for the type and version, even though they should not be counted. Adding those it is 96 Characters

The math, should be search at 0.3 WUs, each item returned (730.015=1.095) and characters returned (73770.000003=0.016863) or can be (7396*0.000003=0.021024) for a grand total of 1.411863 or maybe 1.416024

THE PROBLEM IS WHAT I WAS CHARGED

I know bubble rounds in these values, but it displays as 1.46 which is around 0.044 WUs more than it should be. Now, obviously this difference of 0.044 is not a large discrepancy, but it is different than anybody should realistically expect based on publicly available pricing information published by Bubble, and for anybody expecting to scale on Bubble, if you had 1,000 daily active users, all of whom perform this search once per day, that is 30,000 searches x 0.044 WUs too many, resulting in 1,319.28 WUs more than we should have been charged. Again that is not excessive, but we should only be charged for what Bubble states we are charged for.

Figuring out why the differences displayed in the WU metrics tab should not be so difficult, as things should just line up properly as NUMBERS DON’T LIE.

@fede.bubble @Eram_BubbleSupport could be get an investigation on this subject or at least an explanation that would help explain why there is an additional 0.044 WUs charged?

Now, onto the test for the post.

Below are images of WU metrics based on a single repeating group on the page searching for 73 items with no custom fields

The result from the logs and the Network tab are as expected, one repeating group performing one search is charged for one search.

Below are images from logs and network tab for the same page now, with 2 repeating groups, one that does the search and the second that references the first repeating group

What I found is that when I have two repeating groups, one with search and the second referencing the first, I get the results as to be expected, only one search of the database was performed and I was charged WUs associated with performing just one search

NOW ADDING IN SOME FUN TO REALLY TEST IT OUT. I added to that data type a single field, that is related to an option set called color. I applied a random color to each of the 73 items in the DB. One the page I added another repeating group for each of the 5 colors with a filtered expression. Total number of RG on page is 7, the first that does the search, the second that just references the first, and the other 5 are referencing the first (ie: the one that does the search) with filters for the colors

In the network tab, I still only see one search and one msearch, which implies Bubble is doing what has always been expected, which is a single search of the DB, but now I see 6 maggregate searches…these are the first that takes place when do a search for DB, and the other 5 are from the reference plus filter

do keep in mind that an maggregate is the same type of charge as if we reference a rg and its count, which is 0.20 WUs, which is less than an actual search. Below is what I see in the logs, and one strange thing is that there are 2 searches, and when I click to dig deeper it brings me to the same single search on first RG, which is very strange since when I tested with just two RGs and one referenced the first, I was only charged 1 search there, so no idea why I am charged for 2 searches in this scenario as it doesn’t show as if there is another search on any other RG taking place. What is also weird is that the charge for 2 searches is just 1.78 WUs, which is not the equivalent of (1.46X2=2.92), nor is it what I would expect for the cost of the additional characters returned for the color, so pretty odd to see the original one search charged as 1.46 WUs and this other apparently 2 searches charged as 1.78WUs, unless what it is is just the 0.3WUs for a search plus the characters for the colors which adds up and could make sense, but still odd that I wouldn’t have been charged for the 73 results that should have been returned from the 2nd search

So, overall on the search with multiple RGs referencing the first and filtering, the charges on first test were below at 2.98 WUs

I ran the same thing after seeing the odd 2 Searches, and got the same results.

So, as a baseline, for having 7 RGs, with 1 referencing original and no filters, and 5 referencing the original with filters, have total WUs of 2.98 charged.

When I alter the approach and do searches on each RG and use constraints

I see very similar results in the network tab, in that there is only 1 search and 6 aggregate searches. Below are screenshots from the logs

What is really strange is that now the 2 searches do show different searches and one is for the original search and another is for the search with constraint, but oddly, there are 5 rgs with searches with constraints but WU metrics in logs only show one

Based on my simple test, it seems like the refernce with filter operator is cheaper by 0.01 WUs, at 2.98 versus 2.99

When I alter the constraint to be an advanced filter, I get more msearch in network tab

In logs see very strange results as things just don’t match up with previous results. Most important being the charge for the 1 search from the original search with no constraints…it only cost 1.15WUs the last time

Final test had total cost of 2.28 WUs, which is lower than other tests at 2.98 or 2.99 WUs from other tests.

So, seems like best approach is to have one RG do the search, and other RGs reference the first and have them use an advanced filter to force a client side filtering.

[QIRO]

Wow. Now that’s a comprehensive analysis, I may need to spend some time going over it so I understand everything you’ve demonstrated.

I understand there’s some additional clarity required on WU calculations, hopefully this can help users beyond just me make better use of their WUs.

[georgecollier]

@boston85719

I know bubble rounds in these values, but it displays as 1.46 which is around 0.044 WUs more than it should be. Now, obviously this difference of 0.044 is not a large discrepancy,

Check the number of characters in the response (including field names/IDs), and the number of characters in the entire response body.

@boston85719

What I found is that when I have two repeating groups, one with search and the second referencing the first, I get the results as to be expected, only one search of the database was performed and I was charged WUs associated with performing just one search

Yes, because you effectively have two repeating groups with identical search expressions, which only need to be retrieved once.

If you have 70 results, it’s likely not all are actually downloaded on the page. Bubble tends to download 20 at a time. Try on a smaller data set (e.g 5) which ensures they’ll all be fetched, and you’ll probably see different results which I’d be interested in seeing.

Also, keep in mind your maggregate searches may be coming from the debugger which gets counts when you inspect an RG.

[tylerboodman]

​ ​ ​ ​ ​ ​ ​ ​ ​ ​

[boston85719]

@georgecollier

Check the number of characters in the response (including field names/IDs), and the number of characters in the entire response body.

So are you saying I should be using the character count tool with a value set like below?

@georgecollier

Yes, because you effectively have two repeating groups with identical search expressions, which only need to be retrieved once.

Yes, this is expected and has been a long known way that Bubble operates with the same search on the same page in multiple elements. I have to do this for SEO to get Structured Data to show correct entry when not using a current page thing.

@georgecollier

If you have 70 results, it’s likely not all are actually downloaded on the page. Bubble tends to download 20 at a time.

Show all items immediately boxed was checked to ensure all got downloaded, the RG has no set number of rows and fixed height of 1, plus the network tab showed 73 items returned.

@georgecollier

Try on a smaller data set (e.g 5) which ensures they’ll all be fetched, and you’ll probably see different results which I’d be interested in seeing.

Test on a data set of 73 with same set up as me, I’d be interested to see if you get different results from my test

@georgecollier

Also, keep in mind your maggregate searches may be coming from the debugger which gets counts when you inspect an RG.

Only if you inspect using the debugger, which during the test I did not.

@QIRO

Wow. Now that’s a comprehensive analysis, I may need to spend some time going over it so I understand everything you’ve demonstrated.

This is not complicated. I personally had never used the network tab in developer console prior to this test. Just open the developer tools, click network and then refresh the page, then look at the same type of value sets like maggretate, search etc.

@QIRO

I understand there’s some additional clarity required on WU calculations, hopefully this can help users beyond just me make better use of their WUs.

Yeah, Bubble is not doing a good job at publicly disseminating all that goes into WU calculations, especially for how to understand number of characters returned, or just generally with expressing all that is charged.

For example, an issue I raised in another thread that still no other developer has tested for when we use the recurring backend, when we set the action there are additional charges that Bubble does not address publicly. And the fact that the way they charge for WUs has effectively made some valuable functions obsolete, like recurring events, as why would anybody pay the extra costs of it when they can just schedule backend workflows?

I’ve discussed this behavior with our engineering team and they informed me that there is a database read and write that occurs when setting a recurring event.

To provide some context, recurring events have limits on the number of recurring events per thing in the database. More on this here: Recurring event | Bubble Docs

When setting a recurring event on a thing, such as the Current User, Bubble will first read the thing to see how many recurring events have already been set and then store information about the new recurring event on the relevant thing.

It is also pretty difficult to get support to provide accurate information when communicating through email, as a major part of my issue with the recurring action was support at first stated the charges of conditional evaluation of 0.63 WUs on an action that had no conditions was expected as it is a charge to determine if a condition exists or not, and not actually evaluating the condition, since there was no condition. When pushing back on this, they ignored it in their response, so then having to follow up again, I finally got this response below.

Thank you again for your patience!

I’m reaching out to let you know that I’ve escalated this behavior to our engineering team for further investigation. I’ll be sure to reach out as soon as I hear back from the team!

It is a shame that so many new users would just support word as they are expected to have the answers, but on so many issues I’ve raised with WU charges, it seems like support is as in the dark as everybody else is. It is also a shame that not more developers are testing things and holding Bubble to a higher standard when discrepancies are found, or calculations are confusing. An example of this is the issue of character count returned. I don’t see Bubble doing anything to show a simple example to demonstrate ALL of the characters that are being counted, so that when somebody like me does a simple test, we can verify the results as correct and expected easily.

@georgecollier

the number of characters in the entire response body.

Where in the network tab can we see that?