How to Client Side filter a list?

[elkadeo]

I read Petter’s great book on Bubble app performance. My understanding is that it should be possible to perform filters on a list on the Client side of the app and thereby save WU consumption (since all the processing is done on device). I’ve spent the day trying to figure out how to achieve this and am still seeing :filter expressions showing up in the App Metrics indicating that those expressions are consuming WU. What am I missing?

Here’s a simple scenario:

I have a repeating group (A) that populates a list of records by doing a search in the database.
Elsewhere in my app I have another repeating group (B) that takes the list from (A) and applies a filter to it to present a new filtered list.

The filter is very simple, and just filters the records in (A) against fields of data they already possess. It doesn’t take any extra steps that may require looking up additional data from the server (like connections to associated records etc)

Shouldn’t a filter that simple take place entirely client side? And thereby not consume any workload? All of these filtering operations continue to display WU consumption and I’d love to understand what I’m missing.

[boston85719]

@elkadeo

My understanding is that it should be possible to perform filters on a list on the Client side of the app and thereby save WU consumption (since all the processing is done on device).

From my understanding WU consumption related to searches has a charge based on the number of characters returned from the server, so if your search is returning all entries in the DB because you did not place any constraints on the main search and instead are using the :filtered operator for client side filtering, you will be likely to be charged more WUs than if you simply did the constraints via the search so the filters are applied server side. This is because if constraints are on the search and performed server side, the number of characters returned to device will be likely less than if there were no constraints and all data entries were returned.

The amount of processing based on search and WU consumption, as I understand it, is related to the amount of data returned as mentioned, plus the trigger and execution of the search, but does not include the amount of processing required for the server to apply filter constraints to narrow the search to fewer results.

@elkadeo

All of these filtering operations continue to display WU consumption

Please share screen shots…but main question is, are those part of the Search itself or are you drilling down and seeing a wholly separate charge of WUs for the :filtered operator. If you are seeing a wholly separate charge for use of :filtered operator, you may want to reach out to Bubble support so they could fix what seems like a bug causing apps to be overcharged for WUs.

BTW, there have been other threads recently discussing implications of the :filtered operator and whether or not the constraints are applied client side or server side. I reached out to Bubble support requesting a definitive answer from engineering on the subject.

[elkadeo]

Thanks for your reply @boston85719

When I drill down in the Metrics page the WUs are bing charged to the expression doing the filtering. It’s not a separate charge specifically to the :filtered operator.

The way I’ve been imagining this working is that once you perform a search to populate the list of a Repeating Group, (that initial search should expectedly cost WUs to populate) but once that list loads, shouldn’t any amount of filtering other Repeating Groups on the page do to the list of records from RGA be essentially free? Since simple filtering on that list shouldn’t require any more data from the server?

[elkadeo]

Am noticing in other threads that only Advanced Filtering is performing client side. So moving the filter into the advanced option seems to make a difference. Heres a screenshot of two duplicated Repeating Groups that are performing the same simple filter. One filtering List A with the :filtered operator directly after. And the other also filtering the same list, but moving the filter into the Advanced area.

Moving the filtering into the advanced section is about 1/3rd the WUs than filtering directly with the :filtered operator.

[boston85719]

@elkadeo

shouldn’t any amount of filtering other Repeating Groups on the page do to the list of records from RGA be essentially free?

It should be, and based on what I recall from Bubble list of WU consumption costs, it should be, so if you are not experiencing that you should reach out to Bubble support to inform them of the situation and ask them to clarify WU consumption costs for client side activities.

@elkadeo

Since simple filtering on that list shouldn’t require any more data from the server?

That is the case, when the data is already on page, Bubble is not supposed to be requesting data from the server again.

@elkadeo

Am noticing in other threads that only Advanced Filtering is performing client side

This is why I reached out to support to get engineering to provide a definitive answer on the topic, as in the past, it had always been purported to be that any filter constraints provided via the :filtered operator are happening client side, not server side, and it seems like there is confusion among the community as to whether or not that is the case.

@elkadeo

Moving the filtering into the advanced section is about 1/3rd the WUs than filtering directly with the :filtered operator.

This is reminiscent of the discussion around differences between using a count operator and different ways to perform same function with different expressions, all of which were to get the count…users showed huge differences in WU consumption.

As developers we should not be tasked with guessing and testing which approach is cheaper in terms of WU consumption and whether or not we should be :filtered operator constraints onto an Advanced filter to save WU consumption.

I believe, if constraints on the :filtered operator are happening server side, it may be a bug or some new change to intended/expected behavior that was not well communicated. Additionally, it seems like a bug that searches with constraints filtered server side are getting charged a higher rate of WU consumption than those with client side filtering since naturally it is expected the number of characters returned when filters are applied server side would be less than those with client side filtering, plus the expectation that the number of constraints we put onto a search to be performed server side is meant to speed up the search and return fewer results which would make apps more performant compared to those with client side filtering.

I’m wondering if all these “issues” with WU consumption is causing unnecessarily bad development practices and less performant apps to proliferate on the platform.

Below is from the manual

One thing I don’t know what is an aggregate query to the database, and I don’t see a reference to what it is in the manual

There is nothing in the manual within the section linked referring to the number of constraints applied to a search as contributing to WU consumption, and not indication that the use of client side filtering contributes to WU consumption, plus the ongoing confusion over whether or not All filter constraints applied via a :filtered operator are happening client side or not.

But, in my opinion you should not be seeing the :filtered operator contributing less in terms of WUs because of the way searches are supposed to be charged based on the Bubble manual.

[georgecollier]

@elkadeo

once that list loads, shouldn’t any amount of filtering other Repeating Groups on the page do to the list of records from RGA be essentially free? Since simple filtering on that list shouldn’t require any more data from the server?

In theory. However, I’ve found this to not be true. I contacted Bubble and they have been waiting for an answer from engineering for two months and still no answer other than we’ve flagged it despite continuously pushing it.

@boston85719

But, in my opinion you should not be seeing the :filtered operator contributing less in terms of WUs

Filtered is definitely server side where Bubble sees it possible. Petter Amlie is the source and he’s the one helping Bubble understand the system they created

[boston85719]

@georgecollier

Filtered is definitely server side where Bubble sees it possible.

Is there any indication on what the ‘where Bubble sees it possible’ means in terms of a list that could be published for people to know when it is and when it is not?

@georgecollier

I contacted Bubble and they have been waiting for an answer from engineering for two months and still no answer other than we’ve flagged it despite continuously pushing it.

Was this through a bug report or an email? I’ve found bug reports are a bit easier to follow up on when they ‘get lost in the shuffle’ and go unanswered for months at a time. It seems like something that should be addressed as it feels like that would be Bubble overcharging users and not charging as per their public documentation on how things are charged.

[georgecollier]

@boston85719

s there any indication on what the ‘where Bubble sees it possible’ means in terms of a list that could be published for people to know when it is and when it is not?

Basic search constraint like Field = X. Anything that you could put in a Do a search for, presumably. Of course, we have to negotiate the Bubble black box

@boston85719

t seems like something that should be addressed as it feels like that would be Bubble overcharging users and not charging as per their public documentation on how things are charged.

My thoughts exactly, I followed up again today saying this level of wait is unacceptable but who knows. I initially asked the question by email but it seems to have been treated like any other bug report.

[boston85719]

@georgecollier

I followed up again today saying this level of wait is unacceptable but who knows.

You may look at the website to see the list of Product Managers and attempt to reach out to them directly. It feels like as the Bubble team grows, including that of support, there has been too much liberty to make decisions in the hands of support, so certain items may not be making it to the attention of the right people.

[boston85719]

This is a response from Bubble support as an answer from engineering as to how it works

At its core, “:filtered” is just an operation to add new constraints on top of a list. Before ever executing a query, we simplify it as much as we can, and therefore, :filtered will either become “just a new set of constraints on top of the parent search” (very fast) or be a manual filter on top of each of the rows of the parent list (potentially very slow).

Here’s a quick guideline:

1. If :filtered contains an advanced filter, filtering will be done “row-by-row”
2. If :filtered contains one or more constraints, and the previous part can be condensed into a simple search with constraints, then we consolidate the constraints and do it as a simple database query
3. If :filtered contains one or more constraints and the previous part is already too complicated to do as a simple database query, then we will do the filtering row-by-row.

In terms of references to client or server actions, the actual distinction is technically closer to “Bubble-logic” or “database queries.” Database queries are typically fast and efficient, but when it’s not possible, Bubble (usually on the client but sometimes on the server depending) will be forced to pull data directly from the database and perform filtering and such row-by-row as mentioned, which is why it tends to be slower.

We’ve of course worked over time to optimize both of these processes, and will certainly continue to do so, but in general with the filtered operator, whether or not it involves server actions depends on whether it can “compile” a database constraint; what makes :filtered compilable or not depends on whether its “parent” search is database-compatible.

• Something that starts with a non-db-compatible list is almost never going to end up db-compatible (ex. list custom states, API call results, list fields)
• Something that starts with a db-compatible list is very likely to stay db-compatible as long as you don’t use list operators (ex. :each item’s Thing) or advanced filters.

In short, usually if you don’t operate on already-loaded data, the process will need to iterate through whole database tables, making it slower.

[elkadeo]

Thanks very much for sharing this response @boston85719
That’s very helpful.