Bubble DB Challenge - Searching By Tags



Screen Shot 2024-07-30 at 8.37.18 PM

Data type has a field that is text, and you store the list of tags as a single text, separated by comma.

On page you have a custom state set to text and is a list

On RG, use the search with constraint contains keywords

I don’t have a bunch of test data to tell you that it is performant on 100k items, but it meets all other criteria I believe

2 Likes

Edit 2: This causes way more searches than is necessary on Bubble’s end. Don’t use this method. Keeping here for posterity!

Okay, here goes.

This returns a list of unique IDs of Contents that contain at least one tag. Getting the content themselves is then just a case of Do a search for content where unique ID is in that list.

So, what’s going on here?

  1. For each tag, :format as text
  2. Inside format as text, Do a search for Content (Tags contains This Tags):each item’s unique ID. This returns a list of unique IDs of Contents that contain this Tag.
  3. Split the results, and get the unique elements, and this will return a list of unique IDs for all Contents that contain at least one selected tag.

If you take a step back, you’ll notice this is a bit like doing Search for Content where Tag contains Tag 1:merged with Search for Content where Tag contains Tag 2:merged with etc etc.

In terms of performance (and probably WU), my approach is roughly equivalent merged searches. However, the benefit of using :format as text is that you can have unlimited searches, and don’t need to hard code a merge expression for each possible tag.

For this method, Total number of searches = number of tags + 1. That will mean that it will slow down to some extent if users have loads of tags, but it’s easily maintainable, and allows infinite tags.

:saluting_face:

Edit: something’s causing more searches to be done than is necessary here. Will circle back…

Edit 2: This causes way more searches than is necessary on Bubble’s end. Don’t use this method.

1 Like

My 2 cents here…
From what I saw, similar to the other solutions…

Editor

I think the challenge is here :point_up_2:

1 Like

I’d do this but use the UIDs instead. Single text field containing each tags’ Unique ID. Shouldn’t confuse the “contains keyword” operator. Closest to regex.

2 Likes

I can’t think of a way without the intersects operator from an advanced filter

2 Likes

That is what I used in my example…
But I don’t know how it is going to performe in a 100k+ filtering :sweat_smile:

Here is another way to do ‘contains at least 1 tag’ without advanced filters:

CleanShot 2024-07-30 at 23.38.47@2x
CleanShot 2024-07-30 at 23.39.01@2x

For each Tag in Content’s List of Tags, have a separate thing Content Tag. Keep in sync using trigger to create/delete Content Tags as necessary.

  1. Get Content Tags where the Tag is in the list of selected tags
  2. Get each Content Tag’s Content (:unique elements could well be redundant here).

Nice due to its simplicity. I haven’t tested for WU - it may be the case that storing Content’s unique ID as a text on Content Tag and then Do a search for Content (unique ID is in xxxxx) is more WU efficient, but I’d go with what I did for maintainability.

The only additional WU costs are maintaining the join table of Content Tags, the costs of which will vary depending on your use case. They’ll likely be minimal compared to the WU saved and performance improvement you can gain by implementing something efficient.

All the big guns have come out for this one :rofl:

2 Likes

The issue there could be the # of contents in tag. This could reach maximum allowed by Bubble (10 000). This will not happen from content to tag but could happen from tag to content.

I don’t think any of this method runs into the 10,000 limit. The only list is the List of Tags on the Content which is the normal approach for any blog. Lists in searches (I believe) have a much higher limit, of 100,000 when unsorted. That is to say that List of Content Tags:each item’s Content can be up to 100,000 long (again, tentatively, because I haven’t seen any evidence otherwise yet)

The issue is not fetching the tags each items content. But I think that # of content will grow over time and the maximum number of “content” field in tags DB could be reached. If the # of content doesn’t grow, I don’t see any issue with this method.

1 Like

That’s the same for any search, no? If we take the simple single tag search:

Do a search for Content where Tags contains X will encounter exactly the same limits as Do a search for Content Tags (Tag is X):each item’s Content right?

Maybe I don’t understand correctly your idea.
But the problem is before this step.
When you create a new content, you “make change to a list” that is the list of tags selected for this new content.
In this action, you will do content add “step 1 created content”. And this step could fail someday and content will stop to be added to the content field in tag because you reach the maximum allowed item in reference field. No?

No, we have three data types:

  • Content (contains content and List of Tags)
  • Tag
  • Content Tag (contains Content and Tag)

For every Tag in Content’s List of Tags, there is a corresponding Content Tag thing. If I add a tag to Content, I must Create a new Content Tag with the corresponding Content and Tag. Similarly, if I remove a Tag from Content’s List of Tags, I need to delete the relevant Content Tag.

Content’s List of Tags is 100% fine as it’s highly unlikely there’d ever be more than a few dozen tags on a single Content :slight_smile:

1 Like

haaa , I understand. Thanks

I currently use Advanced Filtering, I haven’t found a good alternative to what I’m doing (specifically comparing a list of ingredients on a recipe against what the user has marked they have in their inventory) but I love some of the suggestions in this thread.

For @adamhholmes actual need, here’s my approach:
Use-case 1:
Filtered: Advanced: (This Content’s Tags [intersect with] Selected Tags): Count > 0

Use-case 2:
Filtered: Advanced: (This Content’s Tags [intersect with] Selected Tags): Count = This Content’s Tags Count

I have yet to test this to true scale as I only have around 120 recipes that are searchable…and even as that grows, I can’t see it growing to more than 1000 or so.

2 Likes

i use for 1. just a text field which contains concatenated all tags as text (comma-separated)

1 Like

The assignment is simply not possible :fearful: All these solutions are pretty good until they need to satisfy the first requirement:

  1. Be truly scalable.

I know this is probably the last thing ya’ll want to hear - because you’ve heard me say it… but Bubble’s database is designed for flexibility, not scalable search.

Hate to be a wet towel but there’s really no good way around it. Even if we could magically rewind to the days before WUs, running this class of search in Bubble’s native db is going to take a long time / lock up your browser.

Some (all?) of you know I actually did create something that satisfies these requirements. It’s Omnisearch, which leverages Algolia/Typesense as the search engine in parallel to keeping the records in Bubble (requirement 6). In that linked e-commerce example, we’re only searching 10k records but this solution scales to millions. It works because Algolia/Typesense are designed specifically for this problem (and it does it better than sql databases). So that demo is:

  1. :heavy_check_mark: Truly scalable.
  2. :heavy_check_mark: Performant. Takes less than a second.
  3. :heavy_check_mark: WU efficient. The WU cost to return n excellent matches from m number of records is only the cost to fetch n matches from you Bubble DB.
  4. :heavy_check_mark: Allows selecting any number of Tags to search. Open the linked example and you can pick as many brands as you want in the left-hand filter menu (each brand is stored in our Bubble DB as a tag).
  5. :heavy_check_mark: Displays the total number of results. Of course! Our plugin demo shows this.
  6. :heavy_check_mark: Uses the Bubble database. This is how Omnisearch works by default.

But as a bonus, you can now also do full text search - which Bubble could never practically do on 3k+ records.

My friends, I spent a year solving this problem - there’s no other way 'round it. If there is, you must tell me YOU MUST TELL MEEEEE!

5 Likes

A very fair point :sweat_smile: I will say the recent livestream the Bubble team offered did give me hope this is an area they’re truly focusing on.

1 Like

@payam.azadi @allenyang for your attention :slight_smile:

All the big guns are saying this is a Bubble limitation. I don’t think you can get any better product feedback than this! Keeping in mind the necessity of search in any kind of app, can we expect Bubble to look into this anytime soon?