🔍 Introducing Scious Search: solve instant search once and for all

Showcase Plugins
21

Thank you for your interest Markus! Just messaged you with details :slight_smile:

1 Like
22

following… interested to give this a try soon

1 Like
23

@zelus_pudding this sounds awesome.

The privacy rule issue has stopped me being able to use Algolia so far - can you tell us some more of how you protect privacy rules?

1 Like
24

Great question! This discussion bumps into “why even use Bubble’s database to store records at all? Fetching data from Bubble only rings up WUs so wouldn’t it be better to avoid that altogether?”

Aside from the fact that the WU cost from fetching individual records is small (compared to the WU cost of Bubble’s native search), keeping our data in Bubble has two benefits:

  1. It’s as easy to use Bubble’s editor to work with our search results as it is to work with any other Bubble thing (because our search results are Bubble things).
  2. We get to leverage Bubble’s privacy rules.

When you normally do a search, Bubble will check what privacy rules apply to the data being requested and filter out records the user is not authorized to see.

When Scious Search does a search, the request that would normally go to Bubble goes instead to Algolia or Typesense. Those search providers will do what they do best - return results really fast - and we get back a list of the Bubble record IDs that matched your search query. Then, our plugin asks Bubble to fetch the actual data in the records we, at this point, only have IDs for. By now, Scious Search’s job is done. But Bubble will look at each record ID we asked for and - just as it does for it’s native search - will A) apply privacy rules to redact fields the user should not see and B) return the record with redactions.

In practice - this means that if a Scious Search result has fields your user shouldn’t see, then they simply won’t (those fields never make it to the browser). If the entire record was redacted because privacy rules prohibit them from seeing everything in the record, then (due to a quirk) they would simply see an empty entry as a search result. Repeating that: In this quirky case, even though no privileged information makes it to browser, the entry itself will take up an “empty” slot in your repeating group. This isn’t ideal, but what it really means is that the query generating these “blank entries” isn’t as “airtight/narrow” security wise as it should be. Here, the query needs to be tweaked to properly filter search results down to the number of records (since the content of those records will already be redacted) the requesting user should be able to see in the first place. At least we make it obvious :slight_smile:

Bottom line: if a user shouldn’t see a field from a Scious Search result, then they wont. If they’re drawing blanks in their search results, then tighten the filters.

3 Likes
25

I received an email from the forum indicating the I’d been mentioned in a post in one of the Workload discussions. Now it appears that post may have been taken down.

I will seriously want to evaluate Scious Search before long.

My main concerns, without any real investigation (because I don’t know where or how to start) is how seamlessly it works as a data source for all kinds of things that in “pure Bubble” would be done with a Do a Search. Things like resolving tags and mentions (# and @) in text blocks, Jici’s Selectize Dropdown, and I don’t know what else. And how easy and reliable is it to keep searchable datasets in sync with Bubble.

Anyway, I’ll get to it eventually. I have enough other issues to resolve, but effective, reliable, fast search are essential for making my app come close to the vision I have for it.

(This coming from an old timer who no longer wants to be a tech jock.)

2 Likes
26

I appreciate your comment, Laurence! I see now that my message was removed. Strange, that’s never happened to me :man_shrugging:

In any case, I’m happy to answer any question you have when you circle back on this. Of the points you did make, I just want to confirm our search does return native Bubble things which can be used in a repeating group or Jici’s Selectize Dropdown just like any list or Do a Search result. In terms of your desire to resolve tags and hashtags - I’m unsure if you’re asking whether Scious Search is able to match single symbols (like the @ or # sign) or search a list of user handles. In either case, the answer is yes. And finally, we’ve spent a lot of time making sure our synchronization capabilities (via bulk sync and CRUD operations) are robust.

When you’re ready to take a closer look, you can peruse our demo and docs (if you haven’t already), or ping me to get setup. Best!

27

Very impressive, @zelus_pudding! Great job! Going to keep this in my back pocket.

1 Like
28

Thanks Eli! That means a lot :slight_smile: been working on this for over a year so feels good to finally get the community’s eyes and positive feedback from forum legends like you :beers:

1 Like
29

Got a DM today with a good question. Decided to share here. The question is:

Why do we need to involve a 3rd party service to do instant search?

Answer as follows. If we take a peek at Bubble’s documentation, they say:

Bubble does have native search capabilities, but Algolia provides a more performant and customizable search experience at scale… Algolia is more suitable for Bubble apps at a large scale that rely on search as an important part of their user experience

Algolia is what’s know as a “Search Engine as a Service.” That means the product is designed to deliver results with the relevancy and speed of, say, a Google search. Google, however, only allows you to search webpages whereas Algolia allows us to upload our own data to make searchable.

To get to the heart of your question - why can’t we just use Bubble’s native search to get instant search like Algolia’s? The reality is, Bubble is built on a search engine similar to Algolia - it’s called ElasticSearch. While I can’t be sure of their exact setup, I do know that Bubble engineers need to balance tradeoffs in the way they integrate any system, and Elastic would be no exception. So there’s something about the cost vs complexity vs “also needing to serve data in non-search contexts” equation that results in Bubble’s native full-text search being slow.

This is why they recommend using a 3rd party search engine. Lucky for us, Algolia’s not the only game in town and we can shop alternatives like Typesense and Meilisearch (which can be cheaper as your database grows).

Would it be better if we could get instant search without having to integrate and pay for another service? Absolutely! Unfortunately, given how Bubble is setup, there really is no other way around it :frowning: Any plugin that’s going to actually perform the indexing-and-search of your data on page (like the ZQ Fuzzy Search plugin) is going to freeze the page and burn WUs when the number of records grow over a few thousand. But, that we can use another service to extend Bubble in this way I think underscores how flexible Bubble still is - we’ve always been able to improve apps via plugins and now, via ours, we can add instant-search (which is a premium capability… all things considered) rather cheaply and flexibly.

2 Likes
30

So, how’s it going?

31

Very interesting! What will be the pricing once released?

32

Hello Guillaume! There are two cost components to using our plugin - the plugin itself and the search provider you choose. We’ve priced our plugin at $9 monthly or $85 annually (where our annual subscription comes with a free half hour of one-on-one support). After that, the cost of using Typesense or Algolia comes down to the number of records to search as well as monthly search volume. Because that’s different for everyone, we’ve made a pricing calculator to help folks estimate their costs (note, our default for Average record size kb is a decent average you can start with if unsure).

To speak more to our pricing - we’ve designed our subscription to cover our costs while saving Scious Search users money. We actually do that in three ways. By:

  • Making it easy to shop around: there are many scenarios where choosing one search provider instead of another will save more money per year than the cost of our plugin.
  • Making it fast to integrate a search provider: we’ve spent a lot of time (unwillingly) discovering the corner cases where a search integration fails in Bubble. There’s lots of them. While an experienced Bubbler can setup a crude search integration without our plugin in short-ish order, it would still have ongoing maintenance and improvement costs that our solution simply eliminates.
  • Making it easy to switch search providers: We don’t expect anyone to switch providers for the fun of it but when the time comes, perhaps because your provider raised prices, or another reduced prices (as Algolia did recently), you don’t have to worry about paying someone to rip out the old integration to put in a totally new one. Switching is almost as easy as selecting your new provider from a dropdown menu. There’s a little more to it, since each provider has different filter syntax that would need updating. But those changes are minor. The time it took us to make the Typesense ecommerce template once we had the Algolia one made, was measured in minutes not days.

Of course, subscribing to our annual plan also saves money! That’s probably way more than you wanted to know about our pricing :grinning_face_with_smiling_eyes: but I figured it was relevant to share all the ways we’re ultimately - in my biased opinion - very cost effective. Hope that helps!

2 Likes
33

Pretty swell man! We still have a few slots available for the restricted beta but all in all it’s been a fun week onboarding our second round of early adopters. Been good for refining our docs while learning more about the cool businesses people have been building on Bubble :slight_smile:

If you’re interested in giving us a whirl, I’ve got room for ya!

1 Like
34

Looks good! I would also be interested in joining the beta

1 Like
35

Thanks for your interest Daniele! Just sent you a PM with next steps :slight_smile:

36

We just published a new section to our docs detailing how to use filters in Scious Search. Check out our Filters deep dive here!

And to speak to our docs for a moment, we remember what it was like being new to Bubble - the learning curve to using search effectively. Sometimes flexibility gives rise to complexity.

We see some of that in Scious Search. For the aspects that are harder for newcomers to pickup, it’s our mission to provide the best resource for mastering our products as quickly as possible. We think this new section is a step in that direction and, of course, welcome feedback to make it even better. Thank you!

1 Like
37

Looks great - is there still an open spot in your beta? :slight_smile:

38

Hey Moritzhd!
Just sent you next steps to get started :slight_smile:

For those wondering, we’re opening a few more spots in our beta. As part of that, we’re providing more hands on support to help our earliest users get setup. When all is refined, we’ll formally publish to the marketplace - in about two to three weeks!

39

This could be extremely helpful for me as I’ll have close to 10,000 products soon. I was thinking about hosting my backend on Xano soon because I make a lot of API calls. Would I be able to have This with Xano as a backend? I would imagine I might need two databases.

1 Like
40

Nice question Mack! As I believe you saw, we do require data be stored in Bubble to enforce privacy rules.

We have considered creating our service for the Xano ecosystem - to where you could exclusively store your data there and instant search it while respecting their privacy rules, but we’re just not there yet. Until then, happy to help with other questions you may have!