This is correct. Nevertheless, the figure’s I’ve cited cover the last 30 days - a time period over which I (re)uploaded a CSV with 10,000 records two or three different times and re-synced that data to my search providers 3 to 5 times. You’ll notice on my first pie chart the bright (baby) blue section which show those CSV imports. It’s all in the mix.
1 Like
Just reached out to you via DM!
I’ve been using this plugin in beta for a month or so. I can tell you it has been really helpful for us at get-gigs.com, specifically for location based searches. In particular, our users have geographic areas they are looking in and we need to match three or four of those geographic areas with other users’ three or four areas as well as general metropolitan areas (and add in another 30+ combinations of genres, keywords, etc). Previously I had multiple merged searches using a variety of methods and conditions. It was slow (especially the geographic stuff) and difficult to modify because of the complexity. Using typesense has allowed us to move our main gig-searching queries from eight conditional data sources and multiple “unique id is in” type searches to a single query that is generated in javascript based on a data type that gets updated and triggers a new search in typesense. I can’t speak to WU savings as I don’t have a full handle on it for us as yet. I will say that keeping the typesense db up to date should likely not be done through a data trigger, as I would’ve done previously.
Overall, this plugin has improved the user experience for searches and has allowed us to prototype and consider more complicated maps based functionality for our users. @zelus_pudding has been very responsive and helpful as I was learning the ins and outs of the typesense query nomenclature and also when I ran into a bug or two during development. Would recommend you give it a try if it meets your use case.
7 Likes
looks interesting, actually I’m currently developing a commercial directory on Bubble and this might be a very awesome solution. I’ll DM you in a couple of weeks to test it out.
2 Likes
You mentioned you’re happy with a webinar recording at this point - I’ve just updated the webinar links so folks can register / get the recording afterwords. Of course, happy to follow up with you if you have questions 
@grottofilms , I hadn’t realized you were able to compress that many conditional data sources into one of our search filter queries. That’s so awesome! Very glad it’s working well for you 
1 Like
I have been using Scious Search in my app for 6 months now and its working incredible well. Some of our tables have >100k records. We were using Fuzzy search and it was taking 25 seconds to return a result.
Now, the search is lightening fast. Scious search returns results in a few milliseconds. The plugin also lets us sort data alphabetically and separate it by user using the privacy rules.
We sync to Typsense which is costing us about $10/m. The cost is negligible compared to the performance we now get.
I highly recommend Scious search for anyone looking to hold more than 2K records in their database.
5 Likes
Is there a support for list containing or intersecting with other list?
Yes! Both Algolia and Typesense support searching for an item within a list - this is a very common operation. We even do it a few times in the demo (note my yellow highlight in our filters setup):
1 Like
Great! Sign me up for beta!
1 Like
Thank you for your interest Markus! Just messaged you with details
1 Like
following… interested to give this a try soon
1 Like
@zelus_pudding this sounds awesome.
The privacy rule issue has stopped me being able to use Algolia so far - can you tell us some more of how you protect privacy rules?
1 Like
Great question! This discussion bumps into “why even use Bubble’s database to store records at all? Fetching data from Bubble only rings up WUs so wouldn’t it be better to avoid that altogether?”
Aside from the fact that the WU cost from fetching individual records is small (compared to the WU cost of Bubble’s native search), keeping our data in Bubble has two benefits:
- It’s as easy to use Bubble’s editor to work with our search results as it is to work with any other Bubble thing (because our search results are Bubble things).
- We get to leverage Bubble’s privacy rules.
When you normally do a search, Bubble will check what privacy rules apply to the data being requested and filter out records the user is not authorized to see.
When Scious Search does a search, the request that would normally go to Bubble goes instead to Algolia or Typesense. Those search providers will do what they do best - return results really fast - and we get back a list of the Bubble record IDs that matched your search query. Then, our plugin asks Bubble to fetch the actual data in the records we, at this point, only have IDs for. By now, Scious Search’s job is done. But Bubble will look at each record ID we asked for and - just as it does for it’s native search - will A) apply privacy rules to redact fields the user should not see and B) return the record with redactions.
In practice - this means that if a Scious Search result has fields your user shouldn’t see, then they simply won’t (those fields never make it to the browser). If the entire record was redacted because privacy rules prohibit them from seeing everything in the record, then (due to a quirk) they would simply see an empty entry as a search result. Repeating that: In this quirky case, even though no privileged information makes it to browser, the entry itself will take up an “empty” slot in your repeating group. This isn’t ideal, but what it really means is that the query generating these “blank entries” isn’t as “airtight/narrow” security wise as it should be. Here, the query needs to be tweaked to properly filter search results down to the number of records (since the content of those records will already be redacted) the requesting user should be able to see in the first place. At least we make it obvious
Bottom line: if a user shouldn’t see a field from a Scious Search result, then they wont. If they’re drawing blanks in their search results, then tighten the filters.
2 Likes
I received an email from the forum indicating the I’d been mentioned in a post in one of the Workload discussions. Now it appears that post may have been taken down.
I will seriously want to evaluate Scious Search before long.
My main concerns, without any real investigation (because I don’t know where or how to start) is how seamlessly it works as a data source for all kinds of things that in “pure Bubble” would be done with a Do a Search. Things like resolving tags and mentions (# and @) in text blocks, Jici’s Selectize Dropdown, and I don’t know what else. And how easy and reliable is it to keep searchable datasets in sync with Bubble.
Anyway, I’ll get to it eventually. I have enough other issues to resolve, but effective, reliable, fast search are essential for making my app come close to the vision I have for it.
(This coming from an old timer who no longer wants to be a tech jock.)
2 Likes
I appreciate your comment, Laurence! I see now that my message was removed. Strange, that’s never happened to me 
In any case, I’m happy to answer any question you have when you circle back on this. Of the points you did make, I just want to confirm our search does return native Bubble things which can be used in a repeating group or Jici’s Selectize Dropdown just like any list or Do a Search result. In terms of your desire to resolve tags and hashtags - I’m unsure if you’re asking whether Scious Search is able to match single symbols (like the @ or # sign) or search a list of user handles. In either case, the answer is yes. And finally, we’ve spent a lot of time making sure our synchronization capabilities (via bulk sync and CRUD operations) are robust.
When you’re ready to take a closer look, you can peruse our demo and docs (if you haven’t already), or ping me to get setup. Best!
Thanks Eli! That means a lot been working on this for over a year so feels good to finally get the community’s eyes and positive feedback from forum legends like you 
1 Like
Got a DM today with a good question. Decided to share here. The question is:
Why do we need to involve a 3rd party service to do instant search?
Answer as follows. If we take a peek at Bubble’s documentation, they say:
Bubble does have native search capabilities, but Algolia provides a more performant and customizable search experience at scale… Algolia is more suitable for Bubble apps at a large scale that rely on search as an important part of their user experience
Algolia is what’s know as a “Search Engine as a Service.” That means the product is designed to deliver results with the relevancy and speed of, say, a Google search. Google, however, only allows you to search webpages whereas Algolia allows us to upload our own data to make searchable.
To get to the heart of your question - why can’t we just use Bubble’s native search to get instant search like Algolia’s? The reality is, Bubble is built on a search engine similar to Algolia - it’s called ElasticSearch. While I can’t be sure of their exact setup, I do know that Bubble engineers need to balance tradeoffs in the way they integrate any system, and Elastic would be no exception. So there’s something about the cost vs complexity vs “also needing to serve data in non-search contexts” equation that results in Bubble’s native full-text search being slow.
This is why they recommend using a 3rd party search engine. Lucky for us, Algolia’s not the only game in town and we can shop alternatives like Typesense and Meilisearch (which can be cheaper as your database grows).
Would it be better if we could get instant search without having to integrate and pay for another service? Absolutely! Unfortunately, given how Bubble is setup, there really is no other way around it 
Any plugin that’s going to actually perform the indexing-and-search of your data on page (like the ZQ Fuzzy Search plugin) is going to freeze the page and burn WUs when the number of records grow over a few thousand. But, that we can use another service to extend Bubble in this way I think underscores how flexible Bubble still is - we’ve always been able to improve apps via plugins and now, via ours, we can add instant-search (which is a premium capability… all things considered) rather cheaply and flexibly.
1 Like
So, how’s it going?
Very interesting! What will be the pricing once released?