This is correct. Nevertheless, the figure’s I’ve cited cover the last 30 days - a time period over which I (re)uploaded a CSV with 10,000 records two or three different times and re-synced that data to my search providers 3 to 5 times. You’ll notice on my first pie chart the bright (baby) blue section which show those CSV imports. It’s all in the mix.
Just reached out to you via DM!
Overall, this plugin has improved the user experience for searches and has allowed us to prototype and consider more complicated maps based functionality for our users. @zelus_pudding has been very responsive and helpful as I was learning the ins and outs of the typesense query nomenclature and also when I ran into a bug or two during development. Would recommend you give it a try if it meets your use case.
looks interesting, actually I’m currently developing a commercial directory on Bubble and this might be a very awesome solution. I’ll DM you in a couple of weeks to test it out.
You mentioned you’re happy with a webinar recording at this point - I’ve just updated the webinar links so folks can register / get the recording afterwords. Of course, happy to follow up with you if you have questions
@grottofilms , I hadn’t realized you were able to compress that many conditional data sources into one of our search filter queries. That’s so awesome! Very glad it’s working well for you
I have been using Scious Search in my app for 6 months now and its working incredible well. Some of our tables have >100k records. We were using Fuzzy search and it was taking 25 seconds to return a result.
Now, the search is lightening fast. Scious search returns results in a few milliseconds. The plugin also lets us sort data alphabetically and separate it by user using the privacy rules.
We sync to Typsense which is costing us about $10/m. The cost is negligible compared to the performance we now get.
I highly recommend Scious search for anyone looking to hold more than 2K records in their database.
Is there a support for list containing or intersecting with other list?
Yes! Both Algolia and Typesense support searching for an item within a list - this is a very common operation. We even do it a few times in the demo (note my yellow highlight in our filters setup):
Great! Sign me up for beta!
Thank you for your interest Markus! Just messaged you with details
following… interested to give this a try soon
@zelus_pudding this sounds awesome.
The privacy rule issue has stopped me being able to use Algolia so far - can you tell us some more of how you protect privacy rules?
Great question! This discussion bumps into “why even use Bubble’s database to store records at all? Fetching data from Bubble only rings up WUs so wouldn’t it be better to avoid that altogether?”
Aside from the fact that the WU cost from fetching individual records is small (compared to the WU cost of Bubble’s native search), keeping our data in Bubble has two benefits:
- It’s as easy to use Bubble’s editor to work with our search results as it is to work with any other Bubble thing (because our search results are Bubble things).
- We get to leverage Bubble’s privacy rules.
When you normally do a search, Bubble will check what privacy rules apply to the data being requested and filter out records the user is not authorized to see.
When Scious Search does a search, the request that would normally go to Bubble goes instead to Algolia or Typesense. Those search providers will do what they do best - return results really fast - and we get back a list of the Bubble record IDs that matched your search query. Then, our plugin asks Bubble to fetch the actual data in the records we, at this point, only have IDs for. By now, Scious Search’s job is done. But Bubble will look at each record ID we asked for and - just as it does for it’s native search - will A) apply privacy rules to redact fields the user should not see and B) return the record with redactions.
In practice - this means that if a Scious Search result has fields your user shouldn’t see, then they simply won’t (those fields never make it to the browser). If the entire record was redacted because privacy rules prohibit them from seeing everything in the record, then (due to a quirk) they would simply see an empty entry as a search result. Repeating that: In this quirky case, even though no privileged information makes it to browser, the entry itself will take up an “empty” slot in your repeating group. This isn’t ideal, but what it really means is that the query generating these “blank entries” isn’t as “airtight/narrow” security wise as it should be. Here, the query needs to be tweaked to properly filter search results down to the number of records (since the content of those records will already be redacted) the requesting user should be able to see in the first place. At least we make it obvious
Bottom line: if a user shouldn’t see a field from a Scious Search result, then they wont. If they’re drawing blanks in their search results, then tighten the filters.
I received an email from the forum indicating the I’d been mentioned in a post in one of the Workload discussions. Now it appears that post may have been taken down.
I will seriously want to evaluate Scious Search before long.
My main concerns, without any real investigation (because I don’t know where or how to start) is how seamlessly it works as a data source for all kinds of things that in “pure Bubble” would be done with a Do a Search. Things like resolving tags and mentions (# and @) in text blocks, Jici’s Selectize Dropdown, and I don’t know what else. And how easy and reliable is it to keep searchable datasets in sync with Bubble.
Anyway, I’ll get to it eventually. I have enough other issues to resolve, but effective, reliable, fast search are essential for making my app come close to the vision I have for it.
(This coming from an old timer who no longer wants to be a tech jock.)
I appreciate your comment, Laurence! I see now that my message was removed. Strange, that’s never happened to me
In any case, I’m happy to answer any question you have when you circle back on this. Of the points you did make, I just want to confirm our search does return native Bubble things which can be used in a repeating group or Jici’s Selectize Dropdown just like any
Do a Search result. In terms of your desire to resolve tags and hashtags - I’m unsure if you’re asking whether Scious Search is able to match single symbols (like the
# sign) or search a list of user handles. In either case, the answer is yes. And finally, we’ve spent a lot of time making sure our synchronization capabilities (via bulk sync and CRUD operations) are robust.
Very impressive, @zelus_pudding! Great job! Going to keep this in my back pocket.
Thanks Eli! That means a lot been working on this for over a year so feels good to finally get the community’s eyes and positive feedback from forum legends like you
Got a DM today with a good question. Decided to share here. The question is:
Why do we need to involve a 3rd party service to do instant search?
Answer as follows. If we take a peek at Bubble’s documentation, they say:
Bubble does have native search capabilities, but Algolia provides a more performant and customizable search experience at scale… Algolia is more suitable for Bubble apps at a large scale that rely on search as an important part of their user experience
Algolia is what’s know as a “Search Engine as a Service.” That means the product is designed to deliver results with the relevancy and speed of, say, a Google search. Google, however, only allows you to search webpages whereas Algolia allows us to upload our own data to make searchable.
To get to the heart of your question - why can’t we just use Bubble’s native search to get instant search like Algolia’s? The reality is, Bubble is built on a search engine similar to Algolia - it’s called ElasticSearch. While I can’t be sure of their exact setup, I do know that Bubble engineers need to balance tradeoffs in the way they integrate any system, and Elastic would be no exception. So there’s something about the cost vs complexity vs “also needing to serve data in non-search contexts” equation that results in Bubble’s native full-text search being slow.
This is why they recommend using a 3rd party search engine. Lucky for us, Algolia’s not the only game in town and we can shop alternatives like Typesense and Meilisearch (which can be cheaper as your database grows).
Would it be better if we could get instant search without having to integrate and pay for another service? Absolutely! Unfortunately, given how Bubble is setup, there really is no other way around it Any plugin that’s going to actually perform the indexing-and-search of your data on page (like the ZQ Fuzzy Search plugin) is going to freeze the page and burn WUs when the number of records grow over a few thousand. But, that we can use another service to extend Bubble in this way I think underscores how flexible Bubble still is - we’ve always been able to improve apps via plugins and now, via ours, we can add instant-search (which is a premium capability… all things considered) rather cheaply and flexibly.
So, how’s it going?
Very interesting! What will be the pricing once released?