Search Large Data Types Efficiently and Fancily

Hello Bubble Friends,
I’ve slowly been working away at building an ERP system and I need to get serious about how to manage searches from a database that could potentially grow to include thousands of items, contain multiple fields, and will allow users to create, add, and delete items.

I’m coming to realize this will require decisions I’m ill informed to make in order to achieve the user experience I desire. I’d be interested in the options available from more experienced developers like yourselves. The use case is pretty simple…

Use Case:

  • User types in input.
  • A group focus drop down appears with a repeating group that populates and updates based on what the user types in the input.
  • User selects an item from the drop down which then populates the search input and probably sets a state.
  • Autocomplete and fuzzy search qualities would be preferred.
  • User will be need to be able to add and delete entires to the database.

Goals:

  • Quick, intuitive, and efficient search return.
  • I understand this will cost some amount of money to maintain, but I’d prefer it to not be ridiculous.

Potential Options:
Algolia search (via native Bubble or Zeroqode plugin):

  • Pro: fast and efficient.
  • Con: potentially very costly?

Xano backend (or an equivalent):

  • Pro: cost efficient? speed efficient?
  • Con: unclear, since I don’t know much about development outside of what I’ve taught myself in Bubble.

Something else I’m not thinking of because I don’t know about it:

  • Pro: it’s great in my dreams.
  • Con: I won’t know about it until you tell me it exists.

Thoughts?!

I can not comment with integrity as I have not used the native Bubble algolia or the zeroqode plugin recently (once about 7 years ago). Maybe a developer with experience using native Bubble algolia could speak to it, but the potential costs based on reading Algolia pricing page could add up quickly. Seems like a ‘search request’ would be performed on any new sort option selected or new filters applied and likely changing characters typed for the ‘matching’ from an input value. I’m not sure though, as I have no real experience to be a source of truth on that.

From what I have heard people like using Xano as a backend. When WU pricing was announced a lot of developers flocked to it. I don’t know how cost efficient it is as you will still get charged for the API calls and characters of data returned from Bubble in form of WU consumption. Some developers seem to have trouble understanding how to use it in their apps and opine that if you were to use Xano as a backend, you might want to use a different front end as in their mind, Bubble is a poor front end platform. So, that could be noteworthy, and perhaps some developers with experience using Xano could speak to their experience with it.

Data Jedi Plugin.

The cost is the monthly subscription or the one time payment. The intuitive is covered by it being all in Bubble, but does take a bit of comprehension around what is a Hybrid Data Structure and how does it enable extreme cost savings and in a lot of instances more efficient searches.

Data Jedi Plugin is not a one trick pony. It has a ton of features, elements and actions. One of the elements is the Searchbox which I built as I was tired of having to build these ‘custom dropdowns’ using the input, group focus, repeating group etc. for each data type I want to search by.

Check out the Searchbox Jedi element in action

The searchbox Jedi element is complemented by an equally powerful multi select dropdown element. Both allow you to search/filter by a typed value without needing to ‘re-fetch’ the data, so costs are nil.

With the plugin Data Jedi element and it’s associated client side and server side actions, this is as simple as expected in a NoCode environment like Bubble. No code needed, no hoops to jump through for ‘index matching’, just simple, straightforward actions like the actions any bubble developer is used to in Bubble.

There are numerous ways to save the data, and locations you can do it to as the plugin has an export feature that allows you to export the data as JSON file, CSV, or 5 other file types. This means, depending on your data and use case for that data, you have flexibility and options of how to store it. Gives a competent developer that ability to really think deeper on the data structure and decide on the approach that fits the data, the app, it’s users and their intended use of the data.

I’ve been building in Bubble for 7.5 years and I built the plugin to solve the problems I encounter in Bubble and make ‘impossible’ possible in Bubble. I build with it everyday myself, so I am a bit biased as the more I build with it, the more possibilities emerge.

I’m sure other developers have their own opinions based on their own experiences.

Always smart to think about tomorrow, today, so you do not end up building something that doesn’t match your needs as your business grows. Nobody wants to get stuck paying more than they need to for the same functionality, or need to rebuild features whose data needs cause them to be too costly.

Is your app as an ERP meant for a single company or multiple companies? Have you set it up as a SPA or is multiple pages?

Enterprise apps like an ERP system benefit a lot from the Hybrid Data Structure approach and Data Jedi through cost savings of 95% on data fetch since the user base is expected to interact with it throughout their day, it is cost efficient to just return all data on first load (as a SPA normally does) that the user is expected to require. Then on a needs basis, fetch and return the data at the time of the request.

1 Like

For sure not aloglia, as it exposes data outside which is suicide with any ERP.
I heard about Data Jedi which was mentioned above - would need to try it someday.
If you go for Xano, it’s in the end cheaper to go with WeWeb than bubble. Depending on how much operations you need to do there. Also - if you are in EU and want to have any personal data - Xano or other backend is a must as bubble is in US and it’s EU server costs rocket high. In Xano, much less. Wish they would go with it for less.

2 Likes

Thank you for such a thorough response. I’ll be sure to check out Data Jedi. I’ve been reticent about plugins because I’m paranoid about relying on their continued support from their creator. I don’t know how realistic that paranoia is though, ha.

I’m making the ERP system for my own job, because I’m in a very niche industry and there are no options customized for us. But if my system works out as planned, I’d be well positioned to offer it to other facilities in my industry.

Thank you!

here are some from my experience:

  • it’s 1WU per search whatever your database size is
  • loading duration is almost the same if you only use simple filtering in do a search for
  • using 1-2 level deep do a search for will tank it
  • using advanced filter will tank it
  • don’t use realtime search bindings (using input values in do a search for function)
  • use a default query in do a search for, for the first load, then update/filter it with a click of a button
  • dont use client-side :filter function
  • you have to use pagination for best performance