Impossible? API Workflows on large (heavily filtered) lists

Ok - I’ve got a tricky situation here. Hoping any of you Bubble gurus might have an idea I haven’t yet thought of!

  • BACKGROUND INFORMATION: My app is a B2B SaaS that helps businesses interact with their customers in all sorts of ways. These customers have all sorts of data stored on them: Names, phone #s, Birthdays, labels, tags, addresses, notes, etc… Each “Customer” (thing) also has information stored about what interactions have already occurred with them, whether they’ve clicked on links, etc…
    Also note that my app has a very robust filtering system, and users can filter their customer lists (repeating groups) down based on any of the criteria / data held on those customers.

  • OBJECTIVE: I want to be able to reliably schedule API workflows on large, filtered, repeating group lists.

  • PROBLEM: When I “select all” customers and send to ALL of them (not filtered) recursively, it’s all good. When I have a filtered list that displays fewer than ~250-500 customers, it’s all good. Both of those scenarios are reliable in Bubble.
    The problem, however, is when I filter, for example, my repeating group’s list of 10,000 customers down to 5,000 & try to “select all” & schedule an API worflow on those 5,000.
    (1) I can’t reliably schedule this API workflow on the list of “customers in that repeating group”, because it takes forever & usually gives up. I can’t “make changes to a list of things” to tag those 5,000 somehow, because that also takes forever & crashes my app.
    (2) I can’t schedule this list recursively, because it’s not actually a list saved in the database (it’s just a filtered repeating group), and because the thousands of possible filtering combinations don’t permit me to reliably know how to tell the API workflow to “Do a Search for” X list :filtered to :filtered to :filtered to : filtered to… etc…

So, I’m stuck with either selecting ALL customers & sending recursively, OR filtering down to extremely small lists… Anything in between is unreliable…

Thoughts? I really hope I’m missing something glaringly obvious here. Thanks in advance!

@keith’s List Shifter plugin may be able to help you out here with it’s iteration capability.

Here’s Keith demonstrating it in a video: List Shifter: Reverse, Rotate, Swap and ITERATE (Loop) Over Bubble Lists | Now at v1.4: Adds Numeric Option, GET INDEX Action

1 Like

Is my understanding correct that all the filtering is happening client-side? If so, I wonder if a proper long term solution would be to refactor your logic such that all “filtering” is applied as constraints server-side.

If you then saved “views” (sets of filter criteria) to the DB, in addition to being a cool feature for your clients, that would also enable scheduled workflows on any defined “view” of the data; and it would likely improve performance as well, since only the filtered subset of data would be fetched from the server.

1 Like

Having done a bit more thinking and tinkering (thinkering?), it seems this should be possible using Bubble’s data API, since that would allow the search constraints to be saved as strings which could then be passed in the query string to fetch the data.

Basically, the app would make repeated API calls to itself (an exposed data endpoint) to process a large list in batches of 100 (the largest “page size” allowed by the data API).

I don’t have a working example, but it seems doable.

1 Like

This is your problem. Of course, you CAN reconstruct the search that lead to the list so that you can do this request entirely server-side. You just need to design an abstraction layer between (what I presume your users do to create the) lists and the database.

But here’s my other question/suggestion:

It sounds like you do some complex search in a page that results in a list of Clients or whatever.

Why not, while you have that list in the page, just save it to the database?

Are you worried that object (a list of Clients or whatever) is large? It’s not.

Bubble lists of app data objects are just lists of unique ids. They are just POINTERS to the underlying data objects. (As they should be.)

So the question is: What happens if you save your example list of about 5000 clients to the db? (Make a new data type “Client List” that has one field in it — a list of type Client.) Then, schedule your API Workflow on that list.

If it doesn’t work, that sounds like a bug. (File a bug report with Bubble.)

If you’ve tried this, let us know the results.

Back to the first part of my answer: To build an abstraction later for searches, you’ll create a data type that has search constraint and parameter and just parse that back. Not the easiest thing to do in Bubble and a little coding would help out, but don’t think this is “undoable.”

1 Like

Wow. I thought I had thought of everything. This was a last-ditch hailmary & you all gave me newfound hope!! This community never ceases to amaze. THANK YOU all!!

@eli - this plugin looks AMAZING, I can’t believe I’d never heard of it before - thanks! Will definitely give it a try… The video seems to show what I’d need…

@sudsy - This is a very interesting idea… You mean simply use the Bubble app connector to have my app call itself?

@keith - Thanks for the thorough answer. I have repeatedly tried saving the list of 5,000 to a CUSTOM STATE list, but not to a list in the database… For whatever reason, saving to a custom state’s “list of customers” takes forever & usually times out. I just assumed saving to the database would give the same result… Should the database be different?

1 Like

Yes, that’s the idea,. While I don’t have a lot of experience with the App Connector (might be possible using that), I did take it far enough with the API Connector to be convinced that all the pieces are there and that It should be possible using pure Bubble.

Needless to say, if @keith’s suggestion of saving to the database works, that’s the ticket for getting over the hump in the short-term. Moving the filtering logic to the server, though, seems like the best long-term option - whether it’s through an “abstraction layer” or using the Data API.

1 Like

This topic was automatically closed after 70 days. New replies are no longer allowed.