Will 100M backend workflows affect user experience?

[vovahumnytskiy]

Yep, I’ve already told above that Xano wouldn’t work since we have a complex use case and need to process a lot of data in a complex manner. We have even given it a try for Xano but weren’t able to make even a PoC.

We need a reasonable solution, not the one we have or someone else has experience with.

[NigelG]

As others have said, this is iterating on lists I would imagine, so we’ll outside Bubble’s current comfort zone at the moment.

I would be looking at some sort of ‘data pipeline’ here (just as you would with code).

These are just guesses, but maybe look at something NoSQL/MapReduce. Or Google Big Query.

Smash the raw API data into something that can handle the scale and then get it into a format that Bubble could access.

There are data pipeline tools that are ‘nocode’ and will work with BigQuery

[tylerboodman]

#ad​​

[troy.roberge]

I think you should give us a lot more information so we can give you intelligent answers. You have given zero info and expect help, it’s not likely.

Detail your use case, how you get and manage your data, and what you do with it. You don’t need to go into specifics of what the data is, but useful examples will help.

So far your Initial questions amount to… "I’m trying to pull a really big trailer with a vehicle, and it’s taking forever.

You aren’t clarifying the weight of the trailer, nor whether you are driving a Honda fit or a F350.

So please give more, to get more.

[redvivi]

@troy.roberge

I think you should give us a lot more information so we can give you intelligent answers. You have given zero info and expect help, it’s not likely.

I feel a kind of deja-vu…

@redvivi

@vovahumnytskiy

Would you migrate to custom code or pay more $$ for Bubble servers?

I would perform an architecture assessment against the use-case requirements first to provide relevant options.

[troy.roberge]

Yup. Hopefully people will start to learn to help themselves by giving proper info…

[tkamal]

Take it or leave it
One of the most valuable advances of using No-Code is the standard practice for the code
When this standard practice fails to help you in

• Performance
• Scalability
• Resources
• Hard to meantime
  I believe at this moment you should leave the No-code platform totally
  Maybe you can try Low code as I mentioned before if you have money because it is more expensive
  If you don’t have money don’t waste your time switching to traditional programming
  In addition, I do not recommend using multiple No-Code platforms in the same application unless you have strong IT experience to handle it.
  Also, I hope you can give us more information about your business platform

[dannyliu]

@keith

As others have said/implied all you need to do is write a little function in some external function-as-a-service that fetches the data from your Bubble-unfriendly API, transforms it into something more Bubble-friendly and returns that when called. (Basically you create your own API endpoint that is a proxy for the crummy API endpoint.) Or, if you’re in need of Things rather than API objects, your external function might hit your app’s bulk creation endpoints or whatnot.

I’m partial to Google Cloud Functions for this but there are many others (CloudFlare workers, Netlify functions, etc.)

I do exactly this to retrieve and parse iCal files in my GRUPZ app. (This is a very simple case of what I describe above. We don’t have to worry about authentication or whatnot as the iCal endpoints can just be gotten with a simple fetch, but the concept is super simple. iCal files can, in fact, have thousands of events in them going back years, but a simple function that parses and transforms them into a more bubble friendly JSON response takes milliseconds and costs practically nothing.)

This is the type of basic coding project that isn’t very complicated and wouldn’t cost much of anything to have a js/node developer handle it, if you don’t want to DIY.

Surely your bajillions of workflows are just because you’re constantly iterating over lists, right? (This isn’t “complex” or compute intensive, it’s just that Bubble makes it seem that way.)

Yup – basically this. You can consider using another no-code backend or switch to your own AWS server endpoints and then call them into your bubble app. Bubble just breaks at scale.

Beforehand you should probably hire a consultant to go through your bubble app and make sure you aren’t doing ridiculous stuff (running workflows on lists instead of recursively, loading data twice, running unnecessary WFs, etc).

[rafael.tgt]

I have a similar app here (with less functions calls than 100M). I decided to move it to a different backend. I create a linux server in Oracle (allways free, 4 cpu, 24 GB). A cron job start the python script, and the script send tasks to a queue (message broker using celery), with automatic retries if fail. Then it updates the bubble database with DATA API connection.