Challenge: Message Queues

Need help Database
1

Hey everyone!

What’s the best way to set up message queues in Bubble?

Anyone unfamiliar, a queue sets up requests and processes them in order (or whatever business logic is required).

Interested to hear how others have set this up.

message queue
message queue901×611 2.8 KB

First to resolve this one gets extra kudos

2

How guaranteed must it be that they’re processed in order? What are the consequences if it doesn’t? Start in order, or finish in order? Is it also a requirement that only one can run at a time? I think the answer will depend on the specific constraints of the problem.

3

Thanks @georgecollier ! I was hoping you would answer.

Currently it’s a requirement to handle sequentially - but a good message queue system will be able to work with and change based on different constraints/rules/conditions (i.e. be reused)

I think the easiest answer is to your question is yes - it has to be guranteed to work in order (as it’s easier to work backwards from there)

4

Cool okay.

Thinking out loud but the limitations we must be aware of are:

  • no ‘transactions’ meaning that race conditions will always be an issue, that we must mitigate through other means and/or recover from

Data types

Queue

  • key (text)
  • isProcessing (yes/no)
  • nextSequenceNumber (number)
  • lastHeartbeatDate

Queue Job

  • Queue
  • sequenceNumber
  • Status: Pending, Processing, Done, Failed
  • Type: option set (not strictly required)
  • fields for whatever you need (or store it as JSON in a text field and parse inside the queue)
  • errorMessage (text)

Backend workflows

You’d have something like Enqueue Job and Process Queue where Enqueue Job creates a new Queue Job, sets sequence to Queue's nextSequence + 1and then increases the nextSequenceNumber of the Queue

Then Process Queue would have Only when Queue's isProcessing is no, set isProcessing to yes, pick up the next job + set the status, and if no job exists, mark as not processing. It would reschedule in whatever your maximum tolerance for delay in the queue is (e.g if you reschedule in 10 seconds when there is no job, then it’ll check every 10 seconds for a new job to pick up). If there was a job then you reschedule it immediately.

It’s far from perfect and definitely not the simplest possible implementation, but it covers a few areas:

  • nextSequenceNumber is useful to guarantee order. You could probably order by Created Date on the Queue Job for first-in-first-out, but that might not cover all cases and wouldn’t allow re-ordering
  • it’s easy to delete jobs
  • you can handle race conditions on the sequence fairly well (e.g if using schedule api workflow on a list to create a ton of queue jobs, you can calculate the sequence number when scheduling the workflow rather than when running it)
  • most importantly, this architecture (queue ‘key’) supports any arbitrary queue that you create with a key. You could have a per-user queue, per-feature queue, per-org etc. The queue key is dynamic and the system works with them. Of course, not every app will require this
  • you’d have some kind of heartbeat thing to make sure queues haven’t gotten stuck/crashed unexpectedly
5

Thanks @georgecollier , this one is going to be fun to test

6

Not sure if relevant to your use case, but I built this queueing system some time back for an on demand service. The service was where a car mechanic would accept a repair job that was posted to the platform.

When a mechanic accepted a job, it created a request queue instead of updating the service request type directly. The queue would take in the mechanics proximity and their average rating, sort it based on the the closest with the highest rating and select #1. Then it would reject the rest.

However if there were two mechanics at the same distance and the exact rating, then based on Bubble’s sort, one of them would get lucky :slight_smile: I never got a chance to test this out in scale, because the app was a personal one and did not go beyond a few test users. One thing that was in the roadmap was handling scenarios where the first selected could not make it so that would then mean selecting #2 or re sending out the request again.

image
image2026×1609 223 KB

7

I tried similar methods to both suggested here. Mine even handled orphans. The chance to break is proportional to concurrency intensity.

As long as a workflow depends on Bubble incrementing a value, it will break.

My solution for atomicity is Bubble> CF Queues > Bubbleor Bubble> CF Durable Object > Bubble

These are easier to setup, cheap and reliable.

8

Hey everyone - the results of the tests are in!

TLDR; Queues that mostly can be set up with Bubble, for mission critical queues, use Cloudflare or an external system.

Kudos Awards:
#1 @ihsanzainal84 for the most production ready system :tada:
#2 @georgecollier for the Bubble native approach

Bubble Queues
You basically do as George outlined

  1. Queue the job
  2. Check for latest processing Job
  3. Run the action
  4. Complete the action (and scheduled the next processing job)

Results:

  • Mostly works (with some concurrency errors slipping through)

Cloudflare Queues
These can be set up fairly easilly with Durable Object, but it does introduce a another external platform to manage. This is far more reliable and customizable with different conditions, delays etc.

You basically:

  1. Create your worker setup
  2. Queue with an ID and timestamp
  3. Call the Bubble App
  4. Let Cloudflare know it’s completed
  5. Cloudflare let’s Bubble know to run the next one

Here’s a video: