Sending emails to +100k users issue

[jaos.pcl]

We are sending emails with the Sendgrid plugin. All looks fine in our account on Sendgrid.

We divide the workflow in three different parts since the maximum number of items that can be retrieved from the DB is 50k(right?):

1. Schedule api-workflow on a list, which runs on users Search for Users:items until #50000.
2. Schedule api-workflow on a list, which runs on users Search for Users:items from #50001:items until #100000.
3. Schedule api-workflow on a list, which runs on users Search for Users:items from #100001.

The workflow it schedules contains the action where we send the email via Sendgrid. We schedule the API workflow “scheduler_send_to_all” from our admin panel.

It works fine in dev mode, but in run mode, nothing happens. No error appears in the server logs either. What am I missing?

Thanks in advance!

[adamhholmes]

How many users do you have in your dev database?

[jaos.pcl]

Around 15 users. But the principle should be the same? Or at least the first 50k users should receive the email, and they have not.

[adamhholmes]

Trying to run a workflow on a list of 50k Users is never going to work (at least reliably) - in your dev version it’s only 15 Users, so of course it will work there. (although I would still expect some of the Users to have received the email - so I don’t know what’s going on there)

Scheduling a workflow to run on a list works fine with a few things (usually only recommended for lists of maybe 10 things, 100 tops - it might even work with 1000 things - but 50k is way too much - at least on the current ‘capacity based’ pricing model, unless maybe you’re on a dedicated plan).

Besides, under the new pricing model it will be crazy expensive doing it this way, even if you could (paying for 100k outbound API calls, 100k Schedule actions, 100k Server actions)

You don’t want to use Bubble to send individual emails for this - it will take forever and cost a fortune.

Use SendGrid’s bulk email actions - you can send an email to 1000 email addresses with a single API call - so you’ll just need to set up a recursive workflow to run 100 times (or until the full list has been sent) - so batch the contacts into groups of 1000.

[jaos.pcl]

Many thanks for your answer, much appreciated.

I’ve never worked with such large lists. What is the best approach to batch the contacts?

If I want to edit values on 100k rows, and if I can’t use the API workflow on a list, how do I go about?

[adamhholmes]

Use a recursive workflow… (with a parameter for a list of Users - which will be the first 100 Users from your database)

Have an action to send the email to those 1000 users, then reschedule the workflow to run, passing the next 1000 users into the list field.

Continue until all Users have been included.

[zelus_pudding]

I bulk email via SendGrid to 10s of thousands of users. Batching in groups of a thousand, in my experience, is even a bit much - not for SendGrid but for Bubble’s backend. For example, ever try bulk deleting 1000 records within the data tab? Bubble times out unless you’re already paying for extra capacity. We batch in groups of 200 to 400 to have overhead available in the event someone wants to trigger additional (concurrent) mass email campaigns. Set this up in a recursive workflow, as @adamhholmes has suggested, and have logic in there to retry the current recursive loop (say up to three times) in the event it fails. This is what we do and it works well.

[jaos.pcl]

Thank you both for your inputs. Would it look something like this?

Can the :minus listoperation handle 500 entries?

@zelus_pudding How exactly do you determine if the event fails? What’s the logic? Thanks!