We have a WU issue with a workflow trigger. We have a single trigger that presently costs us 2,000-3,000 WU per day per user, and we need it to be more like 200-500 per day per user.
We have a master RSS feed from a site that saves jobs to our database.
Users set their filter for their ideal job, and they get notified via email when a job meets their criteria.
The action that triggers the workflow to notify them when a job meets their criteria is the one we have a problem with.
Need help working out a better way to do this.
You probably want to use an external database here like Google Firebase. Make RSS feed write to your Firebase, then create a Google Cloud Function that posts the new jobs to your backend every hour.
Okay so correct me if I’m wrong:
- Every day, a backend workflow checks the RSS feed and saves the data to Bubble.
- Each user has a daily recurring/recursive workflow that searches the Bubble DB for jobs that are created (Created Date) in the last 24 hours that meet specified constraints.
- If the search isn’t empty, it sends the results in an email.
Which step is costing thousands of WUs? This should only be like 10 WUs max per user per day.
Edit: Oh wait, by ‘workflow trigger’ do you mean a backend trigger that runs every time a job is created/updated that checks the criteria against every user’s criteria? If so, that’s not necessary if you only need job notifications to be sent out daily. Just use the method I outlined above
Every 15 minutes a backend workflow checks the RSS feed and saves the data to bubble. But this isn’t the workflow causing concern.
You’re right, every 15 minutes the back end checks if the jobs posted to the main feed match a users set criteria, and if it does it sends them a notification.
But that trigger costs 10x what it should. It’s a core value prop for people to be notified of the job ASAP, so unfortunately can’t do it once per day.
We could indeed do that, but funnily enough the RSS feed updating our back end isn’t the issue, as it doesn’t increase as our user numbers increase.
The issue is the trigger that starts the workflow to get someone notified. It presently looks through the jobs that have been posted sees if they match a user’s criteria and if it does, triggers the workflow
See I don’t understand how that’s costing 2,000 WU per today. For each user, it should be 1. a constrained search 2. trigger a workflow if result isn’t empty. That’s probably like 5 WU total = 480 per day.
Can you share the actions in detail?
Ok then this looks like a data modelling issue. How are your users setting alerts? What’s the data structure?
Jobs data type contains criteria (probably a few fields)
Job User data type contains criteria (probably a few fields) and a User
When Job is scraped, do a search for Job User with the relevant criteria (using search constraint)
Schedule API workflow on the list of Job Users / schedule recursive workflow using the list of Job Users. The recursive workflow will send the email to each User.
Rather than going through each user every time to check if a job is relevant, instead, you can just search for Users that match the job criteria. So, job added, DB search to find all relevant users, recursive workflow to send notifications to all of these users.