Has something changed with the Bulk APIs? It doesn’t seem to propagate updates anymore.
Context
In one of my apps, I’ve got a simple stats widget that displays the number of ‘yes’ answers out of total questions; e.g.: 3 out of 10. When the user updates his answers, creates new snapshot, the widget will update; e.g.: 4 out of 10.
This has been present for months (years?) and it’s always worked. Recently I started getting reports that there was something wrong. Indeed, sometimes, the widget will display something impossible like 4 out of 24 (there are not more than 10 questions). If I refresh the page, it will go back to displaying 4 out of 10.
I’ve done my round of troubleshooting and I don’t find any obvious ‘duhs’. I cannot even create a successful workaround. So – naturally – I come to the forum to do some rubber-ducking / group therapy / check if any of you have experienced this.
Under the hood
Here’s the (simplified) following workflow:
user clicks button
client creates a new Assessment
schedules a backend workflow to populate that assessment
Backend workflow:
calls the Bulk API to create new answers linked to that assessment
Client side:
widget has a native Bubble do a search for looking for the newly created snapshot and counts the answers.
Correct. Yesterday I was bulk data creating about 30,000 things - for each of those things, something was upserted to Pinecone (about 1,000 per minute). Whilst it was happening, Pinecone always showed about 1,000 more vectors than I had things in DB (which should always be equal). I tore my hair out trying to work out why but ten minutes after the process finished, the Bubble counts caught up. So, the things are created, but not indexed/updated (even a page refresh wouldn’t bring the count correctly up to date - it literally required minutes for something to be happening on Bubble’s side.
I thought it was a me thing yesterday, I’m sort of glad to know it was a Bubble issue!
Good news / bad news scenario there. I started reading your post thinking “yes! Someone more intelligent has encountered this problem and has a solution for the problem”.
Nope. I guess we can drown our tears on the same bottle of whiskey .
Could you schedule API workflow on a list and create one thing in that API workflow? Less WU efficient but perhaps uses a different Bubble system to add things to DB than the data API, and the speed would be very similar now the performance of scheduled API workflows is good (about 60 per second in my experience).
how about using the returned data from the api call? any reason why you do the call in a backend workflow? you could update the count by merging the returned list of created ids and the list of the search.
Dude… you’re touching on a major blindspot of mine. I’ve been executing most crucial workflows in the backend. For way too long; it has become a habit. I guess I hang out with too many backend devs. This past week I realized that Bubble does – what I guess – are optimistic updates on the client side.
I’m gonna try:
move the bulk call to a schedule on a list then (George’s suggestion)
then I’ll try to make the call from the frontend
(it’s a big workflow, with lots of checks and side-effects like notifying admins, creating CSVs / XLSs, etc.)
right, if you have other business logic it makes sense to offload it to the backend. my workaround was in the case you only had the bulk create.
if it’s not a huge amount of data created and you don’t mind an ugly workaround you could have a “supporting data type” for the operation with a field as a list of things. Before everything create the support data, so you can get realtime notifications for it, pass it to the backend wf and store the returned list of created things in it (then schedule a cleanup). I never had this specific case so I’m not sure it will pick up the list in realtime.
I don’t doubt you’ve been seeing this, but I think it’s a separate issue.
I see this behaviour but it seems to be on Chrome (never seen it on Safari). Things that are ‘CRUD-ed’ in the backend (especially deleted) aren’t always updated in the front-end. Trouble is, I can never replicate it so not much hope of a fix.
What Rico’s speaking of is an issue with the data API in particular which has similar consequences, but probably a different cause.
Yes, aware this is a separate issue and makes use of the bulk API, but having seen this issue just for the first time today, thought they could be related.
I’d presume that Bubble is making use of similar infrastructure to execute the workflow action ‘make changes to a list’ and the bulk API?
In terms of you seeing similar behaviour - can also confirm this is Chrome. Have not tested on any other browser. We can reliably recreate the issue, so hopefully the bug report helps us make progress on this
.
