I am somewhat stuck in recursive workflow hell and hoping for someone to save me.
I am receiving data from an API I created. The API returns a simple list:
For each date, and for each lesson in a learning platform, how many users were active. There are 6 lessons. So for each date there would be 6 rows, each with an identical reportDate field a unique lessonID field and a number of users field. In addition to that I added a row with an empty lessonID field and the total number of users (all active users summed together) for that date. So for each date there would be a total of 7 rows. Simple enough.
Now on Bubble I want to update a local DB table.
So I have a backend workflow which first deletes all records in that table, calls the API and then schedules an API workflow, passing each field as “each item” like the manual says for recursive workflows:
Then each item is entered into the database exactly like the recursive workflow manual says, by creating a thing for “first item” and then scheduling another workflow (itself) on the next thing, removing the first thing and so on.
The dates are grouped into one row for each date (instead of 7), the number of users is summed somehow as well (the numbers are incorrect) and the lessonIDs are shown once and t hen are empty.
I’ve been around this for a while and kept making the API result simpler and simpler for Bubble but it doesn’t get simpler than this. Weirdly it looks like the condition of “only when” for the rescheduling of the recursive workflow has an actual effect on the data. So when I say “only when the date field count >0” the data is different.
I’m somewhat disappointed I can’t just use “Schedule API workflow on a list of things” for this which was very straightforward, but it looks like any amount of data that is more than tens of rows is too much for Bubble to handle in a backend workflow and a recursive workflow is needed. I don’t list sending the three variables as three different lists and I think this is the origin of the issue, but… I’m a bit lost here. Any words of wisdom?
No matter what make sure you have a condition on the step that scheduled the next one, or else you will create an infinite loop and make the WU gods very angry. If you’ve made any infinite loops check the scheduler on your app and cancel anything.
The steps all seem okay, have you tried checking the box to ignore privacy rules? It’s possible it just can’t find the database entries. Also any reason why you aren’t just doing Make changes to a thing and updating an existing record instead of trying to delete + create?
Actually seems like you’re not trying to change anything since it’s a new date, do you even need to delete anything?
You could actually ditch all the recursive stuff and use the Data API to instantly create those 6 things
Hi Tyler, thanks for the reply.
I’m not using “make changes” because I’m trying to follow the recursive workflow instructions to the dot. However I agree that there is no need to delete all past dates when all I want to do is add a a new date.
That being said, I do need to do it one time for all past dates and I have never been able to complete this task, when it should work. Once I have this done, I can simply limit the date range to the last date knowing that if the data is corrupt I can extend the date range. But this is all wishful thinking. It should work and it doesn’t.
Some weird behavior which can add a hint:
When there are many records in the local table, the delete action takes time and the insertion to the db doesn’t happen at all. I need to manually run it again. When I do run it when the table is empty the data is corrupt as shown.
Your suggestion to “push” data by using the data API is interesting and will check it out. The only downside is that I need to create a special process on another server for this which I’d like to avoid. But it looks like a possible solution. Thanks!
Don’t bother deleting just let them pile up, plus you’ll have some history if you ever need to go back and graph # of users per Lesson.
Yep, figure out the functionality first and we can nuke the bad data later on from a known date (or nuke it all if it’s just your Dev data thats bad)
It’s not a separate service or server, your Bubble app has an endpoint for the Data API . It just needs to be configured in the settings to be enabled for your specific Thing, and privacy rules need to be set to yes, and a few other things.
You have it send an API call to your own app and it instantly creates the things you want in bulk. I can go into more detail but read over the docs they might be self explanatory
Yes Bubble should make this a built-in action we can just use but unfortunately they don’t have it so we use the API Connector
I think this is pretty smart. So let me describe back to make sure I understand correctly and for posterity on this forum.
You’re saying:
Enable the Data API with proper permissions etc.
Create a call from the app itself, to itself, through the data API to insert in bulk the needed data.
I will give this a try now. I can imagine some security vulnerabilities by not doing it internally but I haven’t yet looked deeply enough at the permissions of the Data API. This seems like an interesting workaround. Thanks!
Just make sure your app Settings → API tab you enable the Data API for just the datatypes you actually are going to be using this for.
Then in the Data → Privacy tab under those specific datatypes only enable these as needed, so Bulk Create endpoint just needs the Create via api enabled.
Your app will require authentication for these calls so you probably have it set up already but then create your API token in the Settings → API tab (and obviously don’t share this with anyone)
This was really helpful for me, thanks One thing I did read in the docs was that token’s grant admin access and they ignore privacy rules. As an admin - Bubble Docs so maybe only for other authentication methods.
Thanks Tyler, this may be the right solution even though I have not yet gotten it to work.
The issue is with the way data is delivered to the data API.
“For each item” doesn’t work here since it will then list for each field instead of one object per row. Luckily my own API data should be just fine here since the formatting is identical (it basically passes JSON objects with the same fields). So I was hoping to just use “each item’s raw body text” from my own API. But the result of that is this:
The actual result is verified to be fine (see original first post).
So, how would you pass these fields from one API to the data API?
By the way, on the way to getting this done I’v discovered quite a few Bubble issues:
Enable Data API can be unchecked while you can check what data type the Data API is exposed to, making it seem like it’s on when it’s actually is not.
Documentation says that if you use a token, no need to use rules for the data types as you wrote but this was still a necessary step even with using a token
Documentation for bulk update says in the headline that bulk base URL is “https://appname.bubbleapps.io/api/1.1/obj/bulk” which is confusing because it does not include the typename, but a closer read below reveals that the headline is incorrect and in fact the URL is /typename/bulk
Inability to debug backend workflows makes it all more fun
Take note of the Content-type text/plain , the “User as” is Action, and the data type is Text. The endpoint takes a text format for some reason, they are JSON objects but separated by newlines for each object…
Then put some dummy data in parameters so it can initialize properly. So the datatype parameter put “lesson” or whatever your datatype name is (lowercase one word version of it) then in the body parameter do something really simple like {"activeUsers":1}
Then if that initializes properly put the action in your workflow, type the datatype you want, then for the body do your User daily activity API call so you get the list, but do :formatted as text and create your JSON object referencing “This [api thing]'s [field]” wherever needed. For your example you shouldn’t need the JSON safe operator I needed. Then just press enter in the delimiter so it separates each object with a newline.
I made mine a recursive bulk-create because I needed to do more than 1000 things total, so it’s a bulk create in batches of 50, but I think you won’t need that so you don’t need the :items until #50 or the condition on that action like I do.
Yup that did it. I was already working on “formatted as text” but the whole data as text stuff would probably draw this a few more hours for me so thank you for your help!
I can’t say this is elegant for Bubble, especially since “API workflow on a list” would just be perfect for this, but it is very elegant as a workaround. Thanks again!
Also you’ll see in my screenshot I have two API calls, one says (test) because it has the version-test/ portion in the URL so it makes the things in the test database. Once you get it working copy the entire call but remove version-test/ so it makes them in your live database.
Then put both actions in your workflow but the proper condition on each either App is live version is "yes" or no.
Supposedly doing front end list operations are coming to Bubble this year so we don’t have to resort to API calls and loops.
Well, one last thing I guess. All this experimentation was done on the front end while it actually needs to run, scheduled on the back end. Moving the workflow to the bank end and it does not work. Dummy data to the “body” element works, but as soon as I connect it to the other API with all the fancy “formatted as text”, and nada. Empty table. Same code. Any idea why there would be a difference?
@tylerboodman One difference that I am noticing from your example is the fact that your workflow had a “(path) datatype” field to enter and mine didn’t.
I wonder if you are doing this differently.
On the backend workflow, I call my own API and then run a plugin action for my data API. The window does not have a “(path) datatype” field, just to enter the body.
I put [datatype] in the URL so it can be dynamic because I use the Data API for multiple datatypes. For you if you just want to hardcode “lesson” you probably have that in manually already.
I see. So it should work. Unclear why it doesn’t but I might have to give it a rest.
The amount of time spent on this trivial task, really, kind of negates the whole idea of using Bubble honestly. I’ll report if I find a trick to solve it. Thanks again!
Once it is setup it will be all good, screenshot how it is setup now in the backend? Is the API call actually retrieving a list of things? One misplaced quote will break the JSON formatting
In the body section you’re referring to the Result of step x from your first API call to get the list?
It works as described in the thread. Small adjustments made that are insignificant.
Adjusted the workflow to receive automatically overlap past 2 days, so only 3 days are treated and not the entire dataset (because of the way Bubble displays time zones which I’m saving for later). Bottom line is that this is the solution for handling larger datasets. Thanks again.
One thing I did read in the docs was that token’s grant admin access and they ignore privacy rules. 
