Iterating list items using javascript to bubble

anon44922401 · November 6, 2018, 8:53pm

The current project I am working on is a on-boarding/training app. The app will allow administrative users to create courses, break up the courses into modules that consist of a single video, create quizzes per course that consist of a series of questions with multiple choice answers. Normal users will have the ability to “register” for these courses and ultimately take the quiz for each course that they register for.

I ran into some trouble while creating the processes that create the registered course, its modules, the courses quiz, the quiz’s questions, and the possible answers to the questions.

I envisioned the following steps:

Based on user Course selection, create the User Course
Iterate through the selected Course’s Modules and create User Modules
Take the Course’s Quiz and create a User Quiz
Iterate through the Course’s Quiz’s Questions and create a User Question
and finally, iterate through each Quiz Question’s Answers and create corresponding User Question Answers

Goals:

Stay within the Hobby plan. (I wanted to have a pilot to demo first before having the client dish out $$)
Seamless UX

I hope to explain the problems I came across, how I resolved my issues, and hopefully get some feedback on how others have solved a similar problem or how they may go about this differently.

Problem #1:
No out of the box bubble action exists that allows you to loop through a list. I worked with other rapid application development platforms like Mendix whose visual workflow/logic builder has this feature.

(It would be a great feature to have in bubble. Just putting it out to the universe in hopes that the bubble gods here my plea It’s all love for bubble from me).

I then I found a video about running a scheduled api event on a list by @romanmg Big shout out to you btw. Extremely resourceful with the information you have put out there.

Thinking that this may solve my problem, I whipped up the workflows to handle:

Iterating through the selected Course’s Modules
Iterating through the Quiz’s Questions,
and iterating through the Question’s Answers

Problem #2:
It seemed like it would all work, but I didn’t realize that you have to be on a paid plan to access the scheduled api features. Given that my goal was to stay on the hobby plan for the time being, it was time to move on.

I then opted to program a way using the Toolbox plugin’s javascript to bubble’s feature to iterate through my lists.

On my Course page, where users can register for Courses, is set up with the following:

A repeating group to display available courses to register for
A button within each Repeating Group cell allowing the User to Register for the given course
A custom state for the following
Module list to iterate on
Questions to iterate on
Answers to iterate on
The created User Course
The created User Quiz
The current Question
The current User Question
Javascript to Bubble Elements to iterate on for the following:
Modules
Questions
Answers
(3 total JB elements)

The Register button kicks off the following workflow

The jist here is that I create the User Course, set the created User Course as User Course custom state so I may access it in other workflows, set the Module list custom state to the list of Modules belonging to the Course, run a javascript action passing the Module list custom state’s count into the javascript to bubble’s element I set up for the Modules, and finally creates the user quiz portion of the user course.

In my use case, my function in the JB element for my modules is bubble_fn_userModule. This JB element is set to trigger an event. In the Workflows, I have an event set up to run for that JB element only when its value is greater than 0.

The javascript action in the workflow behind the Register button passes the count of the custom state module list to the jb element like so: bubble_fn_userModule(course_page’s ModuleList’s:count). Because the count is a value greater than 0, the javascript event for the JB element is kicked off.

This workflow iterates through the list of Modules and for every module, creates a User Module and associates it to the User Course custom State that I use to store the User Course created in the first workflow. The last action decrements the javascript to bubble’s module’s value by 1.

To access the data I need to pass to each created User Module, I set each field equal to course_page’s ModuleList’s:item#This JavascripttoBubble’s value’s(corresponding field I need to “copy”).

The break down:

Essentially you access the current iteration’s “thing” through the stored list’s item number that’s equal to the Javascript to Bubble’s current value.

For example if my list of modules consists of 4, when my javascript to bubble event is kicked off, the starting value is 4. So the stored list’s item # when I start is 4. Thus, the workflow will execute on the 4 item in my list. When the javascript to bubble’s module value is decremented, the workflow kicks off again only this time on the 3 item all the way down till there are no more.

@keith explains iterating using these javascript to bubble elements in a much more concise manner than I am in this post. I am trying to add a use case scenario here with my explanation.

The last step in my first workflow creates the user quiz, user questions, and user answer components of the registered course.

The Questions are iterated in a similar fashion where I have a JB element set to trigger an event only when its value is greater than 0.

In a run javascript action within the Create quiz workflow, I pass the Question count of a custom state list to the JB element for Questions thereby kicking off its workflow event that creates a User Question for each Question in the stored list.

I happen to store the current User Question in another custom state to associate it as the parent of User Answers created in a separate Javascript event workflow.

Problem #3:
In my create question javascript event workflow, I pass the Answer count’s value of the current Question to the JB element for Answers. The very next action I have is a Run Javascript action that decrements the javascript to bubble’s questions value by 1.

However, I didn’t necessarily want to decrease the Javascript to Bubble’s Question’s value until all the User Answers have been created for the current question at hand.

I thought that setting up a Only when statement would satisfy this requirement. So I set the Run javascript action to Only run when the Javascript to Bubble’s Answer’s value is 0. This didn’t work.

The workflow processed only once and my end result was everything I needed except that only one question was created and its corresponding answers.

I tried setting an “Imported” attribute on my Answers that would tell me if an Answer had been processed or not. I then tried to set the Only statement to run when all Answers had a “true” value set on Imported. My end result was the same as before with only one question and its corresponding answers created.

The remedy:
I created a Fourth Javascript to bubble element on the Course page set to Trigger an event. The event would only trigger when its value was set to 0.

In my create Questions workflow, I used a Run javascript action to set the value of the fourth javascript to bubble element to the Answer count of the current question. In my create Answers workflow, I decremented both the JB element Answer’s value and the newly created javascript to bubble elements value.

When this new JB element reached zero, it executed a run javascript action that decremented the javascript to bubble’s Question’s value by 1 thereby kicking off the next Question and Answer set.

This worked exactly as I expected except that it takes about 25 seconds!!! No bueno…

So, I achieved my goal in not switching away from the Hobby plan, however my second goal is compromised in that the UX isn’t as seamless as I’d like it to be.

I thought of scheduling a custom event sometime in the future however, I don’t know of a way to easily store the many parameters I am using to make the proper associations within each workflow.

I know that this project will move forward to a paid plan at some point and maybe the scheduled API workflows will work out better for me.

All in all, I hope this really long post may help someone in some fashion.

Again shout out to @romanmg and @keith for all the useful content out there.

Anyone have any similar experience?

How about them scheduled API workflows?

Thanks,

JohnMark · November 6, 2018, 9:02pm

Sincerely I’m a bit confused. At the end, it is a Tips or Help you needed?

anon44922401 · November 6, 2018, 9:04pm

Honestly, I really don’t know.

I would like to know how others are using scheduled API workflows and if what I did was way too complicated.

JohnMark · November 6, 2018, 9:08pm

You have the answered of ‘complicated’ when it took about

Scheduled API is on server side, so no lost of speed for UI. It worth the 16$ per month at least for one month, time for you to complete it. After, downgrade to free. My suggestion.
Maybe change TIPS for Question?

andrewgassen · November 6, 2018, 9:08pm

I use scheduled API workflows for all looping needs. It’s not ideal, but seems like it may be a better solution than your workaround above.

anon44922401 · November 6, 2018, 9:10pm

Point taken.

This was more of a go through the pain to learn type of thing otherwise known as my stubbornness.

romanmg · November 7, 2018, 2:05am

Happy to have helped!

Totally understand wanting to stay within budget, but whenever you’re ready to commit to a plan, you’ll have a much more manageable workflow if you do all of this in API Workflows. Also, Bubble recently opened the ability to run API Endpoints recursively, which might be something to look into: [New Feature] Scheduling API workflows can now be done recursively

keith · November 7, 2018, 2:34am

Just a general comment: I’ve not gone through the entirety of the original post, but I’m gratified that someone was able to implement the in-page recursion technique.

The problem with API Workflows on a list is the time they take, particularly on lower-tier paid plans. and the problem with recursive API Workflows is the even LONGER time that those take. Those two techniques are suitable only for non-realtime-ish processes.

If you have some iterative computation (one that’s relatively lightweight – e.g., on the order of single digits or a couple dozen recursions) and you need it to be close enough to real-time that a user won’t completely lose his or her mind, API WOAL and recursive API Workflow are straight out the window on lower tier plans. Options from slowest to fastest are as follows:

Recursive API Workflow – can be many minutes even for a short list
API Workflow on a list – can be many seconds to a small number of minutes for a short list
Stand up your own external API to do the computation (send it the list of key things and let it ping Bubble recursively to retrieve needed values in an iterative way and then send then computed value back to Bubble via the API Connector). Yes, this is faster than an API Workflow on a list! – can be 5x-ish faster than #2. (Actually, the time for your microservice to complete may in fact be in the millisecond range, but there’s overhead in receiving/ingesting the data on the return-to-Bubble side.)
In page recursive workflows using the JS to Bubble technique I describe elsewhere. – can be 10x or more faster than #2 and is generally faster than #3.
Pure JS in the page. – can be extremely fast (orders of magnitude faster than #4), but computations that require lookups in the bubble database are not always possible to perform.

At present, I’ve not played around with server-side JavaScript (which seems to now be enabled in Toolbox), but I expect that would be somewhere between 3 and 4 on this list, depending on many factors.

The above relative comparison is based on my own experience with designing a method for computing total cost of a hotel-type stay where each night’s price must be individually looked up (on the basis of the date) and then summed and then similar iterative lookups performed to compute related fees.

I use method 4 in my app. You can go visit https://grupz.com/vacation-rental-booking to see this in action. You can select dates in the booking widget and see that most of the time rate computation is more-or-less instantaneous. For a longer stay, you may in fact see the widget totalling up the numbers (it’s actually kind of a fun effect when that happens).

What’s happening is that the selected date range is turned into a list of dates that we can iterate over in workflows. The iterations are triggered by JS to Bubble events using the method I’ve described. Inside of each of those iterations are various lookups to the Bubble database that retrieve the rate for each night (and rates in my system can be quite complex – it supports individual nightly rates/ranges of dates with a given price, day-of-week rates, and default rates and, for each night, needs to establish which type of rate is in effect.

Doing this in a workflow on a list takes several minutes (which is to say, it might as well not even be possible).

keith · November 7, 2018, 3:39am

And a quick clarification: there is an additional iterative process involved in what I describe.

We start with a date range (start and end of stay) and must create a list of date objects from that by looping. Doing that is, in fact, impossible in vanilla Bubble.

Some use Toolbox’s List Item expression to accomplish tasks like this, but I use a Run JavaScript, and the Moment/moment-range libraries to do this as it’s just one line of code and essentially instantaneous for any real world date range. (Also I need moment anyway.)

So I’ve done a couple of versions of #3… one that just pings an api to do the date range to list-of-Dates conversion, and another to do the whole shebang.

Getting a list of dates like this via API takes a couple seconds (even though the process itself will complete in milliseconds), but due to api overhead, there’s a delay. So it’s just way better to do it in the page. Even slow mobile phones can do this.

mishav · November 7, 2018, 4:22am

@anon44922401 Thanks for giving an overview of your app.

I think that the reason the iteration takes so long is that inside the loop you have steps for Create Thing and Modify Thing. When Bubble processes those in succession on a page, the first few - about five in my experience - are done quickly, then the remainder are rate limited, slowing down to about 0.7 of a second each create … exact timings vary hugely.

If you take a typical quiz creation, how many Create Thing steps are involved?

API workflow isn’t an easy escape from this, it can process even slower than the page workflow, although frees up the page.

Another possible cause of slowness is interleaving processing with waiting for results of database interaction, although its not clear to me how much the app has this going on.

A possible approach to making it faster:

Reduce the number of creates, by packing more records into a single record. For example, all the answers for one quiz stored together in a single text. A disadvantage is complexity of data structure, but unpacking can be done at speed in javascript, and the complete structure can be created as a leisurely background process if needed for searching and reporting.

anon44922401 · November 7, 2018, 2:31pm

Thank you all for the feedback.

I think the biggest takeaways for me and hopefully for someone else that may read this post is that the Javascript to Bubble iteration technique may be a solid choice if:

You intend to keep your user on the page
You have one “small” list to iterate over at most maybe two lists…?
If you want to stay in Hobby plan

Its not a solid choice if:

You have many “levels” of iteration like in my case
You want to prioritize UX (don’t want to keep your users waiting)

My project is moving to a paid plan so I am most definitely going to explore the scheduled API on list features as well as what @romanmg mentioned regarding scheduling api workflows recursively.

I do like the idea that the page is freed up using those options.

I hope to have an update soon about my progress.

sudsy · April 29, 2019, 11:05pm

I stumbled upon this thread while researching DB performance issues. Any updates or further thoughts on your project?

keith · April 30, 2019, 1:13am

@sudsy, regardless of how you do iteration on the page, firing off make changes to/create new thing operations will throttle shared plan apps pretty quickly.

It’s easy to experiment with, but cannot be improved. (On the server-side, you cap out when you cap out, ya know?)

Purely page-side iterations are of course limited only by the client device’s processing power.

But (as an aside), most discussions of Bubble “slowness” involve doing silly things like buffering up ton of states that then need to be blasted into the database. This is simply bad bubble practice and should be avoided. I simply do not get why some are resistant to updating the database AT THE MOMENT the change should be made. (Doing this this way - the correct way in Bubble - avoids such traffic jams.)

And, of course, I have a NEW Bubble iteration method that’s somewhere between doing things in pure JS and iterating by sending triggers from Toolbox. It’s just only available thus far in Calendar Grid Pro (for dates only).