Passing 'List' between backend workflows

[gregsheen23]

Hello,

TLDR; I am taking a text input from a user, converting to JSON (this part works fine), and using JSON manipulator plugin to create new objects in my database using backend workflows. The problem I have is the plugin is giving me a ‘List of JSON texts’ but the schedule API workflow does not like this format.

Key question: What list format is needed when passing a list to a scheduled API workflow? How can I convert my list to the required format?

My entire backend workflows:

User inputs text and clicks ‘Save’
Text is converted to JSON by chatGPT API (this works well)

Store the original text and JSON in my database

Use JSON Manipulator to extract part of the JSON (only ‘Experience’ objects)

Pass this list of JSON texts to another workflow to process it

Read each field of the JSON

Create a new object using the extracted data:

Problem:

The object the second workflow is creating has NULL fields:

It is not extracting data from the JSON.

I have done some testing and believe the problem is happening when the ‘list of JSON text’s’ is passed from the first workflow to the second workflow. The second workflow is not seeing it as a list, is it seeing it as a single text object.

It takes all of the ‘Experience’ objects successfully, but the second workflow is treating it as a single text object rather than as a list of texts.

I have tried a few things to resolve this but none have worked, these include:

Using ‘list of strings’ instead of ‘list of JSON texts’

Stripping the ‘[’ and ‘]’ of the JSON before it is passed to the second workflow:

Any help would be appreciated, been stuck for days on this.

Thanks,
Greg

[guillaume.raballand]

@gregsheen23

Pass this list of JSON texts to another workflow to process it

Before starting, I would like to say that I prefer to work with JavaScript with the great Toolbox plugin and “run javascript” action for frontend workflows or “server script” action for backend API workflow, instead of Bubble to manipulate JSONs because Bubble expressions and Types can have a lot of unexpected side effects with JSONs … But if you prefer to keep using Bubble expressions and type, here are some things to check:

1. Have you tried to select the actual Type of things returned by the plugin in action of step 4 “read json Get list of Experiences” instead of basic Type “text” ?
   it could be named like “JSON Manipulator’s JSON” or something similar (refer to plugin documentation) in the Type of things dropdown in Bubble Editor for your step 5 in the 1st API wf AND for your input type “experience_item” in your 2nd API wf

2. Also you can have issues trying to pass JSON as texts with double quotes (") inside the texts for JSON values that can break the format of the JSON, and then you get null values instead of text.
   Valid text for JSON: “This is a valid example without quotes”
   Invalid text for JSON: “This is not a “valid” example because of quotes”
   => you can try to escape these double quotes using :find and replace operator or :formatted as JSON safe (be careful with this one that automatically adds the double quotes at the beginning and end of the string)

3. Also, be careful about lists of texts stored as one text in Bubble that can add extra spaces before and after commas !
   => [one, two, three] can become “one , two , three” => you can use the :join with operator to be sure to know how list text items are actually joined in texts

4. Also, be careful about how Bubble expressions work: in your 2nd API wf step 5, the :find and replace operators are applied to each list item element and not to the list, they actually remove the “[” and “]” from the items inside the list (Bubble has its own format for lists, they are not treated as usual JSON) so they are useless here…
   …:each item…
   The :each item prefix is added to a List operator when the operator acts on an item in the List rather than the List itself. This prefix does not change any functionality, it is merely there to clarify the intended behavior of the selected operator.

Anyway : after running these API workflows, inspect server logs for each action step to see the input value and try to identify which steps are returning correct or incorrect values.

I think you should understand better how Bubble deals with lists and the different operator on lists and their elements because you are using them in the wrong way to get the result you expect…
=> read Bubble documentation for list operators more carefully

[guillaume.raballand]

Read also the documentation and play with the demo of the plugin JSON Manipulator to understand how it works and how to directly extract one specific value using JSONPath syntax:

NB: API workflows in Bubble can also get JSONs as input, but only read the 1st level (key-value pairs) and not nested JSONs inside the JSON, so maybe you have to use a plugin like this one if you need to extract nested objects.

As an example:
Correctly interpreted JSON by Bubble to extract values with basic type like text or number:

{
"key_1": "value 1",
"key_2": 2,
}

Nested Object and values at further depths not retrieved by Bubble API inputs:

{
"key_1": "value 1",
"key_2": 2,
"object_3": {
  "object_key_1": "not retrieved",
  "object_key_2": 6
  }
}

[mitchbaylis]

Just need to reference the json object in your api workflow to then explode it into the fields to map from. You can do this for the nested jsons as well - just needs more api workflows for each level.

[boston85719]

@gregsheen23, @mitchbaylis and @guillaume.raballand offer some good ideas on how this can be achieved.

I’ll offer some other method that is optimal.

Firstly, don’t both with the schedule api workflow to save the values to the database. Instead make sure your key names in the json are exact match of the data type names in the database data type you wish to create and simply take the list of text from json manipulator output and send those values through an API call to your app for a bulk create operation.

So take the json mainpulator list of text (that is json formatted) and then add operator to join with arbitrary text which is just a new line (ie: press return on keyboard)…in my screen shot the find and replace is for my use case, so not of relevance for yours.

This method allows you to create as many data entries at once as required based on the number of outputs from the JSON manipulator.

So that is how you can create the items in your database.

But, I’d like to discuss the setup of processing text. You do not need an API call to ChatGPT to do the extraction and JSON creation. Instead you should have a server script that does this for you. I’ve used chatGPT to help me create server scripts for this purpose and it works well. If you did this properly you’ll eliminate steps 2, 3, and 4 from the process csv workflow series and replace it with a step 2 that is run server script and a step 3 which is run API call to your app to create the database entries. You might even get away with eliminating step 1 as well since it might not be essential.

This will also it everything is done as per my suggestions, eliminate all steps in process_experience workflow.

If you follow my suggestion, you will have basically only two steps, 1 is server script to process the csv and the second is api call to your bubble app to bulk create the data entries. Based on what I can see in screen shots, this will reduce the entire process from 9 steps to 2, saving on average at least 3.5 WUs per run if each run only processes one data entry. If the process were to run 2 data entries the savings would be around 5.5 WUs…

If you are already familiar enough with chatGPT to get it to process the CSV via API, you should be capable enough to get chatGPT to create the server script for you…if not, you will still at least benefit from sending the json list as an API call to your app to bulk create rather than attempting to send it in via schedule backend workflow on a list.

[gregsheen23]

Thank you for your helpful responses and alternative methods. I also reached out to the plugin creator who noticed a very simple fix which ended up working.

The core issue was that I was passing $.Experience to the second wrkflow, which caused Bubble to treat the entire list of JSON objects as a single text item. The fix was to change it to either Experience or $.Experience[:]. This adjustment ensured that the second workflow received the list as individual items rather than a single block of text.

After making that change, the second workflow processed each JSON object correctly, and the database entries were created with the expected values.

@boston85719 I appreciate the alternative suggetion for bulk creating entries using a single API call. This approach is definitely more efficient, especially for larger datasets, and I’ll probably implement it in future iterations to reduce workflow usage and optimise performance. It also makes sense to simplify the process further by using a server script instead of relying on multiple steps.

@guillaume.raballand Thanks for the additional details and example! I appreciate the heads-up about how Bubble’s API workflows handle JSON inputs, especially the limitation of only reading first level key value pairs. That clarification makes a lot of sense in explaining why I encountered issues with nested objects.

Thankfully, the JSONPath adjustment fixed the immediate issue, and I’ll definitely take your advice to explore the plugin’s documentation and demo more thoroughly to better understand its capabilities.

@mitchbaylis Yes exacly, my plan is to duplicate this workflow for each level.

Thanks again for the help