Importing JSON to a DataType

Hi Folks,

getting a bit stuck importing a longish (4-5K) list of data into a data type (creating new records of that type)

I have an API that fetches the data as JSON (Im doing it this way, as importing as a CSV has 100 record limits)

The API is set to be used as Data and grabs the body as a list (and I can refer to it in the API’s) it looks like this

    "body": [
        {
            "depotId": 37,
            "carId": 200,
            "jobId": 75295,
            "operatorId": 343,
            "actualDuration": 320,
            "actualStartTime": "2022-09-01T00:35:00",
            "actualEndTime": "2022-09-01T05:55:00",
            "bubbleUid": "Redacted :)"
        },
        {
            "depotId": 36,
            "carId": 188,
            "jobId": 75296,
            "operatorId": 434,
            "actualDuration": 625,
            "actualStartTime": "2022-09-01T03:09:00",
            "actualEndTime": "2022-09-01T13:34:00",
            "bubbleUid": "Redacted :)"
        },

From what I understand I should setup a backend workflow to call the import API then I need to parse the body into a list and setup a recursive workflow to load each item into their own record.

Im good with recursive workflows, Im stumped on how to format the list to import it.

It seems counter productive because I already have the Array in JSON but do I format the list as JSON and send it on to the next workflow? And do I set up that workflow as a list of text?

Thanks in advance!
John

You shouldn’t need to parse the data into a list. It should create an API data type once you initialize the API call. Then from your workflow you can use the “Results of Step x” which will return your data type that you could access ApiDataType.body.first-item.depotId. So you would need to run a workflow on a list of ApiDataType.body and for each item in body (inside the workflow) you would “Create a Thing” by setting each respective property.

Hey @bubble.trouble thanks for responding . Just stepping through this:

  1. It should create an API data type once you initialize the API call. - Done

  2. Then from your workflow you can use the “Results of Step x” which will return your data type that you could access - can also do this

  3. So you would need to run a workflow on a list of ApiDataType.body and for each item in body (inside the workflow) you would “Create a Thing” by setting each respective property.

two things on this point

  1. the returned array is around 4,000 records, I thought recursive workflows were better because Bubble can fall over with a list this big and you never know where you’ve been left off.
  2. If the list approach is fine, how is the “list to run on” setup? I can work on a list of data types but they are currently a list of JSON bodies
    image

Thanks again for helping,
John

  1. Yes well this is probably true but the point remains, you need to loop over the list one way or the other and the approaches are slightly different but similar.

Question: Is this a one time import or do you need to run this frequently? The reason is that these workflows recursive or otherwise are limited in how quickly they run so iterating over 4000 items regularly could bring your application to a grinding halt or could hit your app capacity limits causing other problems in your app.

  1. Without knowing more its hard for me to answer this question., I don’t know why you have a list of JSON bodies and not data models?

The way you guys are talking about should work fine. As an alternative, I have also done something similar using this plugin to upload csv files over 100 rows at a time. You could do thousands without an issue using a recursive backend workflow.

Hope that helps give another option. Either way should work just as good. :+1:

thanks again @bubble.trouble & @J805 ,

You’ve both been great help. I’ve got this working on 250 records, just tweaking everything before I run a full sized import.

to answer your questions - It’s a regular thing (couple of times a month) that I need to run it.

Bubble is the front end for a rostering solution. I need to send the list of shifts to my backend - do a bunch of calculations and bring it back into bubble for presentation. This is managing the last step of this process.

the UID is my link to the original jobs upload, the reset is then calculated at import.

I use recursive workflows as I can set a delay between each record which keeps the processing to under 40% giving users headroom to do other things. I also use @eli 's excellent CSV uploader as you suggested in another area of my app but I was fixated on the 100 record limit.

I’ve since fixed my original issue (importing the data) using @jacobgershkovich’s video How to Save Data From an API to Your Bubble Database | Bubble.io Tutorials - YouTube (thanks Jacob!)

I then run CSV create on the input to build a load file I can set through.

I’ll come back with more detail when I have it working smoothly. This is quite a complex backend workflow.

John

1 Like