[New Feature] Native JSON parsing is now available in the API Connector

[paul.weill]

@nick.carroll

Now, the implication of this is that it may involve re-mapping fields in the UI or workflow if you re-initialize the call. Question to those impacted - if it was relatively straightforward to re-map, is keeping the nested shape of a response desirable? or is it still better to have the option of keeping parts of the response flattened?

I think the new nested parsing is better for new API calls, the mapping is clearer and more structured.

The main issue is with existing API calls. A lot of users may re-initialize a call for one reason or another, without expecting the field selection behavior to change. In that case, it can create errors and require quite a bit of manual remapping.

In my case, it took around 1.5–2 hours to remap the affected UI/workflow expressions, comparing the live version with the old flattened expressions and the dev version side by side.

So yes, I think keeping the nested shape is desirable going forward. But for existing calls, it would be really helpful if Bubble could either preserve the previous flattened behavior, or automatically apply the new parsing to existing expressions wherever the API call is already used in the UI/workflows.

Thanks !

[julie01]

@nick.carroll This has significantly impacted my manual response API calls (I’m using the Xano connector). Any change made to an API call, even after resolving the resulting errors, leaves the call in a broken state with no way to revert to the previous state. I logged a bug on 3 June and am hoping for a swift fix. Trying to support a production system also with a major launch next week is proving very stressful!

[paritosh.mehta19]

I agree the direction with cleaner support for nested structure makes sense, but if this can be auto mapped correctly for the existing API’s where we have the flattened elements structure after re-initialization on the UI / workflows, that would be fine. Alternately, make this an option for us to choose. We have 100’s of custom API’s and we need to re-initialize these regularly - and re-doing this mapping will be fairly challenging and require a lot of testing effort too.

[boston85719]

Keep flexibility, allow for flattened if user opts for it

[loic1]

I completely agree with @boston85719 .
One of my apps is going to break because everything is built around the Data API.

How can you make changes like this without giving us any option to choose between flattened and non-flattened data?

We rely on workarounds in large apps, and that represents months of work. Did no one even consider this possibility?

Please add a way, as soon as possible, to choose between flattened and non-flattened data, or at least a setting to enable or disable this system. As it stands, this is just not possible.

[georgecollier]

I’m so confused, what’s the issue? There surely no breaking change to the data API?

[loic1]

My entire app is built on data mapped through the API Connector.
I have a lot of flattened objects.
If I reinitialized : it all breaks.
Plugins that relied on the old flattened system are broken as well.
I would also have to rework all my filters (which were based on flattened data, so I’d potentially have to switch them to advanced filters). This changes everything and represents a huge amount of work that we’re taking full force.

[boston85719]

@georgecollier

I’m so confused, what’s the issue?

So a flattened object is where nested JSON becomes flattened …imagine a json structure of contact, it has keys of name but a nested object of address that has keys like city…a flattened object would have a key like contact.address.city

A nested object would have address as a separate object with keys of city.

They map to fields differently…so if a field was contact.address.city and then api call reinitialized it to nested object structure all data references in the app to the field of contact.address.city are broken, all workflows, data sources, everything needs to be mapped to a new field.

[nick.carroll]

Hi everyone - apologies for the confusion. Existing flattened references will continue to work, but we do flag the old mappings in the issue checker as references for where to remap if sticking with the new structure. We agree its pretty aggressive to not have an easier way to opt into the nested structure from the existing flattened structure.

We are working through the bug reports as quickly as possible and will explore a smoother transition. stay tuned.

[pork1977gm]

I can’t be sure of this but figured it was worth mentioning because it’s going to catch some other plugin developers out for sure, if indeed it is related.

Previously, if you used the API connector inside a plugin to create a type of data type state, let’s say it was called “Cues” (a list) and defined like:

{
"id": "string value",
"videoTime": 42.5,
"lat": 50.7192,
"lng": -1.8808
}

To update that in the code, we might be running something like this:

let myCues = [];
myCues.push({
_p_id: “123”,
_p_videoTime: 36,
_p_lat: 70.2344,
_p_lng: -1.2332
});

If we ever needed to add a new key value pair to the state, so it might become this:

{
"id": "string value",
"videoTime": 42.5,
"lat": 50.7192,
"lng": -1.8808,
"alt": 56.00
}

All we would have had to do was to simply add it, update the code and refresh the editor.

If you do that now, the state becomes un-associated in the App editor, so you get this show (which never happened before). If you then just do what you’d expect, and change that in all places to reference the state correctly, then the code no longer push values to that state.

The fix is to delete the state from the plugin, recreate it using your new object and then re-associate everything in the App editor to read it. It seems as if the code still tries pushing to the old reference but because you have to change it through the App editor, you don’t see the values, hence why the deleting and recreating works.

Obviously pushing an update like this will break everyone’s app who is using it and given the size of some people’s Apps, this could be a painful exercise for the user to carry out.

Paul

[ihsanzainal84]

You mean specifically in testing or is it breaking in existing production apps?

[pork1977gm]

As far as I’ve seen, it’s in the test app only, but I’ll be pushing an update shortly then updating another app with the plugin installed. Will let you know what happens as soon as that’s done.

[pork1977gm]

Here we go, before update:

After update (10 issues)

Easily fixable by simply getting users to reselect the states, but gets more of a pain if they’ve used that state as a field in their database because it will require a new field to be created since you can’t change the data type after a database field has been setup.

[almi.77.dev]

100% agree, An API call to an LLM defines the basic call. However, the response may be a structured reply specific to each type of call. And it is this response that needs to be parsed. As it stands, this requires defining an API connector entry for each use case.

[parthapatel0723]

Yeah this is real. Right now, I have a sys prompt that sends empty key vals in the JSON strictly, sending all keys just so I can register it in Bubble for every single tool outcome.

[net-tt]

I can’t see the values in list any more. Not even in the raw text. So now I don’t know what values I am getting back when I initialize. What can I do?

[jbutt]

Same issue here, julie!

And moreover, support to Date (UNIX) seems to be gone. Luckly I was able to revert to a previous version after making some changes, but any change in any API call I do breaks my entire app straight away! Wondering if it’s the moment to migrate my entire front-end back to pure code. This is really stressful and is completely out of our hands by the looks of it.

Let us know if you find a solution for this case. I’ll do the same here!
Thanks!

[mike_pacnw]

This re-mapping is not trivial, even though it’s straightforward. It’s incredibly time-consuming. After sanitizing data in one API response, I ended up with >90 issues requiring re-mapping across many workflows and UI elements. And I have many more API responses to clean up in order to avoid data leaks. Previously, I could easily sanitize data in an API response or even add a new field to the response and not have to worry about major re-mapping issues. I would only have to worry if I changed a previously existing field’s name or type. I would honestly like to turn off this “native JSON parsing” feature so I can avoid hours of re-mapping just to clean up data in my client’s API responses.

[mike_pacnw]

same issue - this is a huge time drain.

[nick.carroll]

@jbutt

And moreover, support to Date (UNIX) seems to be gone.

Its still there, but I believe there is a bug that is not recognizing Date (UNIX) when a call is re-initialized, so it just needs to be re-mapped using the type dropdown.

@mike_pacnw

This re-mapping is not trivial, even though it’s straightforward. It’s incredibly time-consuming

Agreed, this is not ideal. We are looking into making the flattened / nested response decision optional so users that need to modify existing calls don’t need to remap the entire frontend. Stayed tuned - its top priority to make this more seamless.

@net-tt

I can’t see the values in list any more.

This is not expected behavior - have you filed a bug report for this yet?