Optimal Strategy for Storing and Updating Large JSON API Responses

I’m searching for a straightforward way to store an API response and use it at a later time. I retrieve product data, which only changes once every two weeks. I don’t want to execute a new API call each time the page loads. Instead, I aim to save this data and have a scheduled workflow update it weekly. The solution I’ve come up with so far is:

  1. Setting Up the API Connector: I first added my API through Bubble’s API Connector.

  2. Creating a Data Type: I went to the ‘Data’ tab in Bubble and created a new data type. This data type contains fields that correspond to the response from my API.

  3. Setting Up a Workflow: I created a workflow that triggers the API call. Once the API call is successful, I store the response in the database under the data type I previously set up.

  4. Retrieving Data: Whenever I need the stored API response, I fetch it from the Bubble database instead of initiating a new API call. I achieve this using the ‘Do a search for…’ action in Bubble.

  5. Refreshing Data: To update the stored data, I’ve considered setting up a workflow that deletes the old entries and then calls the API to store the fresh data.

However, I’m facing a challenge. The response is a JSON with hundreds of fields. Is there a method to directly input this JSON back into the API Connector or something similar?

What would be the best approach for this situation?

2 Likes

That’s exactly what I need too, I’m thinking about using backendworflow or going to xano, but I’d like to stay on the bubble.
I’m unsure of the best way to do this.

1 Like

Martijn,

Overall your flow seems to be correct.

"Is there a method to directly input this JSON back into the API Connector " I don’t think you mean to do this but instead to extract all the products from the JSON response and store them.

If so, there are a bunch of posts / tools / workarounds how to parse the JSON (e.g., [🌟 New Features: Ultra-Deep Drill] JSON Assistant Plugin: Your Solution to Complex JSON & API Data in Bubble - #46 by NoCodeDataArtisan).

This depends exactly how the data is being returned (array or flat) and extracting it may not be simple.

Some other considerations:

  • Can you modify how the data is being formatted from the API?;
  • Can you retrieve only updated products in the call (assuming not e/t is changed every 2 weeks)?
  • If you want to avoid dealing with all this JSON and the update only needs to be done every 2 weeks you may consider a workaround (Warning: It’s def not best practices but may be quickest for short term) where you spend 5 minutes every 2 weeks to manually import the new products (setting some DB triggers or WFs to override the old ones etc).
1 Like

Dear Jacob,

Thank you so much for recommending our plugin as a solution for Martín. It’s truly the best feeling when we see our work being endorsed and appreciated by peers like you.

Warm regards,
Mehrdad

1 Like

@martijntenpas @dhdonato
Hey there,

We’re thrilled to introduce you to JSON Assistant. It’s tailored to handle any JSON challenge you might encounter. Plus, we’re quick to add requested features, ensuring you always have the best tools at your fingertips.

If you have any questions or need further assistance, don’t hesitate to reach out.

Best wishes,
Mehrdad
@NoCodeDataArtisan

1 Like

Thanks for all the reply’s, it brings me some new ideas! Thanks. I will try some stuff and will come back with the solution I used.