My particular issue has been troubling me for over a month now and I cannot seem to resolve it. I have been able to talk with a Bubble engineer who validated that my solution was not achievable with Bubble right now. My original solution was to send a dynamic body value (JSON output from JS) to the Data API Bulk call to create a list of things. This was their response:
-
our API connector currently does not allow passing freeform content with a custom content-type. This is commonly requested, and is something that we want to address in the short to medium term, since that’s a relatively way to expand the API connector’s capabilities. The content-type header is imposed in all POST requests, and set according to your choice of body type in the dropdown: form-data imposes the ‘form-data’ value, while JSON imposes ‘application/json’. Therefore, when triggering this call in the API connector, the text/plain header is just ignored, because that’s how we currently generate the call object.
-
our Bulk data add API endpoint will not accept a JSON array. The reason for this is mostly to prevent large JSON parsing from bypassing the capacity system, but given that is already a fringe use-case, we’ll be discussing options to accept an array of objects instead of newline-separated json as a string.
I replied back asking for the best possible way to achieve what I want and will update here what they say. In the meantime I wanted to post the issue here and see if any Bubble expects have any ideas.
Issue
What I am trying to achieve is turning a JSON of bubble things into db entries.
I have created JS to output a JSON object of bubble things.
Here is an example of the JS output:
[{"home_team":"1614857369762x407703519835146430","away_team":"1614857371185x610581487636067100","fixture":"1614857368353x368193783608377340","gameweek":1}, {"home_team":"1614857370153x260604513178683840","away_team":"1614857370804x551033893963115440","fixture":"1614857368353x368193783608377340","gameweek":1}]
Each object will represent a new bubble thing in a table called league.fixture.fixtures.
home_team = bubble thing (league.team)
away_team = bubble thing (league.team)
fixture = bubble thing (league.fixture)
gameweek = number
I have attempted parsing the data via the API connector and was yet met with another bubble limitation confirmed by the Bubble engineer. The Bubble API cannot map back to bubble things because as quoted because it is mostly designed to access third party services, not itself.
If anyone has any ideas on how to get this working would be amazing.
Why aren’t you just doing this with schedule an API workflow on a list? Are you creating a big list of permutations from three smaller lists?
If your requirement is “I have some stuff over here” and “I need bulk bubble instances created over there”
Then your current solution is “I made a Java script to turn it into a JSON list and I want to fire it into the bulk api”
But if your requirement can be done another way, would you take it?
The reason I am using Javascript to produce the JSON is I need to use an algorithm to generate said list. The short story is, the JS is a round robin algorithm to create a list of fixtures using a set of parameters which is a combination of other bubble things, text and numbers. This list will be all the fixtures (match ups) for a sports league. This list cannot be created via a bubble workflow as there is complex math thats required.
I cannot use the Bulk API due to the limitation Bubble has stated to me, therefore i need to find another solution to turn the JSON into bubble things.
How many rows does it have to be created?
How about this…
get your JavaScript to output a single text value (huge) instead of your JSON.
Delimit with comma between fields. And bar between rows “ | ”
API workflow #1 pass the ginormous text into it in a text data type
API workflow #2 expect a list of texts
API workflow #3 expect 4 parameters, the four in your data, in the types of team, team, fixture and number and use it to make a single instance of your match
You should only need one action in workflow #1, to set up an api workflow on a list for #2. The list you pass will be your input text, extract with Regex. Use expression. [^|]+
Set them a few seconds apart.
Split using the same regex in workflow #2 only with comma in place of bar only do that in your “do a search for” (team where unique id = input text: extract with regex: item# (0,1,2,3 I think, maybe 1:,2,3,4)) (search for results, :first item)
Use workflow #2 to trigger #3 which you’ve passed all nice bubble types into to make your fixture
Regex is the key to this
Ok I’ve had a crack and here is what I’ve done. Is this what you suggested?
Is this is the format you were proposing?
1614857369762x407703519835146430,1614857371185x610581487636067100,1614857368353x368193783608377340,1|1614857370153x260604513178683840,1614857370804x551033893963115440,1614857368353x368193783608377340,1
I managed to get it working and can create the fixtures but only the gameweek number value is working. Bubble doesn’t seem to like using a regex value when searching for a data type. In the logs it’s just blank so I will continue to debug. I’ve tested using the unique I’d directly in the search and it works. Using the extracted regex text just doesn’t seem to work.
1 Like
Yes that structure is good and the data format is nearly right but when passing data to the next level from 1 to 2 you need to be sending or searching in THIS TEXT not the fixtures list.
I don’t think that “bubble” doesn’t like anything, Regex should be Regex.
If the first workflow split the text on bar, and your week numbers made their way to the end, then the second workflow should have no problem splitting it on comma.
Try these fixes
Richard
I got it working!
I now just need to update the JS to output the new format and pass that to the workflows.
Thanks for your help @richard10
2 Likes
Great news!
I love Regex sometimes!
1 Like
if it is possible to send me this script, I would be grateful!
I’m facing the same issue and I can’t update my plan yet. I need a way around this
I’d stay away from Workflows in this case. If you’re looking to speed up the inputs, bulk insert is the way. Also, you’ll spend more on the apps by running workflows (higher cost).
What you want is a processor that will parse your string and submit it as REST to your DB.
One way or another, you’ll need a code for it. Amazon has a service they call Lambda functions that fit perfectly to your case. You will:
- receive the JSON from your Bubble call
- Send the raw JSON text to the Lambda function
- The lambda function will parse and format the JSON
- Then, the function will send a REST call to your DB API
- You’ll have a high-performance/low-bandwidth and low-cost solution
I had that same thought for Bubble to get the RAW JSON text, parse, and format it in Javascript, then call itself to insert bulk. I had a 1000% performance improvement when I decoupled this call - in my case, I used a jRuby script in my ERP platform to do that for me. In the past, I processed everything with a Scheduled workflow, and the performance was awful.
I think Bubble should deal with JSON and API calls the same way it handles databases or “things”. That would be a major improvement if I could map my API response to a “thing,” and Bubble itself resolves the inserts in real-time without any interference. We could have an API type such as “thing” in addition to “Action” and “Data”. We could also do some transformations before the data is written in the database or “thing”.
2 Likes
