Save or send Request Data from a webhook

[david.j.hansford]

My app receives webhooks from Slack. Annoyingly they send all webhooks to the same URL but the structure of the data they send changes depending on the triggering event. The main issue is that one webhook might send the property ‘user’ as text:
user: name
And a different webhook might send user as an array:
user: {id: 1234, name: name, age: 30}

As a result I have to choose one structure to initialize my workflow with and lose any data that arrives in the other structure. Is there a way to…

1. Pass request data from one bubble workflow to another? That way I could set up a secondary workflow to expect ‘user’ in an array format perhaps?
2. Save request data to the database so that I can extract what I need from there instead?
3. Any other approach that will let me separate out my range of webhook payloads based on the event that triggered them?

This is a really big issue for me at the moment so any help would be amazing!

[shpak.serhiy]

You can refer in an action conditional the webhook payload. For instance, if in the payload there is event name then it might be perfect for the conditional.

[david.j.hansford]

Thanks but the conditional isn’t the issue. I can make a conditional but I can’t send the whole JSON payload to another backend workflow when that condition is met. Without the data another workflow can’t do anything.

I’ll add some more detail here in case it helps someone understand my problem:

I am subscribed to a number of events in Slack’s API. Here are two examples:
Reaction Added
{ "team_id": "T********", "event": { "type": "reaction_added", "user": "U**********", "reaction": "+1", ...etc }

Team Join
{ "team_id": "T********", "event": { "type": "team_join", "user": { "id": "U**********", "name": "david", ...etc }, "time": "*****", ...etc }

(Note that user is a String in reaction_added but is an array of objects in team_join)

In order to let slack ‘Detect data’ for a range of webhooks each with different data types I have made one big payload that is a combination of all data types I might receive from all webhooks and sent it to the workflow initialization process using Postman. This allows me to use the data for most things but not when a data type varies between one Object and an array of Objects, such as user in the example above.

So how do I get around this problem? Is there a way to save the payload in whatever form it arrives so that I can access it afterwards and extract the information I need?

Alternatively, can I pass the payload on to another workflow based on a conditional (such as event_type = team_join), where the second workflow knows to expect user is an array (despite the original workflow expecting a string), thus allowing me to access user's contents in the second workflow?

[shpak.serhiy]

@david.j.hansford

Alternatively, can I pass the payload on to another workflow based on a conditional (such as event_type = team_join), where the second workflow knows to expect user is an array (despite the original workflow expecting a string), thus allowing me to access user's contents in the second workflow?

This is exactly what I mean. If conditional detects the event_type = team_join, then you know which properties belong to this web hook and can pass them to an API workflow devoted to this event type.

[david.j.hansford]

But how do I resolve the problem of changing data types? Let’s say I set up an API workflow called switchboard who’s only job is to detect the event_type and send the JSON payload on to a second workflow designed to handle that event. Let’s continue the example of a team_join event.

switchboard could receive the user type as a String or an Array of Objects, depending on the event_type it receives. However, I am forced to tell it to expect either a String or an Array (when I setup the workflow with ‘Detect data’). So let’s say I tell switchboard to expect user as a String. When a team_event is triggered, switchboard will expect user to be a String but it will be an Array. It will still accept the data but how do I pass that on to the second workflow designed to handle the team_join event?

I can’t extract the user data as variables to pass through because switchboard thinks user is just a String but it is actually an Array. If I pass user or try to save it to the database from switchboard it appears as {Object Object} and the actual contents are lost.

I also can’t just pass the entire Request Data to the second workflow, that isn’t an option.

So, how do I get switchboard to handle two different types of user, passing the correct data onto secondary workflows? Is there a solution I just don’t know? (Really hope so!)

[shpak.serhiy]

@david.j.hansford

In order to let slack ‘Detect data’ for a range of webhooks each with different data types I have made one big payload that is a combination of all data types I might receive from all webhooks and sent it to the workflow initialization process using Postman. This allows me to use the data for most things but not when a data type varies between one Object and an array of Objects, such as user in the example above.

You suppose to be able to receive and read any whebhook from Slack. Based on the event in conditional you schedule another backend API workflow and send there only parameters that belong to the event from the conditional.

[david.j.hansford]

I’m sorry but there is clearly a misunderstanding in communication here. I know bubble can receive any webhook. I know how to use conditionals. The problem is that Slack sends different data payloads to the same URL each time and certain data types (such as user) change type depending on the event. Bubble needs to know in advance the format of incoming data, but that format varies.

Again, for clarity, here are two example payloads:

reaction_added:

team_join:

Note how user is a string in reaction_added but an array in team_join. Currently I initialize my bubble workflow with user as a string because it’s the most common format. However, whenever I receive a team_join event I cannot access the user data because it expects a String but receives an Array. Is there any way around this?

[shpak.serhiy]

Yes, I didn’t got the point right.

I don’t have a solution for this issue as I have not faced and solved it personally. But here is how I would approach it.

The problem narrows down to the user object that is sent in different formats.

You can either do not receive the event “team_join”, which sends user as an array.

Or you can search for ways on how to parse JSON from string in the backend workflow. For events when you receive user array in string format. I know there are plugins for this that work on the front-end and I am not sure that there are similar plugins for the backend - and such plugin might be a solution here.

Another potential solution is requesting slack api an event based on id. But my quick research show that slack doesn’t provide such method.

[david.j.hansford]

Sadly I can’t ignore the team_join event (or other examples that suffer the same issue) as they are integral to our app. I also looked for a slack API call to request data on an event by its id but equally failed to find any options in their API documentation. (This begs the question: why bother to give us an event_id if we can’t use it?)

I’ll have a look for a plugin that can parse JSON from a string in the backend. Would the idea here be to setup my workflow to expect a String instead of JSON, then parse this String for every webhook it receives to turn it back into a JSON and retrieve the data?

[shpak.serhiy]

@david.j.hansford

I’ll have a look for a plugin that can parse JSON from a string in the backend. Would the idea here be to setup my workflow to expect a String instead of JSON, then parse this String for every webhook it receives to turn it back into a JSON and retrieve the data?

Yes, correct.

[david.j.hansford]

Great, thanks for the idea. Fingers crossed I can make it work!

[kkirner92]

Hi @david.j.hansford I had the same problem recently and I solved it using hookdeck. You can use their transformer to modify the payload of each hookdeck before it ends up in bubble → thus you always get the same structure.

[david.j.hansford]

This looks perfect. I’ll definitely check it out, thanks!

[kkirner92]

If you need help with the transformer, you can reach out (I also answered to Kourosh in slack)

[david.j.hansford]

Been playing with hookdeck.com for a few days, this looks like the answer I’ve been searching for for ages. Not only can you set up filters to split requests towards different endpoints, you can also handle rate limiting, retries after failure and you can go back and look at the webhook payloads whenever you need to debug anything. Thanks for the suggestion!

Quick price info too in case it helps:
$39pm for 1M requests/month and up to 10 requests/second.
There’s a free option and other options from $179pm if you need more.

[23cubed]

@david.j.hansford

Not sure if this works for you but saved me. Basically theres just a couple steps here and I think it dramatically simplifies the workflow.

1. Create a GET call in the API connector for the specific data type
2. Create a Webhook that receives request data. I assume this request data will come with an ID identifying the object.
3. When the request data is received, call your GET call from the endpoint and you will have a payload that you can parse in Bubble!

I find with this solution my data types only need 2 fields, one to hold the ID and one to hold the Payload. The ID field being for when you need to update an object in bubble you must filter by the ID.