How I setup Sendgrid Inbound Parse

We’ve recently built an inMail type feature where a user of our app can message another user using Sendgrid parse.

The flow is as follows: User1 messages User2 from within our app. User2 gets an email from our app with a reply to email of uniquekey@parse.ourdomain.com. User2 replies to the email from User1. The email with it’s special reply to domain is routed through sendgrid back to our messaging app in Bubble. The uniquekey is used to figure out who sent who a message and tie that message back to the right inmail box.

Seems complicated, but it was surprisingly easy. This is how we did it…

  1. Create an endpoint in Bubble’s API Workflows. Let’s call the endpoint “inmail”.

  2. Follow this guide to set up your DNS.

You’ll need to add an MX record to your DNS. (I added 2, one for prod and dev)
prod: “parseprod@mydomain.com
dev: “parsedev@mydomain.com

  1. In Sendgrid, left nav, select inbound parse, and click “Add Host & URL” (upper right). You’ll do this for each of your MX records.

subdomain = “parseprod”
domain = “yourdomain”
destination URL =
prod - “https://mybubble.is…/api/1.1/wf/inmail”

subdomain = “parsedev”
domain = “yourdomain”
destination URL =
dev - “https://mybubble.is…/version-test/api/1.1/wf/inmail”

  1. Back to Bubble, set the “inmail” endpoint to “detect request data” . Response type = JSON

  2. As the first action, store what the endpoint receives so that you have the full message to troubleshoot later

  3. Let’s test it…on dev. Trigger an email from your app to yourself, but use a reply_to email that has the following format. Also store in Bubble a table with “To user”, “From user” and “RandomString”. This will enable you to figure out who is sending who a response when you get the inbound parse message back (from User2).
    randomstring@parsedev.yourdomain.com

  4. Alls well, you should get an email from your Bubble app. Reply to it (eg. reply to randomstring@parsedev.yourdomain.com), this will go through sendgrid and get routed to your “inmail” endpoint then get stored in your app because that’s the 1st action.

  5. That’s it. If you want to further parse the contents of the email
    :extract with Regex (?<=<)(.*)(?=@) in bubble to get the “To” email address (who wrote who back)
    :extract with Regex to parse off only the most recent reply email. Sendgrid by default sends the entire email chain back, so we grab the characters prior to sendgrid timestamp. I’m using python to do this step because the regex was a pain, but it’s also likely doable fully with :extract with Regex.

This is the gist of the regex I’m using.

left of … On [J|F|M|A|J|S|O|N|D][a-z]{2}.[0-9]{2}, [0-9]{4}
or
left of On [J|F|M|A|J|S|O|N|D][a-z]{2}.[0-9]{4}, [0-9]{4}

It works, though if your users’ respond with a timestamp of their own, their message will be cut short. Ideally, sendgrid would have an option to inbound parse the most recent message. I didn’t see it. Would love to know if someone has a better solution.

If anyone is interested in figuring out the regex in bubble, I’ve include a sample below. The goal is to uniquely isolate the left-hand side “…new message”
…new message > On Feb 10, 2019, at 8:34 PM, …old message
…new message \n\n\n\n\n\nOn Sat, Feb 9, 2019 11:22 AM, …old message
…new message \n\n\nOn Thu, Nov 29, 2018 1:46 AM, …old message

31 Likes

Thank for sharing @kramwe!!!

1 Like

Great share, @kramwe. This has been something on my list for a while to explore…

1 Like

This is very helpful. You have saved me a lot of time that I would have spent for my marketplace comms.

Very useful post, instant bookmark, I’ve always pondered how this is possible to do with Bubble + SendGrid, glad to know its achievable. Cheers.

Awesome thanks @kramwe! By chance do you know how to get rid of email signatures in the body of the email? May be asking a lot to get most recent reply and hunt for that also.

That is a great question I don’t have a solution for.

To my knowledge there isn’t a standard way that email clients denote an automated signature. You might be able to use an NLP package that spots names or third party software (like this) that can extract (and remove) the signature, but I’d doubt it’s foolproof.

Thanks! I’m working on trying to get a regex for most recent email reply and its really tough, I’ll keep you posted if I figure it out. As of right now I’m basically using what you’re using but a slight variation for days of the week instead of the months

(.*)(?=On Mon|On Tue|On Wed|On Thu|On Fri|On Sat|On Sun)

I think what I’m going to do though is put an email preheader in all my email templates with the phrase "–please reply above this line–to know that’s where we should chop off the text.

Can you explain a little more about what you meant by using the regex in python? Never worked with that before.

I also put a feature request in at Sendgrid about getting only the most recent message.

From easiest to hardest

  1. Sendgrid feature
  2. Regex it
  3. Python.

The python solution is Bubble Connector+AWS Gateway+Lambda. It’s definitely not worth setting up for this task alone, but it’s something we use extensively throughout our app to do math, loops, and other process intensive tasks that aren’t straightforward in Bubble today.

I don’t know how to code plugins, but I do have some libraries to contribute if someone else takes the lead.

  • This regex one
  • Timezone converter
    etc…

Cool thanks, I think Regex is the best way for now I’m just having a lot of trouble getting it to pop up right in bubble. I’m using regex testers online and the expressions are picking up the right stuff but they won’t in bubble. Really weird.

Trying to think of Bubble text functions can help this like number of characters, truncated but I don’t think so

Bug report time?

I ran into these 3 variant cut-points. Some include month.
On [J|F|M|A|J|S|O|N|D][a-z]{2}.[0-9]{2}, [0-9]{4}
On [J|F|M|A|J|S|O|N|D][a-z]{2}.[0-9]{1}, [0-9]{4}
On [M|T|W|F|S][a-z]{2}, [J|F|M|A|J|S|O|N|D][a-z]{2}.[0-9]{2}, [0-9]{4}

I don’t know if it’s a bug report because if you use some regex editors online some give you a different answer than others.

I think the issue with the regex is capturing the text that is line breaks above the expression that appears every time, On Mon, this guy wrote, etc. The regex doesn’t let you get all that text regardless of line breaks above. Gonna keep cranking at it.

1 Like

Ok I think I figured it out/almost there. Here’s the regex as a starting point:

[\s\S]+(?=;–Please)(.*)(?=)

Here’s the email body:

"Hello this is a test

Test
""This is a test

Hello this is great

On Thu, Feb 25, 2019 at 1:16 PM redacted@gmail.com wrote:

Fooba

;–Please place all replies above this line–

https://u3853310.ct.sendgrid.net/wf/click?upn=RrBS4Qj2mjD8Uz60ms06NIvnAhuwoEqiY0wMwK5PWdE-3D_8oxW4HKyAHmpsFLR5-2FCi08gmdP3zHJYBpJF1TwngPjP5-2FGVVAzS5x-2BSz2y4U8T5ynmYs4-2Fv5ZAd003pJS0-2BK\70baDlvMVaUD2BTgh4Q77HScN6UsApLQ8-2FMVLJXGXb-2B7okfuktsNl4nV5ahOPiyFB8SWrikXkanViWa1Lv1Y60pBzIC70pyrlqAdNBsOa9hf6UPgq

You can reply to %VAR3% by replying directly to this email!
%VAR3% says: %VAR4%
Visit %VAR3%'s dock

%VAR7%
Arrival:

%VAR9%
Departure:

%VAR10%
Reply to Message

Boater Support
Have any questions or need any assistance? Please feel free to email us at
redacted@redacted.com and we’ll be glad to help you!

“”
"

This will chop everything after ;–please reply above this line. I think there’s another regex that I found online that leaves every single character in the entire body of text except for something which I’m going to work on next.

I think at the minimum you can do a multi stage :extract with regex over and over again until you chop the text away you don’t need.

I think it’s really important that in your sendgrid email templates you set up in your email preheader in the template headings a similar “Please reply above this line” that would appear in every single email, regardless of mail client. That will allow you to just demolish everything below that line automatically, and then in 99% of cases you’re always going to have a “On Mon, Jan, 23, 2018, jonathan.whitney@hello.comsaid this”, so you can chop that away as the last step in the regex.

1 Like

After playing around with this further I discovered Regex is really tough for this and the root cause is Sendgrid not having built in functionality to send you the most recent reply. You really need an external service with a big development team to do this unless you’re a huge organization that wants to do it in house with your own code.

I ended up integrating Mailgun which through initial testing I’ve found to be phenomenal. If anyone wants any help on how to set it up feel free to send me an internal message.

1 Like

@kramwe @jonathan.timianko Hey guys, have either of you tried saving attachments from the emails?

I’m having trouble here:

How to I turn this into a file that Bubble can save?

Hey thanks for your question, I abandoned sendgrid inbound parse a while back, i highly recommend checking out mailgun if you have a chance it’s must easier to set up and works much better with Bubble.

2 Likes

Hi,
Does it mean that you’re able to retrieve attachments from an email and integrate it into DB ?

thanks !

I am only a starter and I even have configured sendgrid for sending emails for my WordPress web website online the use of put up SMTP plugin. It is running fine. As I am the use of google cloud for web website hosting with cloud flare. I don’t understand a way to installation sendgrid for receiving emails for my WordPress web website online the use of the e-mail call I even have installation in Wordpress.I observed the commands in sendgrid documentation for inbound parsing however I don’t apprehend in which the URL ought to factor to. Is it a mail control provider or some thing else? I want a few assist please.
ahead incoming electronic mail for your mobileular telecellsmartphone as a textual content message. While this isn’t realistic for each electronic mail you acquire there can be instances in which a textual content message is simpler to acquire than an electronic mail (say, even as on vacation…). We’ll use Twilio Programmable SMS and Twilio SendGrid to attach those together.
ThaNkS for the tip sir

You lost me at step 1. How do you create an endpoint in Bubble’s API Workflows?