Getting/downloading API data to bubble database

Hello and happy new year!

I got a burning question that builds on my new application.
I’m using a scraper (webscraper.io) to collect data from websites. The scraper has an API with different functions such as GET, POST and endpoint URL, and sends data in JSON format.

The webscraper.io sends a notification to the endpoint URL when a scraping job has finished.

My question is, how do I get the data to automatically save to my bubble database using the API connector setting up endpoint URL with the webscraper.io’s API and to download the data to the database.

The API connector as the name suggests is used for CONNECTING to various third party API’s
But in your case you need the third party API to connect to your API…
You need to create an “API workflow” for the same and then give the “API Workflow’s URL” to your third party

In the API Workflow you could create the “thing” to be stored in your bubble database

Great got that working.

But now another problem arises.
When i try to GET the download via CSV (the format it is send in) with the API, I just get a bunch of random letters as in the image:
api problem

Is there some way to store/read CSV data in the database via API call?

Best regards,
Niklas

Hi Niklas
Great to hear that!

The output it seems from your third party is not CSV…(Since a CSV is a comma seperated file…which is basically text seperated via comma)
But on the output you have attached I see a garbled response…
This generally may happen due to the fact the file may be .docx or .odt…

But in any case if you want to parse CSV files…you might want to first search if someone has released a plugin that would allow you to parse the file…

Worst case scenario…
Write a program in JAVA,PHP etc to parse the thrird party’s data and then pass the formatted output to your Bubble’s API endpoint…

Hello Andrew,
I greatly appreciate your help!

The information given on the API webpage of the scraper is the following:

I’m pretty sure its CSV.

I’ll try to search plugins, but maybe i’m doing something wrong.

Any other suggestions?

Best regards,
Niklas

When you try to access the URL via your browser what is the output?
i.e on your browser eg. Firefox…type in the URL along with the API Key similar to what you have attached in your screenshot…

I get the following

Endpoint result

Here is what the file contains

Try accessing the API via Postman…
If the API is working via Postman, there should be no issue in connecting with it via your Bubble API endpoint

The only other possibility that I can think of is that, that your API Endpoint has not been configured properly

Trying Postman now. It works fine with postman

My api call is setup as the following image:


Is it wrong?

Best regards,
Niklas

Your data type is JSON and also the body type is JSON
Whereas the ouptput content type is a CSV(Which is nothing but a Text file)

I believe the issue lies there…

Try setting the data type to TEXT

I did, and it returned the messed up text again.

Here is the workflow

Also tried with Form-data and it returns with the same messed ud text

It seems that your third party API needs the request to be in particular format…
And as a result is returning a garbled response

Since I have not used this particular API before you might need to take this up with them…

I’ll try.

I greatly appreciate your time Andrew. Thank you so much.

Best regards,
Niklas.

niklasf,

Did you ever figure all of this out? I have a sitemap in webscrape.io that I would love to make public via a bubble.is app

Hello Scott,

Unfortunately not. Bubble wants to put all the data into 1 cell - it won’t put 1 thing into 1 cell. So i ended up making a scraper in R which made it possible - just not with bubble, which is unfortunate as it would be great to be able to access the data on my bubble app.

If you find a solution please share it here for future visitors. :slight_smile:

Hi Niklas,

It’s been some time now since your thread, but I’m currently in the same situation with the Webscraper.

I have found out that they closed their API’s and just created a cloud platform instead. Do you have any suggestions for a plugin or any other solutions that I could use to implement scraping functionality into my Bubble app?

Thanks!

@Yurchax.

I’m looking at this at the moment - did you find a good solution?

Thanks!

@darius You could take a look at apify.com. When a crawl is finished they can send a webhook, and the data can be fetched as json. Generous free tier as well.

@simon7

thanks - appreciate the speedy reply!

Hey @simon7 I have a specific apify challenge would you have any tips?

How would I send a dynamic user generated request, such as a unique Airbnb URL in the Apify API call? The use case is to speed up vacation rentals onboarding flow by sending Apify to crawl a site like airbnb and then saving the image files to the database so the user doesn’t have to upload all of their home/property images again.

Any ideas on best ways to think about this are super appreciated. I can get it working on Apify manually but linking the two up and then setting up some sort of workflow is where I could do with a bit of direction and advice.