Hi,
I was wondering how one could get API access to a website to scrape data directly to your app? Looking to send data from external URLs to my app. If you have a way you recommend to do this, please let me know.
-Nick
You either gotta use code or use third party tools. @rony.kris is the expert of this.
Explore how backend workflows and the api connector plugin works.
Below a few videos to consider:
Hi,
Thanks for joining in on the conversation and offering your help! The first video you sent was pretty close to what I am looking for. But instead of sending data to a sheet from an internal form, I am looking to pull, for example, listings from Ebay (in first video) to my app. But, I want to use a site that doesn’t offers API’s for everyone to use.
Hope this makes sense.
-Nick
Hey @nick.dingler
If you want to get data from a site and not via an API that they make available, then you need to scrape the site.
The challenge is that most modern sites don’t want you doing this (understandably) so if you use basic measures they will block you really quickly.
So if you want to go down this path you might like to look at services such as ScrapingBee for this sort of thing. There’s actually an elaborate process needed behind the scenes to visits a lot of pages on a site, make it look like normal traffic, understand the structure of their pages and then extract the information you want from that page
Having said all that - be careful - you’re 99.999999% going to be in breach of the site’s T&Cs and open yourself up to litigation if they find out what you’re doing and don’t like it. I mean most likely you’ll get away with it, but that risk doesn’t go away.
Josh @ Support Dept
Helping no-code founders get unstuck fast 
save hours, & ship faster with an expert 
on-demand
1 Like
Hey @nick.dingler ,
As josh pointed out scraping could be tricky. But if you are up for it then there are ways to scrape data from sites and store in your bubble app.
I do not know if there is a nocode way of doing this. Therefore, if you have a javascript background you’ll be able to get it done pretty quick. Below are couple of options you may want to explore.
-
Write a plugin - the goal of the plugin would be to scrape data from the website of interest and store it in bubble’s db. You could write up the plugin using node modules axios and cheerio. If you know js this should be too difficult.
-
Host your js code to scrape (90% of it is similar to 1) in a fucntion as a service infra. eg: AWS lamba, Google Cloud to name a few. This mechanism would create an API which could be accessed via the Bubble API connector.
To request html from the sites you could use scraperapi and use cheerio to scrape content from it. Note: scraperapi rotates your IP and proxies too. Therefore, chances of getting blocked could be low.
Anyways, good luck!
1 Like
Check out https://www.parsehub.com/ it might be helpful. You will just need to know a little bit about css selectors and the DOM to do more complex things.
1 Like