Hi all,
For sometime now, I’ve used Scraper API for a number of useful tasks so I ended up with a paid account, which works great for pretty much everything I use it for.
The number of credits I get are 100,000 which is way more than what I need, so rather than all those lovely credits go to waste (they’re reset every month), I thought I might as well cobble a few actions together, pop them into a plugin and basically give it to the community.
So… this plugin is a pre-paid scraper plugin which everyone can use.
The only thing I’ll say, is try not to abuse it! Depending on what options you select, will dictate how many credits are used per scrape. Once the credits have gone, they get reset at the end of the month. I’ve incorporated a rate limiting technique into the client-side action that prohibits any calls being made that are less than 10 seconds apart.
Just to give a bit of clarification on that, if you had a workflow which looked like this…
Bubble will run all of them in succession, but the plugin will only make the request in the first action. After 10 seconds have passed, upon re-running the action it will then be allowed to execute another call. I’ve done it like this in an effort to stop too many requests from happening and degrading the service.
To see some sort of visual appearance of this (if you’re interested) then if you were to open up the console, you’d see something like this… (top arrow started running, everything in the middle is me spamming the action, end arrow is when you’re allowed to run the action again).
Other than that, I’m hoping it will be of some use like it has for me.
PLUGIN
https://bubble.io/plugin/scraper-api-1675679788944x872848273950900200
DEMO
https://paul-testing-4.bubbleapps.io/scrape
EDITOR
https://bubble.io/page?type=page&name=scrape&id=paul-testing-4&tab=tabs-1
ELEMENT
Scraper API
You only need to put this element onto your page if you want to run the Scrape (client-side) action.
ACTIONS
The settings shown below are the same for all variations, so I’ve outlined them once here.
-
Scrape (client-side)
Allows you to scrape a specific URL with options, using an asynchronous fetch method.
-URL
Supply a URL to scrape and the data will be returned in the 'data' state.
-Render
Activate Javascript rendering by setting this value to true. The API will automatically render the Javascript on the page and return the HTML response after the Javascript has been rendered. Only set this value to true if you really need it, more credits are consumed and most scrapes will be just fine without this being changed. This option can make the request take a little longer.
-Country code
Activate country geo-targeting by setting this value to either us or eu. These are the only two country codes supported in this plugin.
-Premium
Activate premium residential and mobile IPs by setting this value to true. Only set this value to true if you really need it, more credits are consumed and most scrapes will be just fine without this being changed. This option can make the request take a little longer.
-Device type
Set your requests to use mobile or desktop user agents by setting this value to either desktop or mobile.
-Ultra premium
Activate an advanced bypass mechanism by setting this value to true. Only set this value to true if you really need it, more credits are consumed and most scrapes will be just fine without this being changed. This option can make the request take a little longer. -
Scrape (server-side)
Allows you to scrape a specific URL with the same options defined above, but as a server-side action.
STATES
Only populated when running the “Scrape (client-side)” action. When running the “Scrape (server-side)” action, the same data is returned as Result of step …
-
data
This state holds the data returned when running the Scrape action. -
status code
This state holds the status code from a scrape. -
is running
This state gets set to yes when the scrape action running.
EVENTS
-
scrape successful
Triggered when a scrape has succeeded. -
scrape errored
Triggered when a scrape has errored.
And that’s it!
Paul
