[PLUGIN] - Scraper API (free for everyone)

Hi all,

For sometime now, I’ve used Scraper API for a number of useful tasks so I ended up with a paid account, which works great for pretty much everything I use it for.

The number of credits I get are 100,000 which is way more than what I need, so rather than all those lovely credits go to waste (they’re reset every month), I thought I might as well cobble a few actions together, pop them into a plugin and basically give it to the community.

So… this plugin is a pre-paid scraper plugin which everyone can use.

The only thing I’ll say, is try not to abuse it! Depending on what options you select, will dictate how many credits are used per scrape. Once the credits have gone, they get reset at the end of the month. I’ve incorporated a rate limiting technique into the client-side action that prohibits any calls being made that are less than 10 seconds apart.

Just to give a bit of clarification on that, if you had a workflow which looked like this…

Bubble will run all of them in succession, but the plugin will only make the request in the first action. After 10 seconds have passed, upon re-running the action it will then be allowed to execute another call. I’ve done it like this in an effort to stop too many requests from happening and degrading the service.

To see some sort of visual appearance of this (if you’re interested) then if you were to open up the console, you’d see something like this… (top arrow started running, everything in the middle is me spamming the action, end arrow is when you’re allowed to run the action again).

Other than that, I’m hoping it will be of some use like it has for me.


PLUGIN
https://bubble.io/plugin/scraper-api-1675679788944x872848273950900200

DEMO
https://paul-testing-4.bubbleapps.io/scrape

EDITOR
https://bubble.io/page?type=page&name=scrape&id=paul-testing-4&tab=tabs-1



ELEMENT

Scraper API
You only need to put this element onto your page if you want to run the Scrape (client-side) action.


ACTIONS

The settings shown below are the same for all variations, so I’ve outlined them once here.

  • Scrape (client-side)
    Allows you to scrape a specific URL with options, using an asynchronous fetch method.

    -URL
    Supply a URL to scrape and the data will be returned in the 'data' state.

    -Render
    Activate Javascript rendering by setting this value to true. The API will automatically render the Javascript on the page and return the HTML response after the Javascript has been rendered. Only set this value to true if you really need it, more credits are consumed and most scrapes will be just fine without this being changed. This option can make the request take a little longer.

    -Country code
    Activate country geo-targeting by setting this value to either us or eu. These are the only two country codes supported in this plugin.

    -Premium
    Activate premium residential and mobile IPs by setting this value to true. Only set this value to true if you really need it, more credits are consumed and most scrapes will be just fine without this being changed. This option can make the request take a little longer.

    -Device type
    Set your requests to use mobile or desktop user agents by setting this value to either desktop or mobile.

    -Ultra premium
    Activate an advanced bypass mechanism by setting this value to true. Only set this value to true if you really need it, more credits are consumed and most scrapes will be just fine without this being changed. This option can make the request take a little longer.

  • Scrape (server-side)
    Allows you to scrape a specific URL with the same options defined above, but as a server-side action.


STATES

Only populated when running the “Scrape (client-side)” action. When running the “Scrape (server-side)” action, the same data is returned as Result of step …

  • data
    This state holds the data returned when running the Scrape action.

  • status code
    This state holds the status code from a scrape.

  • is running
    This state gets set to yes when the scrape action running.


EVENTS

  • scrape successful
    Triggered when a scrape has succeeded.

  • scrape errored
    Triggered when a scrape has errored.


And that’s it!
Paul

5 Likes

Hi,

I see the plugin isn’t for free anymore, are the calls still limited to 10 seconds apart from each others?
And if yes are those 10 seconds for the whole people or per plugin bought?

Thanks,

Franck

Hi Frank, sorry so I had to make a few changes because someone was using it so much, they were caining 20,000 credits daily.

Good question though and the limit is still in place, I can remove that. It’s a limit for that one action only (the client side one).

I’ll remove the limit since there’s no need for it now.

2 Likes

The rate limit is removed now @franckwebpro

Hi,
thank you for your quick answer.

Nice I’ll check your plugin! :+1: