Live Bubbling with @keith: Mega video about Image URLs, Scraping Data from External Sites, and Configuring the Amazon Product Advertising Search API!

keith · December 11, 2018, 2:43am

Inspired by what would seem like a very simple, innocent question in another “Need Help” post, I recorded this completely unrehearsed and talky-but-possibly-interesting hour-plus video that gives you a glimpse into how I think about Bubble and how I develop new ideas.

Starting with: “Can I get an image from another page’s meta images URL and put it in a repeating group without making the user have to upload it?” (short answer: yes), I go all over the place and describe some completely basic junk (displaying an image URL in an image element), some moderately advanced stuff (glossing over how to build a microservice / your own API), and on to using the impossibly complicated Amazon Product Advertising Search API with Bubble.

And it’s free. All you have to lose is your time and your mind. Come bubble along with me!

So here’s an hour and 12 minutes or so of me rambling on about interesting things that I did in response to this question. Showing an image URL in an image element, Funny cat pictures, weird Amazon products, building a simple meta data scraper with nodejs, the metascraper NPM library and webtask.io, tips about repeating groups, doing interesting stuff completely in the page and… wait for it… a very long and talky and possibly COMPLETELY confusing explanation of how to configure the NOTORIOUSLY difficult Amazon Product Advertising Search API (https://webservices.amazon.com/scratchpad/index.html)!

wOOT wOOT!

https://vimeo.com/305624479

keith · December 11, 2018, 5:04am

Errata: At about 28’30" I say that the image carousel is a Slick Slideshow element… It’s not. It’s just a fixed cells repeating group (duh - anyone could see that, but I was just rambling on…) and clearly I set up the next/prev arrows myself. Kinda looks like slick slideshow though!

At about 37’00" I say “'cuz this is what Bubble wants (regarding the Timestamp)”… I meant to say 'cuz this is what AMAZON wants.

The very end gets boring as I debug something… But – spoiler alert! – I get it working!

Enjoy.

luke2 · December 11, 2018, 9:14am

Nice video there @keith - plus got some last minute Christmas ideas haha

Thanks for coming through and explaining both processes out. I must explore https://webtask.io/ further, I didn’t realise how seamless adding the NPM packages are. Bit a learning needed next year on getting the basics for Node.js

Amazon’s authentication process with the certificate is quite fascinating and by the sounds of it, on the frustrating side

Cheers

keith · December 11, 2018, 3:59pm

[email protected] Glad you got something out of it!

I like webtask a lot as just a cloud based place to write and debug JavaScript. No local environment required. So that’s a good way to get started with it.

And the NPM integration is awesome. It used to be only some modules but now it’s anything.

So you can find or see something interesting in NPM and then immediately start playing around with it. (NPM’s Runkit feature allows a bit of that, too, but your work in webtask is persistent and more immediately useful in Bubble, via the API Connector.)

I should prolly do a very basic webtask example.

The Amazon Product Advertising API setup was an interesting exercise for sure. It took a lot of note-taking and trial and error to get it worked out. But the general technique there is applicable to any type of hash/signature computation you need to do server-side.

luke2 · December 12, 2018, 8:48am

Yeah I will definitely give Webtask.io a try, looks a bit more user friendly than Amazon’s alternative suite.

The method of saving the string in a data type and adding {} to then use the find & replace function was a neat little trick.

Your reaction to the search ‘Tuna taco glasses’ was priceless

Thanks again

sounderly.help · December 29, 2018, 5:23am

@keith, you’re on a totally different level. Great content!

keith · March 22, 2020, 10:52pm

Web scraping, implementing Amazon product Advertising Search and other nonsense: #lockdownandlearnwithkeith

sophie1 · February 15, 2021, 5:18pm

Hi Keith,

WebTask is no longer accepting new users. Is there something else that would work instead?

edwardbutcher · February 15, 2021, 7:14pm

You can now do a lot more on Bubble’s server than when Keith recorded this… I.e the things that were being run on WebTask can likely be done on Bubble (which now supports NPM packages).

What is it that you’re looking to do?

keith · February 15, 2021, 7:57pm

@sophie1 and @edwardbutcher: This is true, though the performance and value for money may be worth evaluating. Server Side Action plugins are basically Bubble’s built-in version of functions-as-a-service, and I am in fact migrating some of my webtask-hosted functions to SSAs.

There are lots of FaaS services available like Netlify Functions, AWS (which is harder to use than it should be), and many others.

If you’re interested in getting started with SSAs, you can take a look at my open source plugins, several of which contain examples of simple Server Side Actions.

sophie1 · February 16, 2021, 12:51pm

@keith and @edwardbutcher hi guys, thanks so much for your comments. I will take a look into.

I am trying to scrape image, price, shop name and url from external urls to use on my site. The idea is to have these appear in a collage format, e.g. image (which is hyperlinked to site) and price and shop name underneath rather than as a repeating group.

Any ideas on the best way to go about this would be very very appreciated.

sophie1 · February 19, 2021, 6:17pm

@edwardbutcher @keith hey guys

what is the best way of getting data from metascrape to bubble? Is this best done using Netlify functions and setting up an API call or best done using one of bubbles built in functions as services?

or perhaps something like this metadata extraction REST API would work?:

Any help would be much appreciated

edwardbutcher · February 23, 2021, 7:30pm

Several different routes could work, depends a bit on how comfortable you are with JS / Python.
If this were me then I would:

Use JS / Node.js plus a neat scraping framework. BeautifulSoup is a popular Python scraping framework and someone has built an equivalent for JS / Node.js called JSSoup… so I would use that.
Build out a draft of what I’m trying to do outside of Bubble, before bringing it in at the final stage. This is because Bubble hosted server workflows are a bit of a pain to debug. I would use repl.it as it spins up a Node.js environment really easily. I created a test for you here where I’ve brought JSSoup into repl.it. You can fork that to get started. BeautifulSoup documentation is here and applies with JSSoup - although note that underscore_case is replaced by camelCase.
Once this is working and scraping correctly then I would bring it over to Bubble using a Server Side action - as explained in this blog post you can easily bring in Node.js modules (like JSSoup) in a SS action.

No claiming that this is the best / simplest / fastest way. This would just be my approach.
Happy to help if you have any questions.

sophie1 · March 24, 2021, 9:34am

Hey @edwardbutcher thanks so much for your help!!! I followed your suggestion and was able to bring it in using a SS action. Thanks again for getting back and your advice! Amazing!

danella.cher · April 13, 2021, 9:14pm

@sophie1 Sophie

I’m not confiable with any coding unfortunately. Could you share in more layman’s terms how you were able to use his suggestion? I am trying to pull products as well. Thanks for any help you could provide