Several different routes could work, depends a bit on how comfortable you are with JS / Python.
If this were me then I would:
- Use JS / Node.js plus a neat scraping framework. BeautifulSoup is a popular Python scraping framework and someone has built an equivalent for JS / Node.js called JSSoup… so I would use that.
- Build out a draft of what I’m trying to do outside of Bubble, before bringing it in at the final stage. This is because Bubble hosted server workflows are a bit of a pain to debug. I would use repl.it as it spins up a Node.js environment really easily. I created a test for you here where I’ve brought JSSoup into repl.it. You can fork that to get started. BeautifulSoup documentation is here and applies with JSSoup - although note that underscore_case is replaced by camelCase.
- Once this is working and scraping correctly then I would bring it over to Bubble using a Server Side action - as explained in this blog post you can easily bring in Node.js modules (like JSSoup) in a SS action.
No claiming that this is the best / simplest / fastest way. This would just be my approach.
Happy to help if you have any questions.