Web scraping tips

husam.hamdy · 2025-01-18T16:22:58.233Z

Hi bubblers,
My whole app is now relying on database from a council website that shows some data when a postcode is enterd.
I managed to get database with all the postcodes, lsoa,msoa, parish, town etc more than 1k record.
The council don’t mind sharing thd data but they said they don’t have the time or staff.
Before i give up i found a topic about something called scraping which pulls data from another website.
Please, if anyone knows if that would ever work would be so appreciated.
The website procrss is like this

[Input-Postcode] [submit-button]
[Dropdown menu- house number] [submit-button]

When both above buttons pressed the url parameter goes like ?postcode=ABCD&usp=123466

In the page the is a button when clicked shows the data i need

[Show data]

Data shows in a group
A pdf download here

Any tips plz if scraping is possbile for this case
Thank you

hi.luisacosta · 2025-01-18T16:27:31.379Z

It’s a bit unclear what is going on and what you need. Maybe use Ai refine the post and question?

Maybe include screenshots instead of just text explanation

husam.hamdy · 2025-01-18T16:35:55.259Z

Oh my bad.
So basically where i live there is different dates for bin collection depending on the postcodes (zip code in UK)
I have dataset of all postcodes in my area.
The council deliver leaflets to each house stating their collection days.
Some ares have ( rubbish and food waste Mondays fortnighly) and ( recycling and food waste the other Mondays) and so on.
I have tried to find out how they split the postcodes but no luck.
As the council website provide seevice when entering the postcode and house number it shows the collection dates associated with that postcode.
I need a scrape tool to extract this data for every postcode.
Sounds mad lol

mitchbaylis · 2025-01-19T02:28:44.994Z

first I’d check to see if there is an api for it on rapidapi - if there is then problem solved

otherwise you could use a macro to essentially enter a postcode and fetch the website data and repeat all the way through the list… you’ll hit bot blockers and will have to work around them

also check apify to see if there is a scraper already built

othewise octoparse

husam.hamdy · 2025-01-19T13:44:00.339Z

They dont have any APIs unfortunatly.
What is the Marcro please?

I will check the apify you recommended!
Thanks so much

florianherzog · 2025-01-19T14:22:45.601Z

you could also inspect with devtool in your browser (network tab), what exactly the site does, when you enter the postcode & house number. If it hits an endpoint with a GET or POST request, replicate that request in Postman to try out if unauthenticated users can access the API and, if it accepts any client, then replicate the request in a python script (or whatever language you prefer) and write the reply in a database on your disk (or make a backend workflow that accepts the inputs from your script and then directly write to your Bubble DB).